mirror of
https://github.com/mudler/LocalAI.git
synced 2026-05-20 06:35:41 -04:00
Compare commits
3 Commits
v4.2.1
...
ci/layered
| Author | SHA1 | Date | |
|---|---|---|---|
|
9d42a16c20 | ||
|
|
9c1f8b344c | ||
|
|
a3b7c3a819 |
@@ -28,61 +28,13 @@ For Rust backends, you'll typically need (see `backend/rust/kokoros/` as a refer
|
||||
- `run.sh` - Sets `LD_LIBRARY_PATH`/`SSL_CERT_DIR` and execs the binary via the bundled `lib/ld.so`
|
||||
- `sources/<UpstreamProject>/` - Git submodule with the upstream Rust crate
|
||||
|
||||
## 2. Add Build Configurations to `.github/backend-matrix.yml`
|
||||
## 2. Add Build Configurations to `.github/workflows/backend.yml`
|
||||
|
||||
The build matrix is data-only YAML at `.github/backend-matrix.yml` (not inside `backend.yml` itself). `backend.yml` (master push) and `backend_pr.yml` (PR) load it via `scripts/changed-backends.js`, which also handles per-file path filtering so only touched backends rebuild on PRs and master pushes alike. Add build matrix entries to `.github/backend-matrix.yml` for each platform/GPU type you want to support. Look at similar backends for reference — `chatterbox`/`faster-whisper` for Python, `piper`/`silero-vad` for Go, `kokoros` for Rust.
|
||||
Add build matrix entries for each platform/GPU type you want to support. Look at similar backends for reference — `chatterbox`/`faster-whisper` for Python, `piper`/`silero-vad` for Go, `kokoros` for Rust.
|
||||
|
||||
**Without an entry here no image is ever built or pushed, and the gallery entry in `backend/index.yaml` will point at a tag that does not exist.** The `dockerfile:` field must point at `./backend/Dockerfile.<lang>` matching the language bucket from step 1 (e.g. `Dockerfile.python`, `Dockerfile.golang`, `Dockerfile.rust`). The `tag-suffix` must match the `uri:` in the corresponding `backend/index.yaml` image entry exactly.
|
||||
|
||||
**`scripts/changed-backends.js` registration — REQUIRED for any new dockerfile suffix.** This is the single most common omission, because it has no effect on the PR that adds the backend (when no prior path filter could catch it anyway) — it only breaks the *next* PR that touches your backend's directory, which then gets zero CI jobs and looks broken for unrelated reasons. Edit `scripts/changed-backends.js:inferBackendPath` and add a branch BEFORE the more-generic suffixes:
|
||||
|
||||
```js
|
||||
if (item.dockerfile.endsWith("<your-dockerfile-suffix>")) {
|
||||
return `backend/cpp/<your-backend>/`; // or backend/python|go|rust/...
|
||||
}
|
||||
```
|
||||
|
||||
The `endsWith()` test is against the matrix entry's `dockerfile:` value (e.g. `./backend/Dockerfile.ds4` → `endsWith("ds4")`). Specificity order matters here just like it does for importers: more-specific suffixes go BEFORE more-generic ones (e.g. `ds4` before `llama-cpp` even though both end with letters, because some upstream might one day call itself `super-ds4-llama-cpp`). Verify locally before pushing:
|
||||
|
||||
```bash
|
||||
# Confirm your dockerfile suffix is unique enough
|
||||
node -e "
|
||||
const yaml = require('js-yaml'); const fs = require('fs');
|
||||
const m = yaml.load(fs.readFileSync('.github/backend-matrix.yml','utf8'));
|
||||
for (const e of m.include.filter(e => e.backend === '<your-backend>')) {
|
||||
console.log(e.dockerfile, '->', e.dockerfile.endsWith('<suffix>'));
|
||||
}"
|
||||
```
|
||||
|
||||
A quick way to find the right insertion point: `grep -n 'item.dockerfile.endsWith' scripts/changed-backends.js`.
|
||||
|
||||
**`bump_deps.yaml` registration — REQUIRED for any backend pinning an upstream commit.** If your backend's Makefile has a `*_VERSION?=<sha>` pin to a third-party repo, the daily auto-bump bot at `.github/workflows/bump_deps.yaml` won't notice it unless you register the backend in its matrix. The bot runs `.github/bump_deps.sh` which `grep`s for `^$VAR?=` in the Makefile you list — so the pin MUST live in the Makefile (not in a separate shell script). The bump for ds4 (#9761) had to walk this back because the original landed the pin in `prepare.sh`, which the bot can't see. Pattern (for `antirez/ds4`):
|
||||
|
||||
```yaml
|
||||
# .github/workflows/bump_deps.yaml
|
||||
matrix:
|
||||
include:
|
||||
- repository: "antirez/ds4"
|
||||
variable: "DS4_VERSION"
|
||||
branch: "main"
|
||||
file: "backend/cpp/ds4/Makefile"
|
||||
```
|
||||
|
||||
And the corresponding Makefile shape (mirror `backend/cpp/llama-cpp/Makefile`):
|
||||
|
||||
```makefile
|
||||
DS4_VERSION?=ae302c2fa18cc6d9aefc021d0f27ae03c9ad2fc0
|
||||
DS4_REPO?=https://github.com/antirez/ds4
|
||||
...
|
||||
ds4:
|
||||
mkdir -p ds4
|
||||
cd ds4 && git init -q && \
|
||||
git remote add origin $(DS4_REPO) && \
|
||||
git fetch --depth 1 origin $(DS4_VERSION) && \
|
||||
git checkout FETCH_HEAD
|
||||
```
|
||||
|
||||
If you have a `prepare.sh` doing the clone, delete it — the recipe belongs in the Makefile target so `make purge && make` works as a clean-and-rebuild and so the bump bot finds the pin.
|
||||
If you add a new language bucket, `scripts/changed-backends.js` also needs a branch in `inferBackendPath` so PR change-detection routes file edits correctly.
|
||||
|
||||
**Placement in file:**
|
||||
- CPU builds: Add after other CPU builds (e.g., after `cpu-chatterbox`)
|
||||
@@ -94,14 +46,6 @@ If you have a `prepare.sh` doing the clone, delete it — the recipe belongs in
|
||||
- Intel/SYCL: Use `build-type: 'intel'` or `build-type: 'sycl_f16'`/`sycl_f32` with `base-image: "intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04"`
|
||||
- L4T (ARM): Use `build-type: 'l4t'` with `platforms: 'linux/arm64'` and `runs-on: 'ubuntu-24.04-arm'`
|
||||
|
||||
**Per-arch native builds (`linux/amd64` + `linux/arm64`):**
|
||||
|
||||
Multi-arch backends are NOT a single matrix entry with `platforms: 'linux/amd64,linux/arm64'`. Instead, add **two** entries — one with `platforms: 'linux/amd64'` + `platform-tag: 'amd64'` + `runs-on: 'ubuntu-latest'`, one with `platforms: 'linux/arm64'` + `platform-tag: 'arm64'` + `runs-on: 'ubuntu-24.04-arm'` — both sharing the same `tag-suffix`. The script detects the shared `tag-suffix` and emits a `merge-matrix` entry, so `backend-merge-jobs` (in `backend.yml`/`backend_pr.yml`) automatically assembles the manifest list from per-arch digest artifacts. See `-cpu-faster-whisper` in `.github/backend-matrix.yml` for a reference shape.
|
||||
|
||||
**llama-cpp / ik-llama-cpp / turboquant variants only — `builder-base-image`:**
|
||||
|
||||
Entries whose `dockerfile` is `./backend/Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` must also set a `builder-base-image` field pointing at a prebuilt base from `quay.io/go-skynet/ci-cache:base-grpc-*` (CI builds these via `.github/workflows/base-images.yml`). The mapping is by `(build-type, platforms)` — see existing entries for the pattern. CI uses these prebuilt bases to skip the gRPC compile (~25–35 min cold). Local `make backends/<name>` ignores `builder-base-image` and uses the from-source path inside the Dockerfile, so you don't need quay access for local builds.
|
||||
|
||||
## 3. Add Backend Metadata to `backend/index.yaml`
|
||||
|
||||
**Step 3a: Add Meta Definition**
|
||||
@@ -201,7 +145,7 @@ docker-build-backends: ... docker-build-<backend-name>
|
||||
After adding a new backend, verify:
|
||||
|
||||
- [ ] Backend directory structure is complete with all necessary files
|
||||
- [ ] Build configurations added to `.github/backend-matrix.yml` for all desired platforms (per-arch entries with `platform-tag` for multi-arch; `builder-base-image` for llama-cpp / ik-llama-cpp / turboquant)
|
||||
- [ ] Build configurations added to `.github/workflows/backend.yml` for all desired platforms
|
||||
- [ ] Meta definition added to `backend/index.yaml` in the `## metas` section
|
||||
- [ ] Image entries added to `backend/index.yaml` for all build variants (latest + development)
|
||||
- [ ] Tag suffixes match between workflow file and index.yaml
|
||||
|
||||
@@ -8,9 +8,8 @@ Let's say the user wants to build a particular backend for a given platform. For
|
||||
|
||||
- The Makefile has targets like `docker-build-coqui` created with `generate-docker-build-target` at the time of writing. Recently added backends may require a new target.
|
||||
- At a minimum we need to set the BUILD_TYPE, BASE_IMAGE build-args
|
||||
- Use `.github/backend-matrix.yml` as a reference — it's the data-only YAML that lists every backend variant's `build-type`, `base-image`, `platforms`, etc. (`backend.yml` and `backend_pr.yml` consume it via `scripts/changed-backends.js`).
|
||||
- l4t and cublas also require the CUDA major and minor version.
|
||||
- For llama-cpp / ik-llama-cpp / turboquant the matrix also sets `builder-base-image` pointing at a prebuilt `quay.io/go-skynet/ci-cache:base-grpc-*` tag. Local `make backends/<name>` defaults to `BUILDER_TARGET=builder-fromsource` and doesn't need it — the Dockerfile's from-source stage installs everything itself.
|
||||
- Use .github/workflows/backend.yml as a reference it lists the needed args in the `include` job strategy matrix
|
||||
- l4t and cublas also requires the CUDA major and minor version
|
||||
- You can pretty print a command like `DOCKER_MAKEFLAGS=-j$(nproc --ignore=1) BUILD_TYPE=hipblas BASE_IMAGE=rocm/dev-ubuntu-24.04:7.2.1 make docker-build-coqui`
|
||||
- Unless the user specifies that they want you to run the command, then just print it because not all agent frontends handle long running jobs well and the output may overflow your context
|
||||
- The user may say they want to build AMD or ROCM instead of hipblas, or Intel instead of SYCL or NVIDIA insted of l4t or cublas. Ask for confirmation if there is ambiguity.
|
||||
|
||||
@@ -1,120 +1,37 @@
|
||||
# CI Build Caching
|
||||
|
||||
Container builds — both the root LocalAI image (`Dockerfile`) and the per-backend images (`backend/Dockerfile.*`) — share a registry-backed BuildKit cache plus a layered set of prebuilt base images. This file explains how the cache is laid out, what invalidates it, and how to bypass it.
|
||||
|
||||
## Workflow surfaces
|
||||
|
||||
| Workflow | Purpose | Triggers |
|
||||
|---|---|---|
|
||||
| `.github/workflows/backend.yml` | Backend container images on master | `push` to master + tags, weekly Sunday cron, `workflow_dispatch` |
|
||||
| `.github/workflows/backend_pr.yml` | Backend container images on PRs | `pull_request` |
|
||||
| `.github/workflows/backend_build.yml` | Reusable: builds one backend (one arch) by digest | `workflow_call` from above |
|
||||
| `.github/workflows/backend_merge.yml` | Reusable: assembles per-arch digests into a multi-arch manifest list | `workflow_call` |
|
||||
| `.github/workflows/backend_build_darwin.yml` | Reusable: macOS-native backend builds | `workflow_call` |
|
||||
| `.github/workflows/image.yml` / `image-pr.yml` | Root LocalAI image (push / PR) | push / PR |
|
||||
| `.github/workflows/image_build.yml` / `image_merge.yml` | Reusable: per-arch root-image build + merge | `workflow_call` |
|
||||
| `.github/workflows/base-images.yml` | Builds the prebuilt `base-grpc-*` builder bases | Saturdays 05:00 UTC cron, `workflow_dispatch`, master push touching `Dockerfile.base-grpc-builder`, `.docker/install-base-deps.sh`, `.docker/apt-mirror.sh`, or this workflow |
|
||||
|
||||
The matrix that drives `backend.yml` / `backend_pr.yml` lives in **`.github/backend-matrix.yml`** (data-only YAML, not embedded in the workflow). `scripts/changed-backends.js` parses it, applies path-filter logic against the PR diff (PR events) or the GitHub Compare API (push events), and emits the filtered matrix plus a `merge-matrix` for backends with multiple per-arch entries.
|
||||
Container builds — both the root LocalAI image (`Dockerfile`) and the per-backend images (`backend/Dockerfile.*`) — share a registry-backed BuildKit cache. This file explains how that cache is laid out, what invalidates it, and how to bypass it.
|
||||
|
||||
## Cache layout
|
||||
|
||||
- **Cache registry**: `quay.io/go-skynet/ci-cache`
|
||||
- **One tag per matrix entry per arch**, derived from `tag-suffix` and `platform-tag`:
|
||||
- Backend builds (`backend_build.yml`): `cache<tag-suffix>-<platform-tag>`
|
||||
- e.g. `cache-cpu-faster-whisper-amd64`, `cache-cpu-faster-whisper-arm64`, `cache-gpu-nvidia-cuda-13-llama-cpp-amd64`
|
||||
- Root image builds (`image_build.yml`): `cache-localai<tag-suffix>-<platform-tag>` (with a `-core` placeholder when `tag-suffix` is empty, so `cache-localai-core-amd64` for the core image)
|
||||
- Pre-built base images (`base-images.yml`): `cache-base-grpc-<variant>` (one per `(BUILD_TYPE, arch)` permutation)
|
||||
- Each tag stores a multi-arch BuildKit cache manifest (`mode=max`), so every intermediate stage is re-usable, not just the final image.
|
||||
|
||||
The per-arch suffix exists because amd64 and arm64 builds produce different intermediate content; sharing one cache key would thrash on every cross-arch rebuild.
|
||||
- **Tag prefixes**:
|
||||
- Backend builds (`backend_build.yml`) buildkit cache: `cache<tag-suffix>`
|
||||
- e.g. `cache-gpu-nvidia-cuda-12-llama-cpp`, `cache-cpu-vllm`, `cache-nvidia-l4t-cuda-13-arm64-vllm`
|
||||
- Root image builds (`image_build.yml`) buildkit cache: `cache-localai<tag-suffix>`
|
||||
- e.g. `cache-localai-gpu-nvidia-cuda-12`, `cache-localai-gpu-vulkan`
|
||||
- Layered base builds (`base_images.yml`) buildkit cache: `base-<stem>`
|
||||
- e.g. `base-python-cpu-2404`, `base-cpp-cublas-2404-cuda13.0`
|
||||
- Layered base **images** (the OCI manifests consumers FROM): `base-image-<stem>[-pr<N>]`
|
||||
- e.g. `base-image-python-cpu-2404`, `base-image-cpp-cublas-2404-cuda13.0-pr9672`
|
||||
- The cache tags store multi-arch BuildKit cache manifests (`mode=max`); the `base-image-*` tags store ordinary OCI image manifests.
|
||||
|
||||
## Read/write semantics
|
||||
|
||||
| Trigger | `cache-from` | `cache-to` |
|
||||
|---|---|---|
|
||||
| `push` to `master` / tag / cron / dispatch | yes | yes (`mode=max,ignore-error=true`) |
|
||||
| `push` to `master` / tag | yes | yes (`mode=max,ignore-error=true`) |
|
||||
| `pull_request` | yes | **no** |
|
||||
|
||||
PR builds read master's warm cache but never write — this prevents PRs from polluting the shared cache with their experimental state. After merge, the master build for that matrix entry refreshes the cache.
|
||||
|
||||
`ignore-error=true` on the write side means a transient quay push failure does not fail the build; the next master push retries.
|
||||
|
||||
## Pre-built base images (`base-grpc-*`)
|
||||
## Self-warming, no separate populator
|
||||
|
||||
The C++ backend Dockerfiles (`Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}`) compile gRPC from source. On a cold build that's ~25–35 min before any LocalAI source compiles. To skip that on CI, `.github/workflows/base-images.yml` builds and pushes a set of pre-prepped builder bases:
|
||||
There is no cron job that pre-warms the cache. The production builds *are* the populator. The first master build of a given matrix entry pays the cold cost; subsequent same-entry master builds reuse everything that hasn't changed (apt installs, gRPC compile in `Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}`, Python wheel installs, etc.).
|
||||
|
||||
| Tag | Contents |
|
||||
|---|---|
|
||||
| `base-grpc-amd64` / `base-grpc-arm64` | Ubuntu 24.04 + apt build deps + protoc + cmake + gRPC at `/opt/grpc` |
|
||||
| `base-grpc-cuda-12-amd64` | the above + CUDA 12.8 toolkit |
|
||||
| `base-grpc-cuda-13-amd64` | the above + CUDA 13.0 toolkit (Ubuntu 22.04 base) |
|
||||
| `base-grpc-cuda-13-arm64` | the above + CUDA 13.0 sbsa toolkit (Ubuntu 24.04 base) |
|
||||
| `base-grpc-l4t-cuda-12-arm64` | JetPack r36.4.0 base (CUDA preinstalled, `SKIP_DRIVERS=true`) + gRPC |
|
||||
| `base-grpc-rocm-amd64` | rocm/dev-ubuntu-24.04:7.2.1 base + hipblas/hipblaslt/rocblas + gRPC |
|
||||
| `base-grpc-vulkan-amd64` / `base-grpc-vulkan-arm64` | Ubuntu 24.04 + Vulkan SDK 1.4.335 + gRPC |
|
||||
| `base-grpc-intel-amd64` | intel/oneapi-basekit:2025.3.2 base + gRPC |
|
||||
|
||||
**Single source of truth**: the install logic for all 10 variants lives in `.docker/install-base-deps.sh`. Both `Dockerfile.base-grpc-builder` AND each variant Dockerfile's `builder-fromsource` stage bind-mount and execute the same script — so the prebuilt CI base and the local from-source path are bit-equivalent by construction.
|
||||
|
||||
### How variant Dockerfiles consume the base
|
||||
|
||||
`Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` are multi-target. Three stages plus a final aliasing stage:
|
||||
|
||||
- `builder-fromsource` — `FROM ${BASE_IMAGE}` then runs `install-base-deps.sh` and the per-backend compile script. Used when `BUILDER_TARGET=builder-fromsource` (the default; local `make backends/<name>`).
|
||||
- `builder-prebuilt` — `FROM ${BUILDER_BASE_IMAGE}` (one of the prebuilt `base-grpc-*` tags) and runs only the per-backend compile script. Used when `BUILDER_TARGET=builder-prebuilt` (CI when the matrix entry sets `builder-base-image`).
|
||||
- `FROM ${BUILDER_TARGET} AS builder` — alias resolves the ARG-selected stage to a fixed name (BuildKit doesn't allow ARG expansion in `COPY --from=`).
|
||||
- `FROM scratch` + `COPY --from=builder ...package/. ./` — emits the final scratch image with just the package contents.
|
||||
|
||||
BuildKit prunes the unreferenced builder stage, so each build only runs the path it needs. `backend_build.yml` derives `BUILDER_TARGET=builder-prebuilt` automatically when the matrix entry has a non-empty `builder-base-image`; otherwise it defaults to `builder-fromsource`.
|
||||
|
||||
The matrix `(build-type, platforms)` → `builder-base-image` mapping for llama-cpp / ik-llama-cpp / turboquant entries:
|
||||
|
||||
| `build-type` | `platforms` | tag |
|
||||
|---|---|---|
|
||||
| `''` | `linux/amd64` | `base-grpc-amd64` |
|
||||
| `''` | `linux/arm64` | `base-grpc-arm64` |
|
||||
| `cublas` cuda 12 | `linux/amd64` | `base-grpc-cuda-12-amd64` |
|
||||
| `cublas` cuda 13 | `linux/amd64` | `base-grpc-cuda-13-amd64` |
|
||||
| `cublas` cuda 13 | `linux/arm64` | `base-grpc-cuda-13-arm64` |
|
||||
| `cublas` cuda 12 + JetPack base | `linux/arm64` | `base-grpc-l4t-cuda-12-arm64` |
|
||||
| `hipblas` | `linux/amd64` | `base-grpc-rocm-amd64` |
|
||||
| `vulkan` | `linux/amd64` | `base-grpc-vulkan-amd64` |
|
||||
| `vulkan` | `linux/arm64` | `base-grpc-vulkan-arm64` |
|
||||
| `sycl_*` | `linux/amd64` | `base-grpc-intel-amd64` |
|
||||
|
||||
### Bootstrap order when adding a new variant
|
||||
|
||||
If you add a new entry to `base-images.yml`'s matrix, the new tag does not exist on quay until the workflow runs. To consume it from a variant entry safely, dispatch the base-images workflow on the branch first:
|
||||
|
||||
```bash
|
||||
gh workflow run base-images.yml --ref <feature-branch>
|
||||
```
|
||||
|
||||
Wait for the new variant to push, then merge the consumer change. Otherwise the consumer's CI fails with "image not found."
|
||||
|
||||
## Per-arch native builds + manifest merge
|
||||
|
||||
Multi-arch backends (and the core LocalAI image) build natively per arch instead of running both arches under QEMU emulation on a single x86 runner. The pattern:
|
||||
|
||||
- The matrix has TWO entries per multi-arch backend, sharing the same `tag-suffix` but distinct `platforms` + `platform-tag` + `runs-on`. Example: `-cpu-faster-whisper` has one amd64 entry on `ubuntu-latest` and one arm64 entry on `ubuntu-24.04-arm`.
|
||||
- Each per-arch build pushes by **canonical digest only** (no tags) via `outputs: type=image,push-by-digest=true,name-canonical=true,push=true`. The digest is uploaded as an artifact named `digests<tag-suffix>-<platform-tag>` (or `digests-localai<...>` for root-image builds).
|
||||
- `scripts/changed-backends.js` detects shared `tag-suffix` and emits a `merge-matrix` output. `backend.yml` / `backend_pr.yml` have a `backend-merge-jobs` job that consumes it and calls `backend_merge.yml`.
|
||||
- `backend_merge.yml` downloads all matching digest artifacts and runs `docker buildx imagetools create` to publish the final tagged manifest list pointing at both per-arch digests. Same `docker/metadata-action` config as the original monolithic build, so consumers see no tag-shape change.
|
||||
- `image_merge.yml` is the equivalent for the root LocalAI image (`-core` placeholder when `tag-suffix` is empty so the artifact-name glob doesn't over-match across `core` and `gpu-vulkan`).
|
||||
|
||||
**`provenance: false` is required on multi-registry digest pushes**: with the default `mode=max` provenance attestation, BuildKit bundles a per-registry attestation manifest into each registry's manifest list, making the resulting list digest diverge across registries. `steps.build.outputs.digest` only matches one of them and the merge step's `imagetools create <reg>@sha256:<digest>` lookup fails on the other. Setting `provenance: false` keeps the digest content-only and identical across registries.
|
||||
|
||||
## Path filter on master push
|
||||
|
||||
Both `backend.yml` (push) and `backend_pr.yml` (PR) generate their matrix dynamically through `scripts/changed-backends.js`:
|
||||
|
||||
- **PR events**: paginated `pulls/{n}/files` API → filter the matrix to entries whose `dockerfile` path prefix matches the PR diff.
|
||||
- **Push events**: GitHub Compare API (`/repos/{owner}/{repo}/compare/{before}...{after}`) → same path-filter logic. Falls back to "run everything" on first-branch push (`event.before` zero), API truncation (≥300 changed files), missing API token, or any thrown error.
|
||||
- **Tag pushes**: `FORCE_ALL=true` is set from the workflow side (`startsWith(github.ref, 'refs/tags/')`) — releases rebuild every backend regardless of diff.
|
||||
- **Schedule / `workflow_dispatch`**: no `event.before`, falls through to "run everything" automatically.
|
||||
|
||||
The Sunday 06:00 UTC cron on `backend.yml` exists specifically because path filtering can leave Python backends frozen on stale wheels. `DEPS_REFRESH` (below) only fires when the build actually runs, so an untouched Python backend would never re-resolve its unpinned deps. The weekly cron is the safety net.
|
||||
Historically there was a `generate_grpc_cache.yaml` cron that targeted a `grpc` stage in the root Dockerfile. That stage was removed in July 2025 and the cron silently failed every night for 9 months without writing anything. It was deleted along with the registry-cache rollout.
|
||||
|
||||
## The `DEPS_REFRESH` cache-buster (Python backends)
|
||||
|
||||
@@ -129,57 +46,18 @@ Most Python backends ship `requirements*.txt` files that **do not pin every tran
|
||||
|
||||
`DEPS_REFRESH` defends against that:
|
||||
|
||||
- `backend_build.yml` computes `date -u +%Y-W%V` (ISO week, e.g. `2026-W19`) before each build and passes it as a build-arg.
|
||||
- `backend_build.yml` computes `date -u +%Y-W%V` (ISO week, e.g. `2026-W17`) before each build and passes it as a build-arg.
|
||||
- The `RUN ... make` layer's BuildKit hash now includes that string, so the layer invalidates **at most once per week**, automatically picking up newer wheels.
|
||||
- Within a week, builds stay warm.
|
||||
|
||||
This applies only to `Dockerfile.python` because:
|
||||
- Go (`Dockerfile.golang`) pins versions in `go.mod` / `go.sum`.
|
||||
- Rust (`Dockerfile.rust`) pins via `Cargo.lock`.
|
||||
- C++ backends pin gRPC (`v1.65.0`) and llama.cpp at a specific commit; their inputs don't drift between rebuilds.
|
||||
- C++ backends (`Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}`) clone gRPC at a pinned tag (`v1.65.0`) and llama.cpp at a pinned commit; their inputs don't drift between rebuilds.
|
||||
|
||||
### Adjusting the cadence
|
||||
|
||||
Bump the format to daily (`+%Y-%m-%d`) or hourly (`+%Y-%m-%d-%H`) for faster refreshes. For one-shot rebuilds without changing the schedule, append a marker to the tag-suffix in the matrix or temporarily delete that backend's cache tag in quay.
|
||||
|
||||
## ccache for C++ backend builds
|
||||
|
||||
`Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` declare a BuildKit cache mount on `/root/.ccache`:
|
||||
|
||||
```dockerfile
|
||||
RUN --mount=type=cache,target=/root/.ccache,id=<backend>-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
|
||||
bash /usr/local/sbin/compile.sh
|
||||
```
|
||||
|
||||
The compile script exports `CMAKE_C/CXX/CUDA_COMPILER_LAUNCHER=ccache` so CMake threads ccache through gcc/g++/nvcc. `cache-to: type=registry,mode=max` exports the cache mount data into the registry cache, so subsequent builds restore it.
|
||||
|
||||
On a `LLAMA_VERSION` bump, most translation units are byte-identical to the previous version's preprocessed source — ccache returns the previous `.o` and skips the real compile. Same for LocalAI source changes that don't actually touch llama.cpp's CMake inputs. Cache scope is per `(TARGETARCH, BUILD_TYPE)` so e.g. cublas-12 doesn't share with cublas-13 (their CUDA headers differ; cross-pollination would just be cache misses anyway).
|
||||
|
||||
## Composite actions
|
||||
|
||||
Two composite actions handle runner-side prep:
|
||||
|
||||
- **`.github/actions/free-disk-space/action.yml`** — wraps `jlumbroso/free-disk-space@main` plus an explicit apt purge of dotnet/android/ghc/mono/etc. Reclaims ~6–10 GB on `ubuntu-latest`. No-op on self-hosted runners. Used by `backend_build.yml`, `image_build.yml`, `test.yml`, `tests-aio.yml`, etc.
|
||||
- **`.github/actions/setup-build-disk/action.yml`** — relocates Docker's data-root to `/mnt` on hosted X64 runners. GHA hosted `ubuntu-latest` ships ~75 GB of unused space at `/mnt`; combined with the free-disk-space cleanup this gives ~100 GB working space — enough for ROCm dev image + vLLM torch install + flash-attn intermediate layers. No-op on self-hosted and on non-X64 hosted runners. Used by `backend_build.yml`, `image_build.yml`, `base-images.yml`.
|
||||
|
||||
Both actions run before any docker buildx step.
|
||||
|
||||
## Concurrency
|
||||
|
||||
All `backend.yml` / `image.yml` / `test.yml` / etc. workflows use:
|
||||
|
||||
```yaml
|
||||
concurrency:
|
||||
group: ci-<workflow>-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
```
|
||||
|
||||
- **PR events** group by PR number → newer pushes to the same PR cancel old runs (intended).
|
||||
- **Push events** group by `github.sha` → each master commit gets its own run; rapid-fire merges don't cancel each other (this was a real issue prior — two master pushes 11 seconds apart would cancel the first's CI).
|
||||
|
||||
## Self-warming, no separate populator
|
||||
|
||||
There is no cron job that pre-warms the BuildKit cache for individual backends. The production builds *are* the populators. The first master build of a given matrix entry pays the cold cost; subsequent same-entry master builds reuse everything that hasn't changed (apt installs, gRPC compile in the variant `builder-fromsource` stage or skipped entirely when consuming `base-grpc-*`, Python wheel installs, etc.). The base-images workflow's weekly cron is the closest thing to a populator and only refreshes the prebuilt builder bases.
|
||||
If you need a faster refresh (e.g. while debugging an upstream flake), bump the format to daily (`+%Y-%m-%d`) or hourly (`+%Y-%m-%d-%H`). If you need a one-shot rebuild for a specific backend without changing the schedule, append a marker to the tag-suffix in the matrix or temporarily delete that backend's cache tag in quay.
|
||||
|
||||
## Manually evicting cache
|
||||
|
||||
@@ -189,19 +67,19 @@ To force a fully cold build for one backend or the whole image:
|
||||
# Delete a single tag (requires quay credentials with admin on the repo)
|
||||
curl -X DELETE \
|
||||
-H "Authorization: Bearer ${QUAY_TOKEN}" \
|
||||
https://quay.io/api/v1/repository/go-skynet/ci-cache/tag/cache-gpu-nvidia-cuda-12-vllm-amd64
|
||||
https://quay.io/api/v1/repository/go-skynet/ci-cache/tag/cache-gpu-nvidia-cuda-12-vllm
|
||||
|
||||
# List all tags
|
||||
curl -s -H "Authorization: Bearer ${QUAY_TOKEN}" \
|
||||
"https://quay.io/api/v1/repository/go-skynet/ci-cache/tag/?limit=100" | jq '.tags[].name'
|
||||
```
|
||||
|
||||
Eviction is rarely needed in normal operation — `DEPS_REFRESH` handles weekly drift, source changes invalidate naturally, and `mode=max` keeps the cache scoped per matrix entry per arch so a stale tag never bleeds into a different build.
|
||||
Eviction is rarely needed in normal operation — `DEPS_REFRESH` handles weekly drift, source changes invalidate naturally, and `mode=max` keeps the cache scoped per matrix entry so a stale tag never bleeds into a different build.
|
||||
|
||||
## What the cache does **not** cover
|
||||
## What the cache **does not** cover
|
||||
|
||||
- The `free-disk-space` and `setup-build-disk` composite actions run on every job — these reclaim runner-state, not Docker layers, so BuildKit caches don't apply.
|
||||
- Intermediate artifacts of `Build (PR)` are not pushed anywhere — PRs only build for verification.
|
||||
- The "Free Disk Space" / "Release space from worker" steps run on every job — these reclaim ~6 GB on `ubuntu-latest` runners. They are runner-state cleanup, not Docker, and BuildKit caches don't apply.
|
||||
- Intermediate artifacts of `Build and push (PR)` are not pushed anywhere — PRs only build for verification.
|
||||
- Darwin builds (see below) — macOS runners have no Docker daemon, so the registry-backed BuildKit cache cannot apply.
|
||||
|
||||
## Darwin native caches
|
||||
@@ -221,30 +99,182 @@ The Python wheel cache uses the same ISO-week cache-buster as the Linux `DEPS_RE
|
||||
|
||||
The brew Cellar cache requires `HOMEBREW_NO_AUTO_UPDATE=1` and `HOMEBREW_NO_INSTALL_CLEANUP=1` (set as job-level env). Without those, `brew install` would mutate the very directories that were just restored, defeating the cache.
|
||||
|
||||
**Force-link after cache restore**: `actions/cache` restores `/opt/homebrew/Cellar/*` but NOT the `/opt/homebrew/bin/*` symlinks. After a cache hit, `brew install` sees the Cellar entries and decides "already installed" without re-running its link step, leaving the formulas off PATH. The Dependencies step explicitly runs `brew link --overwrite` for every cached formula afterwards to ensure the symlinks exist.
|
||||
|
||||
For ccache, the workflow exports `CMAKE_ARGS=… -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache` via `$GITHUB_ENV` before running `make build-darwin-go-backend`. The Makefile in `backend/cpp/llama-cpp/` already forwards `CMAKE_ARGS` through to each variant build (`fallback`, `grpc`, `rpc-server`), so no script changes are needed. The three variants share most TUs, so ccache dedupes object files across them.
|
||||
|
||||
`backend_build_darwin.yml` also has a llama-cpp-specific build-step branch that runs `make backends/llama-cpp-darwin` (the bespoke script that compiles three CMake variants and bundles dylibs via `otool`), distinct from the generic `make build-darwin-${lang}-backend` path. This was consolidated from a previously-bespoke top-level `llama-cpp-darwin` job in `backend.yml` so llama-cpp on Darwin honors the same path filter as the other 34 Darwin backends.
|
||||
|
||||
### Cache budget on Darwin
|
||||
|
||||
GitHub Actions caches are limited to 10 GB per repo. Steady-state worst case: ~800 MB Go cache + ~2 GB brew Cellar + up to 2 GB ccache + ~1.5 GB × 5 python backends. If the cap is hit, prefer collapsing the per-backend Python keys into a shared `pyenv-darwin-shared-<week>` key (accepts more cross-backend churn for a smaller footprint) before reducing other caches.
|
||||
|
||||
## Self-hosted runners
|
||||
## Layered base images (`ci-cache:base-image-*`)
|
||||
|
||||
`.github/backend-matrix.yml` has zero references to `arc-runner-set` or `bigger-runner` — all backends run on GHA free-tier hosted runners (`ubuntu-latest` for amd64, `ubuntu-24.04-arm` for arm64 native, `macos-14` for Darwin). The migration off self-hosted relied on the per-arch native split (no QEMU emulation) plus `setup-build-disk`'s `/mnt` relocation (~100 GB working space, enough for ROCm dev image + vLLM/torch installs).
|
||||
The registry-backed BuildKit cache deduplicates **within** a matrix entry's
|
||||
cache tag, but each matrix entry has its own tag — so the same `apt-get`,
|
||||
GPU SDK install, and language toolchain bootstrap runs into N different
|
||||
cache tags across the backend matrix. The layered base images factor that
|
||||
shared work out of the per-backend builds.
|
||||
|
||||
One residual self-hosted reference remains in `test-extra.yml` (`tests-vibevoice-cpp-grpc-transcription` uses `bigger-runner` for the 30s JFK-decode timeout headroom). That's a separate concern.
|
||||
They live in the same `quay.io/go-skynet/ci-cache` repo as the buildkit
|
||||
caches, under a distinct `base-image-` tag prefix so the OCI image
|
||||
manifests coexist with `base-<stem>` (the cache for building the base),
|
||||
`cache<tag-suffix>` (per-backend caches), and `cache-localai<tag-suffix>`
|
||||
(root image caches). Reusing `ci-cache` means no new quay repo or robot
|
||||
grant is needed — the same credentials that write the cache also write
|
||||
the image.
|
||||
|
||||
### How it fits together
|
||||
|
||||
```
|
||||
.github/backend-matrix.yaml # raw matrix data (linux + darwin)
|
||||
│
|
||||
▼
|
||||
backend.yml / backend_pr.yml
|
||||
├── derive-bases / generate-matrix
|
||||
│ scripts/changed-backends.js
|
||||
│ reads .github/backend-matrix.yaml
|
||||
│ (PR mode also reads changed files)
|
||||
│ emits:
|
||||
│ - matrix (annotated with base-image-prebuilt)
|
||||
│ - matrix-darwin
|
||||
│ - bases-matrix (deduplicated by tag-stem)
|
||||
│
|
||||
├── build-bases (matrix: bases-matrix)
|
||||
│ uses base_images.yml
|
||||
│ FROM .docker/bases/Dockerfile.<lang>
|
||||
│ pushes quay.io/go-skynet/ci-cache:base-image-<stem>[-pr<N>]
|
||||
│
|
||||
└── backend-jobs (matrix: matrix; needs build-bases)
|
||||
uses backend_build.yml
|
||||
FROM ${BASE_IMAGE_PREBUILT}
|
||||
i.e. quay.io/go-skynet/ci-cache:base-image-<stem>[-pr<N>]
|
||||
only the backend source COPY + `make` remain.
|
||||
```
|
||||
|
||||
The base image is **always** built before backends consume it, in the same
|
||||
workflow run. There is no cross-workflow dependency, no chicken-and-egg
|
||||
on first push, and no manual matrix to keep in sync — adding a backend
|
||||
matrix entry is just an edit to `.github/backend-matrix.yaml`.
|
||||
|
||||
### Tag scheme
|
||||
|
||||
`<stem>` is computed by `tagStem()` in `scripts/changed-backends.js` from
|
||||
the (lang, build-type, ubuntu, cuda, base-image) tuple. Arch is
|
||||
intentionally NOT in the stem — bases are built multi-arch when any
|
||||
consumer needs multi-arch, and single-arch otherwise (the `platforms`
|
||||
field on each base entry is the union of its consumers' platforms).
|
||||
|
||||
| Build-type | Stem template |
|
||||
|---|---|
|
||||
| `''` (CPU) | `<lang>-cpu-<ubuntu>[-<base-image-slug>]` |
|
||||
| `cublas` / `l4t` | `<lang>-<build-type>-<ubuntu>-cuda<major>.<minor>[-<base-image-slug>]` |
|
||||
| anything else (vulkan, hipblas, intel, sycl_*) | `<lang>-<build-type>-<ubuntu>[-<base-image-slug>]` |
|
||||
|
||||
The base-image slug is empty for the default `ubuntu:24.04` and a short
|
||||
parseable suffix otherwise (`jetpack-r36.4.0`, `rocm-7.2.1`,
|
||||
`oneapi-2025.3.2`, etc.).
|
||||
|
||||
| Event | Pushed tag (in `quay.io/go-skynet/ci-cache`) |
|
||||
|---|---|
|
||||
| `push` (master/tag) | `:base-image-<stem>` |
|
||||
| `pull_request` | `:base-image-<stem>-pr<PR_NUMBER>` |
|
||||
|
||||
The buildkit cache for the base build itself lives at
|
||||
`quay.io/go-skynet/ci-cache:base-<stem>` (`mode=max,ignore-error=true`),
|
||||
parallel to the per-matrix-entry caches. The `base-` (cache) and
|
||||
`base-image-` (image) prefixes never collide.
|
||||
|
||||
The script also runs a collision check across consumers of each stem: if
|
||||
two consumers map to the same stem but disagree on `base-image` or
|
||||
`skip-drivers` (and skip-drivers is meaningful for that build-type), the
|
||||
script fails loudly. Resolve by encoding the differing input in
|
||||
`tagStem()` rather than letting the dedup silently pick a winner.
|
||||
|
||||
### PR testability
|
||||
|
||||
PRs run the same pipeline as master: derive bases → build bases (tagged
|
||||
`-pr<N>`) → run filtered backend matrix consuming those `-pr<N>` tags.
|
||||
End-to-end validation always lives within the PR.
|
||||
|
||||
For PRs that only change `.docker/bases/Dockerfile.<lang>` (no backend
|
||||
source touched), `changed-backends.js` adds one canary backend matrix
|
||||
entry per (lang × build-type × arch × cuda × ubuntu) tuple to the filtered
|
||||
matrix so each base flavour gets exercised.
|
||||
|
||||
### Existing language tiers
|
||||
|
||||
| Tier (lang) | Recipe | Consumer Dockerfile(s) | Distinct stems |
|
||||
|---|---|---|---|
|
||||
| `python` | `.docker/bases/Dockerfile.python` | `backend/Dockerfile.python` | 9 |
|
||||
| `golang` | `.docker/bases/Dockerfile.golang` | `backend/Dockerfile.golang` | 8 |
|
||||
| `cpp` | `.docker/bases/Dockerfile.cpp` (apt + GPU + protoc + cmake + GRPC) | `backend/Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` | 8 |
|
||||
| `rust` | `.docker/bases/Dockerfile.rust` | `backend/Dockerfile.rust` | 1 |
|
||||
|
||||
The C++ trio share a single `cpp` base because they only differ in their
|
||||
per-backend `make` targets. `langOf()` in `scripts/changed-backends.js`
|
||||
remaps `Dockerfile.{llama-cpp,ik-llama-cpp,turboquant}` → `cpp` so dedup
|
||||
works across the trio. If a future C++ consumer needs a *different* base
|
||||
(e.g. without GRPC, or with a different protoc version), give it its own
|
||||
`Dockerfile.<newlang>` recipe and remove it from the cpp remap.
|
||||
|
||||
### Adding a new (accel × arch × cuda × lang) flavour
|
||||
|
||||
Just add the matrix entry to `.github/backend-matrix.yaml` for the new
|
||||
flavour. The bases matrix and the per-entry `base-image-prebuilt` are
|
||||
derived automatically by `scripts/changed-backends.js`. Nothing else to
|
||||
change.
|
||||
|
||||
### Adding a new language tier
|
||||
|
||||
1. Create `.docker/bases/Dockerfile.<lang>` mirroring an existing tier
|
||||
(apt + accel install + lang-specific toolchain).
|
||||
2. Slim `backend/Dockerfile.<lang>` to `FROM ${BASE_IMAGE_PREBUILT}` plus
|
||||
the per-backend source COPY + build (no inline accel install).
|
||||
3. Add the new recipe to `baseTriggerFiles` in
|
||||
`scripts/changed-backends.js` so PRs touching it fan out to canaries.
|
||||
4. Add `<lang>: (item) => item.dockerfile.endsWith("<lang>")` to
|
||||
`langTriggerSelector` in the same file.
|
||||
5. Add a `LOCAL_BASE_<LANG>_TAG`, a `docker-build-<lang>-base` target,
|
||||
and a clause in `local-base-tag` / `local-base-target` in `Makefile`.
|
||||
|
||||
The `langsWithBase` set in `scripts/changed-backends.js` is auto-detected
|
||||
from the `.docker/bases/` directory at script startup, so step 1 alone is
|
||||
enough for the script to start emitting bases (and annotating matrix
|
||||
entries with `base-image-prebuilt`) for that lang. Steps 3–5 plug it
|
||||
into the canary fan-out and the local-build path.
|
||||
|
||||
### Why not just rely on `mode=max` cache?
|
||||
|
||||
`mode=max` deduplicates at the layer level, but each matrix entry has its
|
||||
own cache tag (`cache<tag-suffix>`). A change that invalidates the GPU SDK
|
||||
layer in one backend does not invalidate it in any other; each entry pays
|
||||
the full cost on its next rebuild. The shared base image is built once per
|
||||
(accel × arch × cuda × lang), then pulled by every backend that consumes
|
||||
it — that's the actual cross-matrix dedup.
|
||||
|
||||
### Local builds
|
||||
|
||||
All `backend/Dockerfile.{python,golang,cpp,rust}` consumers require
|
||||
`BASE_IMAGE_PREBUILT` (no inline fallback). The Makefile wires the right
|
||||
`docker-build-<lang>-base` as a prerequisite for each backend's
|
||||
`docker-build-<backend>` target, so:
|
||||
|
||||
```bash
|
||||
# Build any backend; the matching base is built first if needed.
|
||||
make docker-build-vllm BUILD_TYPE=cublas CUDA_MAJOR_VERSION=12 CUDA_MINOR_VERSION=8
|
||||
make docker-build-llama-cpp BUILD_TYPE=cublas CUDA_MAJOR_VERSION=13 CUDA_MINOR_VERSION=0
|
||||
make docker-build-rerankers # golang
|
||||
make docker-build-kokoros # rust
|
||||
```
|
||||
|
||||
Or build a base directly: `make docker-build-{python,golang,cpp,rust}-base
|
||||
BUILD_TYPE=...`. Or pull a pre-built one from quay if it exists for your
|
||||
target tuple.
|
||||
|
||||
## Touching the cache pipeline
|
||||
|
||||
When changing `image_build.yml`, `backend_build.yml`, any of the `backend/Dockerfile.*` files, `Dockerfile.base-grpc-builder`, `.docker/install-base-deps.sh`, `.docker/<backend>-compile.sh`, or `scripts/changed-backends.js`:
|
||||
When changing `image_build.yml`, `backend_build.yml`, or any of the `backend/Dockerfile.*` files:
|
||||
|
||||
1. **Don't drop `DEPS_REFRESH=...` from the build-args** without a replacement strategy (lockfiles, pinned requirements). Otherwise master will silently freeze on whichever versions were cached at the time.
|
||||
2. **Keep `(tag-suffix, platform-tag)` unique per matrix entry** — together they're the cache namespace. Two matrix entries sharing a key would clobber each other's cache.
|
||||
2. **Keep `tag-suffix` unique per matrix entry** — it's the cache namespace. Two matrix entries sharing a tag-suffix would clobber each other's cache.
|
||||
3. **Keep `cache-to` gated on `github.event_name != 'pull_request'`** — PRs must not write.
|
||||
4. **Keep `ignore-error=true` on `cache-to`** — quay registry hiccups must not fail builds.
|
||||
5. **Keep `provenance: false` on push-by-digest steps** — multi-registry digest divergence is the Bug We Already Fixed; reintroducing provenance attestation re-breaks the merge.
|
||||
6. **`install-base-deps.sh` is the single source of truth for base contents.** Both `Dockerfile.base-grpc-builder` (CI) and the variant Dockerfiles' `builder-fromsource` (local) bind-mount and execute it. If you add a package to one path, add it to the script — don't fork the logic into a Dockerfile RUN.
|
||||
7. **After adding a `base-images.yml` matrix variant, run the workflow on your branch before merging consumer changes** that depend on the new tag — otherwise the consumer's CI fails "image not found."
|
||||
5. **`tagStem()` in `scripts/changed-backends.js` is the single source of truth for base image tags.** The matrix entries are annotated with `base-image-prebuilt` in the same script run; backend-jobs reads the value as-is. There's no parallel YAML expression to keep in sync. Adding a new dimension to the stem (e.g. a slug for a new base-image variant) is a script change only.
|
||||
|
||||
@@ -1,84 +0,0 @@
|
||||
# Working on the ds4 Backend
|
||||
|
||||
`antirez/ds4` is a single-model inference engine for DeepSeek V4 Flash.
|
||||
LocalAI wraps the engine's C API (`ds4/ds4.h`) with a fresh C++ gRPC server at
|
||||
`backend/cpp/ds4/` - NOT a fork of llama-cpp's grpc-server.cpp.
|
||||
|
||||
## Pin
|
||||
|
||||
`backend/cpp/ds4/Makefile` pins `DS4_VERSION?=<sha>` at the top. The `ds4`
|
||||
target in the Makefile clones `antirez/ds4` at that commit (mirroring the
|
||||
llama-cpp / ik-llama-cpp / turboquant pattern). The bump-deps bot
|
||||
(`.github/workflows/bump_deps.yaml`) finds this pin via grep and opens a
|
||||
daily PR to update it. To bump manually: edit the `DS4_VERSION?=` line,
|
||||
then `make purge && make` (or rely on CI's clean build).
|
||||
|
||||
## Wire shape
|
||||
|
||||
| RPC | Implementation |
|
||||
|---|---|
|
||||
| Health, Free, Status | Trivial; no engine dependency for Health |
|
||||
| LoadModel | `ds4_engine_open` + `ds4_session_create`; backend is compile-time (DS4_NO_GPU → CPU, __APPLE__ → Metal, otherwise CUDA) |
|
||||
| TokenizeString | `ds4_tokenize_text` |
|
||||
| Predict | `ds4_engine_generate_argmax` + `DsmlParser` → one ChatDelta with content / reasoning_content / tool_calls[] |
|
||||
| PredictStream | Same, per-token ChatDelta writes |
|
||||
|
||||
## DSML
|
||||
|
||||
ds4 emits tool calls as literal text markers (`<｜DSML｜tool_calls>` etc.) -
|
||||
NOT special tokens. `dsml_parser.{h,cpp}` is our streaming state machine that
|
||||
classifies token bytes into CONTENT / REASONING / TOOL_START / TOOL_ARGS / TOOL_END
|
||||
events. `dsml_renderer.{h,cpp}` does the prompt direction: turns
|
||||
OpenAI tool_calls + role=tool messages back into DSML for the next turn.
|
||||
|
||||
## Thinking modes
|
||||
|
||||
`PredictOptions.Metadata["enable_thinking"]` gates thinking on/off (default ON).
|
||||
`["reasoning_effort"] == "max" | "xhigh"` selects `DS4_THINK_MAX`; anything else
|
||||
maps to `DS4_THINK_HIGH`. We pass the chosen mode to `ds4_chat_append_assistant_prefix`.
|
||||
|
||||
## Disk KV cache
|
||||
|
||||
`kv_cache.{h,cpp}` implements an SHA1-keyed file cache using ds4's public
|
||||
`ds4_session_save_payload` / `ds4_session_load_payload` API. Enable per request
|
||||
via `ModelOptions.Options[] = "kv_cache_dir:/some/path"`. Format is **our own** -
|
||||
NOT bit-compatible with ds4-server's KVC files (interop is a follow-up plan).
|
||||
|
||||
## Build matrix
|
||||
|
||||
| Build | Where | Notes |
|
||||
|---|---|---|
|
||||
| `cpu-ds4` (amd64 + arm64) | Linux GHA | ds4 considers CPU debug-only; useful only for wiring tests |
|
||||
| `cuda13-ds4` (amd64 + arm64) | Linux GHA + DGX Spark validation | Primary production path on Linux |
|
||||
| `ds4-darwin` (arm64) | macOS GHA runners | Metal; uses `scripts/build/ds4-darwin.sh` like llama-cpp-darwin |
|
||||
|
||||
cuda12 is intentionally omitted. ROCm / Vulkan / SYCL are not applicable.
|
||||
|
||||
## Hardware-gated validation
|
||||
|
||||
`tests/e2e-backends/backend_test.go` in `BACKEND_BINARY` mode:
|
||||
|
||||
```
|
||||
BACKEND_BINARY=$(pwd)/backend/cpp/ds4/package/run.sh \
|
||||
BACKEND_TEST_MODEL_FILE=/path/to/ds4flash.gguf \
|
||||
BACKEND_TEST_CAPS=health,load,predict,stream,tools \
|
||||
BACKEND_TEST_TOOL_PROMPT="What's the weather in Paris?" \
|
||||
go test -count=1 -timeout=30m -v ./tests/e2e-backends/...
|
||||
```
|
||||
|
||||
CI does not load the model; the suite is opt-in via env vars.
|
||||
|
||||
## Importer
|
||||
|
||||
`core/gallery/importers/ds4.go` (`DS4Importer`) auto-detects ds4 weights by
|
||||
matching the `antirez/deepseek-v4-gguf` repo URI or the
|
||||
`DeepSeek-V4-Flash-*.gguf` filename pattern. **Registered BEFORE
|
||||
`LlamaCPPImporter`** in `defaultImporters` - both match `.gguf` but ds4 is more
|
||||
specific, and first-match-wins. The importer emits `backend: ds4`, uses
|
||||
`ds4flash.gguf` as the local filename (matches ds4's own CLI default), and
|
||||
disables the Go-side automatic tool-parsing fallback (the C++ backend emits
|
||||
ChatDelta.tool_calls natively via `DsmlParser`).
|
||||
|
||||
ds4 is also listed in `core/http/endpoints/localai/backend.go`'s pref-only
|
||||
slice so the `/import-model` UI surfaces it as a manual choice for users who
|
||||
want to force the backend on a non-canonical URI.
|
||||
@@ -1,62 +0,0 @@
|
||||
# Working on the SGLang Backend
|
||||
|
||||
The SGLang backend lives at `backend/python/sglang/backend.py` (async gRPC). It wraps SGLang's `Engine` (`sglang.srt.entrypoints.engine.Engine`) and translates LocalAI's gRPC `PredictOptions` into SGLang sampling params + outputs into `Reply.chat_deltas`. Structurally it mirrors `backend/python/vllm/backend.py` — keep them shaped the same so changes in one have an obvious analog in the other.
|
||||
|
||||
## `engine_args` is the universal escape hatch
|
||||
|
||||
A small fixed set of fields on `ModelOptions` is mapped to typed SGLang kwargs in `LoadModel` (model, quantization, load_format, gpu_memory_utilization → mem_fraction_static, trust_remote_code, enforce_eager → disable_cuda_graph, tensor_parallel_size → tp_size, max_model_len → context_length, dtype). **Everything else** flows through the `engine_args:` YAML map.
|
||||
|
||||
Validation happens in `_apply_engine_args`. Keys are checked against `dataclasses.fields(ServerArgs)` (`sglang.srt.server_args.ServerArgs` is a flat `@dataclass` with ~380 fields). Unknown keys raise `ValueError` at LoadModel time with a `difflib.get_close_matches` suggestion — same shape as the vLLM backend.
|
||||
|
||||
**Precedence:** typed `ModelOptions` fields populate `engine_kwargs` first, then `engine_args` overrides them. So a YAML that sets both `gpu_memory_utilization: 0.9` and `engine_args.mem_fraction_static: 0.5` ends up at `0.5`. Document this when answering "why didn't my YAML field stick?".
|
||||
|
||||
**ServerArgs is flat.** Unlike vLLM, where speculative decoding is nested under `engine_args.speculative_config: {...}`, SGLang exposes flat top-level fields: `speculative_algorithm`, `speculative_draft_model_path`, `speculative_num_steps`, `speculative_eagle_topk`, `speculative_num_draft_tokens`, `speculative_dflash_block_size`, etc. There is no `speculative_config:` dict. Same goes for compilation, kv-transfer, attention — all flat.
|
||||
|
||||
The canonical reference is `python/sglang/srt/server_args.py:ServerArgs` (line ~304). When SGLang adds new flags, no LocalAI code change is needed — they're automatically available via `engine_args:`. The validator picks them up because it introspects the live dataclass.
|
||||
|
||||
## Speculative decoding cheatsheet
|
||||
|
||||
`--speculative-algorithm` accepts `EAGLE`, `EAGLE3`, `NEXTN`, `STANDALONE`, `NGRAM`, `DFLASH`. `NEXTN` is silently rewritten to `EAGLE` in `ServerArgs.__post_init__` (`server_args.py:3286-3287`). MTP (Multi-Token Prediction) is the same EAGLE path with `num_steps=1, eagle_topk=1, num_draft_tokens=2` against a target whose architecture has multi-token heads (e.g. MiMo-7B-RL, DeepSeek-V3-MTP).
|
||||
|
||||
| Algorithm | Drafter requirement | Gallery demo target | Gallery demo drafter |
|
||||
|-----------|--------------------|---------------------|----------------------|
|
||||
| `NEXTN` / `EAGLE` (MTP) | Assistant drafter or built-in heads | google/gemma-4-E2B-it, google/gemma-4-E4B-it | google/gemma-4-E2B-it-assistant, google/gemma-4-E4B-it-assistant |
|
||||
| `EAGLE3` | EAGLE3 draft head | (no gallery entry yet) | e.g. jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B |
|
||||
| `DFLASH` | Block-diffusion drafter | (no gallery entry yet) | e.g. z-lab/Qwen3-4B-DFlash-b16 |
|
||||
| `STANDALONE` | Smaller LLM as drafter | (no gallery entry yet) | any smaller chat-tuned LLM in the same family |
|
||||
| `NGRAM` | None — uses prefix history | (no gallery entry yet) | n/a |
|
||||
|
||||
The Gemma 4 demos use `mem_fraction_static: 0.85` (cookbook default) and the cookbook's `num_steps=5, num_draft_tokens=6, eagle_topk=1` parameters. Other algorithms are reachable from any user YAML via `engine_args:` but don't have shipped demos yet — that's a deliberate gallery scope choice, not a backend limitation.
|
||||
|
||||
Gemma 4 support requires sglang built from a commit that includes [PR #21952](https://github.com/sgl-project/sglang/pull/21952). LocalAI's pinned release for cublas12 / cublas13 includes it. The `l4t13` (JetPack 7 / sbsa cu130) build floors at `sglang>=0.5.0` because the `pypi.jetson-ai-lab.io` mirror still ships only `0.5.1.post2` as of 2026-05-06 — Gemma 4 / MTP recipes are therefore not available on l4t13 until that mirror catches up. `backend.py` keeps backward compat with the 0.5.x → 0.5.11 `SamplingParams.seed` → `sampling_seed` rename via runtime detection.
|
||||
|
||||
Compatibility caveats per the SGLang docs: DFLASH and NGRAM are incompatible with `enable_dp_attention`; DFLASH requires `pp_size == 1`; STANDALONE is incompatible with `enable_dp_attention`; NGRAM is CUDA-only and disables the overlap scheduler.
|
||||
|
||||
### `mem_fraction_static` + quantization + MTP on consumer GPUs
|
||||
|
||||
When combining online weight quantization (`engine_args.quantization: fp8` / `awq` / etc.) with built-in-head MTP (`speculative_algorithm: EAGLE`/`NEXTN`) on a tight VRAM budget, sglang's default `mem_fraction_static: 0.85` will OOM during draft-worker init. The reason: sglang quantizes the **target** model's transformer blocks but loads the **MTP draft worker's vocab embedding** at the source dtype (typically bf16). For a 7 B-class model with a 150k-token vocab × 4096 hidden, that's another ~1.2 GiB allocated *after* the static pool is reserved. At 0.85 fraction on a 16 GB card there's no room left.
|
||||
|
||||
Workaround: drop `mem_fraction_static` to ~0.7 so the post-static heap can absorb the MTP embedding alloc + CUDA graph private pools. Verified end-to-end on MiMo-7B-RL + fp8 + MTP on a 16 GB RTX 5070 Ti (`gallery/sglang-mimo-7b-mtp.yaml`) at ~88 tok/s. Models with larger vocabs or more MTP layers (e.g. DeepSeek-V3-MTP) need an even smaller fraction.
|
||||
|
||||
This isn't documented anywhere upstream as of 2026-05-06 — the SGLang Gemma 4 cookbook uses 0.85 because their MTP path doesn't go through `eagle_worker_v2.py` for an embedding-bearing draft module. Don't blanket-apply 0.7 across all sglang YAMLs; only when MTP-with-built-in-heads + quantization combine.
|
||||
|
||||
## Tool-call and reasoning parsers stay on `Options[]`
|
||||
|
||||
ServerArgs has `tool_call_parser` and `reasoning_parser` fields, and the backend does pass them through to `Engine` so SGLang's own HTTP/OAI surface keeps working. But for the **LocalAI** request path the backend constructs fresh per-request parser instances in `_make_parsers` (`backend.py:286`) because the parsers are stateful — the streaming and non-streaming paths each need their own.
|
||||
|
||||
So the user-facing knob stays on `Options[]`:
|
||||
|
||||
```yaml
|
||||
options:
|
||||
- tool_parser:hermes
|
||||
- reasoning_parser:deepseek_r1
|
||||
```
|
||||
|
||||
Putting these in `engine_args:` will set them on `ServerArgs` but the LocalAI-level streaming `ChatDelta` will not pick them up. Don't recommend that path.
|
||||
|
||||
## What's missing today (out of scope, but worth tracking)
|
||||
|
||||
- `core/config/hooks_sglang.go` — there is no SGLang equivalent of `hooks_vllm.go`. The vLLM hook auto-selects parsers for known model families from `parser_defaults.json` and seeds production engine_args defaults. A symmetric hook for SGLang could reuse the same `parser_defaults.json` (the SGLang parser names are different but the family detection is shared) and seed defaults like `enable_metrics: true` or attention-backend choices.
|
||||
- `core/gallery/importers/sglang.go` — vLLM has an importer that resolves model architecture → parser defaults at gallery-import time. A matching importer for SGLang would let `local-ai install` populate sensible parsers automatically.
|
||||
|
||||
These should be a follow-up PR, not a blocker for the engine_args feature.
|
||||
259
.docker/bases/Dockerfile.cpp
Normal file
259
.docker/bases/Dockerfile.cpp
Normal file
@@ -0,0 +1,259 @@
|
||||
# Shared C++ + accelerator base image for the llama-cpp / ik-llama-cpp /
|
||||
# turboquant trio. They differ only in their Makefile targets at build
|
||||
# time; the apt + GPU SDK + protoc + cmake + GRPC install is identical.
|
||||
#
|
||||
# Built once per (build-type, arch, ubuntu-version, cuda-version) combination
|
||||
# by .github/workflows/base_images.yml and pushed to
|
||||
# quay.io/go-skynet/ci-cache:base-image-<tag-stem>[-pr<N>]. Consumed by
|
||||
# backend/Dockerfile.{llama-cpp,ik-llama-cpp,turboquant} via the
|
||||
# BASE_IMAGE_PREBUILT build-arg. See .agents/ci-caching.md.
|
||||
|
||||
ARG BASE_IMAGE=ubuntu:24.04
|
||||
ARG APT_MIRROR=""
|
||||
ARG APT_PORTS_MIRROR=""
|
||||
|
||||
FROM ${BASE_IMAGE} AS grpc
|
||||
|
||||
ARG GRPC_MAKEFLAGS="-j4 -Otarget"
|
||||
ARG GRPC_VERSION=v1.65.0
|
||||
ARG CMAKE_FROM_SOURCE=false
|
||||
# CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
|
||||
ARG CMAKE_VERSION=3.31.10
|
||||
ARG APT_MIRROR
|
||||
ARG APT_PORTS_MIRROR
|
||||
|
||||
ENV MAKEFLAGS=${GRPC_MAKEFLAGS}
|
||||
|
||||
WORKDIR /build
|
||||
|
||||
RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
|
||||
APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
ca-certificates \
|
||||
build-essential curl libssl-dev \
|
||||
git wget && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN <<EOT bash
|
||||
if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
|
||||
curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
|
||||
else
|
||||
apt-get update && \
|
||||
apt-get install -y \
|
||||
cmake && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
EOT
|
||||
|
||||
# Build GRPC into /opt/grpc so we can copy it into the final base without
|
||||
# pulling in the full source tree. Mirrors the original two-stage layout in
|
||||
# Dockerfile.llama-cpp; absorbing it here means consumers no longer pay the
|
||||
# GRPC compile cost.
|
||||
RUN git clone --recurse-submodules --jobs 4 -b ${GRPC_VERSION} --depth 1 --shallow-submodules https://github.com/grpc/grpc && \
|
||||
mkdir -p /build/grpc/cmake/build && \
|
||||
cd /build/grpc/cmake/build && \
|
||||
sed -i "216i\ TESTONLY" "../../third_party/abseil-cpp/absl/container/CMakeLists.txt" && \
|
||||
cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX:PATH=/opt/grpc ../.. && \
|
||||
make && \
|
||||
make install && \
|
||||
rm -rf /build
|
||||
|
||||
|
||||
FROM ${BASE_IMAGE}
|
||||
|
||||
ARG CMAKE_FROM_SOURCE=false
|
||||
ARG CMAKE_VERSION=3.31.10
|
||||
ARG BUILD_TYPE
|
||||
ENV BUILD_TYPE=${BUILD_TYPE}
|
||||
ARG CUDA_MAJOR_VERSION
|
||||
ARG CUDA_MINOR_VERSION
|
||||
ARG SKIP_DRIVERS=false
|
||||
ENV CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION}
|
||||
ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
ARG TARGETARCH
|
||||
ARG TARGETVARIANT
|
||||
ARG UBUNTU_VERSION=2404
|
||||
ARG APT_MIRROR
|
||||
ARG APT_PORTS_MIRROR
|
||||
|
||||
LABEL org.opencontainers.image.source="https://github.com/mudler/LocalAI"
|
||||
LABEL org.opencontainers.image.description="LocalAI C++ (llama-cpp/ik-llama-cpp/turboquant) base image"
|
||||
LABEL org.localai.base.lang="cpp"
|
||||
|
||||
RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
|
||||
APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
ccache git \
|
||||
ca-certificates \
|
||||
make \
|
||||
pkg-config libcurl4-openssl-dev \
|
||||
curl unzip \
|
||||
libssl-dev wget && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Cuda
|
||||
ENV PATH=/usr/local/cuda/bin:${PATH}
|
||||
|
||||
# HipBLAS requirements
|
||||
ENV PATH=/opt/rocm/bin:${PATH}
|
||||
|
||||
|
||||
# Vulkan requirements
|
||||
RUN <<EOT bash
|
||||
if [ "${BUILD_TYPE}" = "vulkan" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
software-properties-common pciutils wget gpg-agent && \
|
||||
apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
|
||||
libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
|
||||
libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
|
||||
git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
|
||||
ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
|
||||
clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
|
||||
if [ "amd64" = "$TARGETARCH" ]; then
|
||||
wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz" && \
|
||||
tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
|
||||
rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
|
||||
mkdir -p /opt/vulkan-sdk && \
|
||||
mv 1.4.335.0 /opt/vulkan-sdk/ && \
|
||||
cd /opt/vulkan-sdk/1.4.335.0 && \
|
||||
./vulkansdk --no-deps --maxjobs \
|
||||
vulkan-loader \
|
||||
vulkan-validationlayers \
|
||||
vulkan-extensionlayer \
|
||||
vulkan-tools \
|
||||
shaderc && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/ && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/ && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/ && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/ && \
|
||||
rm -rf /opt/vulkan-sdk
|
||||
fi
|
||||
if [ "arm64" = "$TARGETARCH" ]; then
|
||||
mkdir vulkan && cd vulkan && \
|
||||
curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
|
||||
tar -xvf vulkan-sdk.tar.xz && \
|
||||
rm vulkan-sdk.tar.xz && \
|
||||
cd 1.4.335.0 && \
|
||||
cp -rfv aarch64/bin/* /usr/bin/ && \
|
||||
cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
|
||||
cp -rfv aarch64/include/* /usr/include/ && \
|
||||
cp -rfv aarch64/share/* /usr/share/ && \
|
||||
cd ../.. && \
|
||||
rm -rf vulkan
|
||||
fi
|
||||
ldconfig && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
EOT
|
||||
|
||||
# CuBLAS requirements
|
||||
RUN <<EOT bash
|
||||
if ( [ "${BUILD_TYPE}" = "cublas" ] || [ "${BUILD_TYPE}" = "l4t" ] ) && [ "${SKIP_DRIVERS}" = "false" ]; then
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
software-properties-common pciutils
|
||||
if [ "amd64" = "$TARGETARCH" ]; then
|
||||
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb
|
||||
fi
|
||||
if [ "arm64" = "$TARGETARCH" ]; then
|
||||
if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
|
||||
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb
|
||||
else
|
||||
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb
|
||||
fi
|
||||
fi
|
||||
dpkg -i cuda-keyring_1.1-1_all.deb && \
|
||||
rm -f cuda-keyring_1.1-1_all.deb && \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
|
||||
if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "$TARGETARCH" ]; then
|
||||
apt-get install -y --no-install-recommends \
|
||||
libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libcudnn9-cuda-${CUDA_MAJOR_VERSION} cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
|
||||
fi
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
EOT
|
||||
|
||||
|
||||
# https://github.com/NVIDIA/Isaac-GR00T/issues/343
|
||||
RUN <<EOT bash
|
||||
if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "arm64" ]; then
|
||||
wget https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
|
||||
dpkg -i cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
|
||||
cp /var/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/ && \
|
||||
apt-get update && apt-get -y install cudss cudss-cuda-${CUDA_MAJOR_VERSION} && \
|
||||
wget https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
|
||||
dpkg -i nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
|
||||
cp /var/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/ && \
|
||||
apt-get update && apt-get install -y nvpl
|
||||
fi
|
||||
EOT
|
||||
|
||||
# If we are building with clblas support, we need the libraries for the builds
|
||||
RUN if [ "${BUILD_TYPE}" = "clblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
libclblast-dev && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/* \
|
||||
; fi
|
||||
|
||||
RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
hipblas-dev \
|
||||
hipblaslt-dev \
|
||||
rocblas-dev && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/* && \
|
||||
ldconfig && \
|
||||
echo "rocBLAS library data architectures:" && \
|
||||
(ls /opt/rocm*/lib/rocblas/library/Kernels* 2>/dev/null || ls /opt/rocm*/lib64/rocblas/library/Kernels* 2>/dev/null) | grep -oP 'gfx[0-9a-z+-]+' | sort -u || \
|
||||
echo "WARNING: No rocBLAS kernel data found" \
|
||||
; fi
|
||||
|
||||
# Install protoc (the version in 22.04 is too old, and grpc's bundled protoc
|
||||
# would pull in a newer absl that breaks stablediffusion).
|
||||
RUN <<EOT bash
|
||||
if [ "amd64" = "$TARGETARCH" ]; then
|
||||
curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-x86_64.zip -o protoc.zip && \
|
||||
unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
|
||||
rm protoc.zip
|
||||
fi
|
||||
if [ "arm64" = "$TARGETARCH" ]; then
|
||||
curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-aarch_64.zip -o protoc.zip && \
|
||||
unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
|
||||
rm protoc.zip
|
||||
fi
|
||||
EOT
|
||||
|
||||
# Install CMake (the version in 22.04 is too old)
|
||||
RUN <<EOT bash
|
||||
if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
|
||||
curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
|
||||
else
|
||||
apt-get update && \
|
||||
apt-get install -y \
|
||||
cmake && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
EOT
|
||||
|
||||
COPY --from=grpc /opt/grpc /usr/local
|
||||
206
.docker/bases/Dockerfile.golang
Normal file
206
.docker/bases/Dockerfile.golang
Normal file
@@ -0,0 +1,206 @@
|
||||
# Shared Go + accelerator base image.
|
||||
#
|
||||
# Built once per (build-type, arch, ubuntu-version, cuda-version) combination
|
||||
# by .github/workflows/base_images.yml and pushed to
|
||||
# quay.io/go-skynet/ci-cache:base-image-<tag-stem>[-pr<N>]. Consumed by
|
||||
# backend/Dockerfile.golang via the BASE_IMAGE_PREBUILT build-arg.
|
||||
#
|
||||
# Mirrors the GPU stack stanzas in Dockerfile.python; the language-specific
|
||||
# tail at the bottom installs Go + grpc tooling. See .agents/ci-caching.md.
|
||||
|
||||
ARG BASE_IMAGE=ubuntu:24.04
|
||||
ARG APT_MIRROR=""
|
||||
ARG APT_PORTS_MIRROR=""
|
||||
|
||||
FROM ${BASE_IMAGE}
|
||||
|
||||
ARG BUILD_TYPE
|
||||
ENV BUILD_TYPE=${BUILD_TYPE}
|
||||
ARG CUDA_MAJOR_VERSION
|
||||
ARG CUDA_MINOR_VERSION
|
||||
ARG SKIP_DRIVERS=false
|
||||
ENV CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION}
|
||||
ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
ARG TARGETARCH
|
||||
ARG TARGETVARIANT
|
||||
ARG GO_VERSION=1.25.4
|
||||
ARG UBUNTU_VERSION=2404
|
||||
ARG APT_MIRROR
|
||||
ARG APT_PORTS_MIRROR
|
||||
|
||||
LABEL org.opencontainers.image.source="https://github.com/mudler/LocalAI"
|
||||
LABEL org.opencontainers.image.description="LocalAI Go+accelerator base image"
|
||||
LABEL org.localai.base.lang="golang"
|
||||
|
||||
# gcc-14 is the default on noble (ubuntu:24.04) but absent from jammy
|
||||
# (the L4T jetpack r36.4.0 base). LocalVQE needs it; the other Go backends
|
||||
# compile with the default gcc shipped via build-essential. Try gcc-14
|
||||
# from the configured repos and fall back gracefully when it's missing.
|
||||
RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
|
||||
APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
git ccache \
|
||||
ca-certificates \
|
||||
make cmake wget libopenblas-dev \
|
||||
curl unzip \
|
||||
libssl-dev && \
|
||||
if apt-cache show gcc-14 >/dev/null 2>&1 && apt-cache show g++-14 >/dev/null 2>&1; then \
|
||||
apt-get install -y --no-install-recommends gcc-14 g++-14 && \
|
||||
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-14 100 \
|
||||
--slave /usr/bin/g++ g++ /usr/bin/g++-14 \
|
||||
--slave /usr/bin/gcov gcov /usr/bin/gcov-14; \
|
||||
fi && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Cuda
|
||||
ENV PATH=/usr/local/cuda/bin:${PATH}
|
||||
|
||||
# HipBLAS requirements
|
||||
ENV PATH=/opt/rocm/bin:${PATH}
|
||||
|
||||
# Vulkan requirements
|
||||
RUN <<EOT bash
|
||||
if [ "${BUILD_TYPE}" = "vulkan" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
software-properties-common pciutils wget gpg-agent && \
|
||||
apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
|
||||
libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
|
||||
libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
|
||||
git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
|
||||
ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
|
||||
clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
|
||||
if [ "amd64" = "$TARGETARCH" ]; then
|
||||
wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz" && \
|
||||
tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
|
||||
rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
|
||||
mkdir -p /opt/vulkan-sdk && \
|
||||
mv 1.4.335.0 /opt/vulkan-sdk/ && \
|
||||
cd /opt/vulkan-sdk/1.4.335.0 && \
|
||||
./vulkansdk --no-deps --maxjobs \
|
||||
vulkan-loader \
|
||||
vulkan-validationlayers \
|
||||
vulkan-extensionlayer \
|
||||
vulkan-tools \
|
||||
shaderc && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/ && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/ && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/ && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/ && \
|
||||
rm -rf /opt/vulkan-sdk
|
||||
fi
|
||||
if [ "arm64" = "$TARGETARCH" ]; then
|
||||
mkdir vulkan && cd vulkan && \
|
||||
curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
|
||||
tar -xvf vulkan-sdk.tar.xz && \
|
||||
rm vulkan-sdk.tar.xz && \
|
||||
cd 1.4.335.0 && \
|
||||
cp -rfv aarch64/bin/* /usr/bin/ && \
|
||||
cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
|
||||
cp -rfv aarch64/include/* /usr/include/ && \
|
||||
cp -rfv aarch64/share/* /usr/share/ && \
|
||||
cd ../.. && \
|
||||
rm -rf vulkan
|
||||
fi
|
||||
ldconfig && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
EOT
|
||||
|
||||
# CuBLAS requirements
|
||||
RUN <<EOT bash
|
||||
if ( [ "${BUILD_TYPE}" = "cublas" ] || [ "${BUILD_TYPE}" = "l4t" ] ) && [ "${SKIP_DRIVERS}" = "false" ]; then
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
software-properties-common pciutils
|
||||
if [ "amd64" = "$TARGETARCH" ]; then
|
||||
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb
|
||||
fi
|
||||
if [ "arm64" = "$TARGETARCH" ]; then
|
||||
if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
|
||||
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb
|
||||
else
|
||||
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb
|
||||
fi
|
||||
fi
|
||||
dpkg -i cuda-keyring_1.1-1_all.deb && \
|
||||
rm -f cuda-keyring_1.1-1_all.deb && \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
|
||||
if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "$TARGETARCH" ]; then
|
||||
apt-get install -y --no-install-recommends \
|
||||
libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libcudnn9-cuda-${CUDA_MAJOR_VERSION} cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
|
||||
fi
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
EOT
|
||||
|
||||
|
||||
# https://github.com/NVIDIA/Isaac-GR00T/issues/343
|
||||
RUN <<EOT bash
|
||||
if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "arm64" ]; then
|
||||
wget https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
|
||||
dpkg -i cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
|
||||
cp /var/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/ && \
|
||||
apt-get update && apt-get -y install cudss cudss-cuda-${CUDA_MAJOR_VERSION} && \
|
||||
wget https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
|
||||
dpkg -i nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
|
||||
cp /var/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/ && \
|
||||
apt-get update && apt-get install -y nvpl
|
||||
fi
|
||||
EOT
|
||||
|
||||
# If we are building with clblas support, we need the libraries for the builds
|
||||
RUN if [ "${BUILD_TYPE}" = "clblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
libclblast-dev && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/* \
|
||||
; fi
|
||||
|
||||
RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
hipblas-dev \
|
||||
hipblaslt-dev \
|
||||
rocblas-dev && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/* && \
|
||||
ldconfig \
|
||||
; fi
|
||||
|
||||
# Install Go
|
||||
RUN curl -L -s https://go.dev/dl/go${GO_VERSION}.linux-${TARGETARCH}.tar.gz | tar -C /usr/local -xz
|
||||
ENV PATH=$PATH:/root/go/bin:/usr/local/go/bin:/usr/local/bin
|
||||
|
||||
# Install grpc compilers
|
||||
RUN go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2 && \
|
||||
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
|
||||
|
||||
# Install protoc (the version in 22.04 is too old, and grpc's bundled protoc
|
||||
# would pull in a newer absl that breaks stablediffusion).
|
||||
RUN <<EOT bash
|
||||
if [ "amd64" = "$TARGETARCH" ]; then
|
||||
curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-x86_64.zip -o protoc.zip && \
|
||||
unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
|
||||
rm protoc.zip
|
||||
fi
|
||||
if [ "arm64" = "$TARGETARCH" ]; then
|
||||
curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-aarch_64.zip -o protoc.zip && \
|
||||
unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
|
||||
rm protoc.zip
|
||||
fi
|
||||
EOT
|
||||
209
.docker/bases/Dockerfile.python
Normal file
209
.docker/bases/Dockerfile.python
Normal file
@@ -0,0 +1,209 @@
|
||||
# Shared Python + accelerator base image.
|
||||
#
|
||||
# Built once per (build-type, arch, ubuntu-version, cuda-version) combination
|
||||
# by .github/workflows/base_images.yml and pushed to
|
||||
# quay.io/go-skynet/ci-cache:base-image-<tag-stem>[-pr<N>]. Consumed by
|
||||
# backend/Dockerfile.python via the BASE_IMAGE_PREBUILT build-arg.
|
||||
# See .agents/ci-caching.md.
|
||||
|
||||
ARG BASE_IMAGE=ubuntu:24.04
|
||||
ARG APT_MIRROR=""
|
||||
ARG APT_PORTS_MIRROR=""
|
||||
|
||||
FROM ${BASE_IMAGE}
|
||||
|
||||
ARG BUILD_TYPE
|
||||
ENV BUILD_TYPE=${BUILD_TYPE}
|
||||
ARG CUDA_MAJOR_VERSION
|
||||
ARG CUDA_MINOR_VERSION
|
||||
ARG SKIP_DRIVERS=false
|
||||
ENV CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION}
|
||||
ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
ARG TARGETARCH
|
||||
ARG TARGETVARIANT
|
||||
ARG UBUNTU_VERSION=2404
|
||||
ARG APT_MIRROR
|
||||
ARG APT_PORTS_MIRROR
|
||||
|
||||
LABEL org.opencontainers.image.source="https://github.com/mudler/LocalAI"
|
||||
LABEL org.opencontainers.image.description="LocalAI Python+accelerator base image"
|
||||
LABEL org.localai.base.lang="python"
|
||||
|
||||
RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
|
||||
APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
ccache \
|
||||
ca-certificates \
|
||||
espeak-ng \
|
||||
curl \
|
||||
libssl-dev \
|
||||
git wget \
|
||||
git-lfs \
|
||||
unzip clang \
|
||||
upx-ucl \
|
||||
curl python3-pip \
|
||||
python-is-python3 \
|
||||
python3-dev llvm \
|
||||
libnuma1 libgomp1 \
|
||||
python3-venv make cmake && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN <<EOT bash
|
||||
if [ "${UBUNTU_VERSION}" = "2404" ]; then
|
||||
pip install --break-system-packages --user --upgrade pip
|
||||
else
|
||||
pip install --upgrade pip
|
||||
fi
|
||||
EOT
|
||||
|
||||
# Cuda
|
||||
ENV PATH=/usr/local/cuda/bin:${PATH}
|
||||
|
||||
# HipBLAS requirements
|
||||
ENV PATH=/opt/rocm/bin:${PATH}
|
||||
|
||||
# Vulkan requirements
|
||||
RUN <<EOT bash
|
||||
if [ "${BUILD_TYPE}" = "vulkan" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
software-properties-common pciutils wget gpg-agent && \
|
||||
apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
|
||||
libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
|
||||
libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
|
||||
git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
|
||||
ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
|
||||
clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
|
||||
if [ "amd64" = "$TARGETARCH" ]; then
|
||||
wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz" && \
|
||||
tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
|
||||
rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
|
||||
mkdir -p /opt/vulkan-sdk && \
|
||||
mv 1.4.335.0 /opt/vulkan-sdk/ && \
|
||||
cd /opt/vulkan-sdk/1.4.335.0 && \
|
||||
./vulkansdk --no-deps --maxjobs \
|
||||
vulkan-loader \
|
||||
vulkan-validationlayers \
|
||||
vulkan-extensionlayer \
|
||||
vulkan-tools \
|
||||
shaderc && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/ && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/ && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/ && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/ && \
|
||||
rm -rf /opt/vulkan-sdk
|
||||
fi
|
||||
if [ "arm64" = "$TARGETARCH" ]; then
|
||||
mkdir vulkan && cd vulkan && \
|
||||
curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
|
||||
tar -xvf vulkan-sdk.tar.xz && \
|
||||
rm vulkan-sdk.tar.xz && \
|
||||
cd 1.4.335.0 && \
|
||||
cp -rfv aarch64/bin/* /usr/bin/ && \
|
||||
cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
|
||||
cp -rfv aarch64/include/* /usr/include/ && \
|
||||
cp -rfv aarch64/share/* /usr/share/ && \
|
||||
cd ../.. && \
|
||||
rm -rf vulkan
|
||||
fi
|
||||
ldconfig && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
EOT
|
||||
|
||||
# CuBLAS requirements
|
||||
RUN <<EOT bash
|
||||
if ( [ "${BUILD_TYPE}" = "cublas" ] || [ "${BUILD_TYPE}" = "l4t" ] ) && [ "${SKIP_DRIVERS}" = "false" ]; then
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
software-properties-common pciutils
|
||||
if [ "amd64" = "$TARGETARCH" ]; then
|
||||
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb
|
||||
fi
|
||||
if [ "arm64" = "$TARGETARCH" ]; then
|
||||
if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
|
||||
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb
|
||||
else
|
||||
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb
|
||||
fi
|
||||
fi
|
||||
dpkg -i cuda-keyring_1.1-1_all.deb && \
|
||||
rm -f cuda-keyring_1.1-1_all.deb && \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
|
||||
if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "$TARGETARCH" ]; then
|
||||
apt-get install -y --no-install-recommends \
|
||||
libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libcudnn9-cuda-${CUDA_MAJOR_VERSION} cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
|
||||
fi
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
EOT
|
||||
|
||||
|
||||
# https://github.com/NVIDIA/Isaac-GR00T/issues/343
|
||||
RUN <<EOT bash
|
||||
if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "arm64" ]; then
|
||||
wget https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
|
||||
dpkg -i cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
|
||||
cp /var/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/ && \
|
||||
apt-get update && apt-get -y install cudss cudss-cuda-${CUDA_MAJOR_VERSION} && \
|
||||
wget https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
|
||||
dpkg -i nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
|
||||
cp /var/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/ && \
|
||||
apt-get update && apt-get install -y nvpl
|
||||
fi
|
||||
EOT
|
||||
|
||||
# If we are building with clblas support, we need the libraries for the builds
|
||||
RUN if [ "${BUILD_TYPE}" = "clblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
libclblast-dev && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/* \
|
||||
; fi
|
||||
|
||||
RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
hipblas-dev \
|
||||
hipblaslt-dev \
|
||||
rocblas-dev && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/* && \
|
||||
# I have no idea why, but the ROCM lib packages don't trigger ldconfig after they install, which results in local-ai and others not being able
|
||||
# to locate the libraries. We run ldconfig ourselves to work around this packaging deficiency
|
||||
ldconfig \
|
||||
; fi
|
||||
|
||||
RUN if [ "${BUILD_TYPE}" = "hipblas" ]; then \
|
||||
ln -s /opt/rocm-**/lib/llvm/lib/libomp.so /usr/lib/libomp.so \
|
||||
; fi
|
||||
|
||||
# Install uv as a system package
|
||||
RUN curl -LsSf https://astral.sh/uv/install.sh | UV_INSTALL_DIR=/usr/bin sh
|
||||
ENV PATH="/root/.cargo/bin:${PATH}"
|
||||
# Increase timeout for uv installs behind slow networks
|
||||
ENV UV_HTTP_TIMEOUT=180
|
||||
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
|
||||
|
||||
# Install grpcio-tools (the version in 22.04 is too old)
|
||||
RUN <<EOT bash
|
||||
if [ "${UBUNTU_VERSION}" = "2404" ]; then
|
||||
pip install --break-system-packages --user grpcio-tools==1.71.0 grpcio==1.71.0
|
||||
else
|
||||
pip install grpcio-tools==1.71.0 grpcio==1.71.0
|
||||
fi
|
||||
EOT
|
||||
47
.docker/bases/Dockerfile.rust
Normal file
47
.docker/bases/Dockerfile.rust
Normal file
@@ -0,0 +1,47 @@
|
||||
# Shared Rust base image for the kokoros backend.
|
||||
#
|
||||
# Built once per (ubuntu-version) by .github/workflows/base_images.yml and
|
||||
# pushed to quay.io/go-skynet/ci-cache:base-image-<tag-stem>[-pr<N>]. The
|
||||
# current rust matrix is CPU-only, so this base skips the GPU SDK stanzas;
|
||||
# if a future rust backend needs cublas/rocm/etc., promote this recipe to
|
||||
# mirror Dockerfile.python's GPU stack. See .agents/ci-caching.md.
|
||||
|
||||
ARG BASE_IMAGE=ubuntu:24.04
|
||||
ARG APT_MIRROR=""
|
||||
ARG APT_PORTS_MIRROR=""
|
||||
|
||||
FROM ${BASE_IMAGE}
|
||||
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
ARG TARGETARCH
|
||||
ARG TARGETVARIANT
|
||||
ARG UBUNTU_VERSION=2404
|
||||
ARG APT_MIRROR
|
||||
ARG APT_PORTS_MIRROR
|
||||
|
||||
LABEL org.opencontainers.image.source="https://github.com/mudler/LocalAI"
|
||||
LABEL org.opencontainers.image.description="LocalAI Rust base image"
|
||||
LABEL org.localai.base.lang="rust"
|
||||
|
||||
RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
|
||||
APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
git ccache \
|
||||
ca-certificates \
|
||||
make cmake wget \
|
||||
curl unzip \
|
||||
clang \
|
||||
pkg-config \
|
||||
libssl-dev \
|
||||
espeak-ng libespeak-ng-dev \
|
||||
libsonic-dev libpcaudio-dev \
|
||||
libopus-dev \
|
||||
protobuf-compiler && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install Rust
|
||||
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
|
||||
ENV PATH="/root/.cargo/bin:${PATH}"
|
||||
@@ -1,30 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared compile logic for backend/Dockerfile.ik-llama-cpp.
|
||||
# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages.
|
||||
|
||||
set -euxo pipefail
|
||||
|
||||
export CCACHE_DIR=/root/.ccache
|
||||
ccache --max-size=5G || true
|
||||
ccache -z || true
|
||||
|
||||
export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache"
|
||||
|
||||
if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
|
||||
CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
|
||||
export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
|
||||
echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
|
||||
rm -rf /LocalAI/backend/cpp/ik-llama-cpp-*-build
|
||||
fi
|
||||
|
||||
cd /LocalAI/backend/cpp/ik-llama-cpp
|
||||
|
||||
if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
|
||||
# ARM64 / ROCm: build without x86 SIMD
|
||||
make ik-llama-cpp-fallback
|
||||
else
|
||||
# ik_llama.cpp's IQK kernels require at least AVX2
|
||||
make ik-llama-cpp-avx2
|
||||
fi
|
||||
|
||||
ccache -s || true
|
||||
@@ -1,244 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Single source of truth for builder-base contents.
|
||||
#
|
||||
# Used by:
|
||||
# - backend/Dockerfile.base-grpc-builder (CI prebuilt-base source of truth)
|
||||
# - backend/Dockerfile.llama-cpp (builder-fromsource stage)
|
||||
# - backend/Dockerfile.ik-llama-cpp (builder-fromsource stage)
|
||||
# - backend/Dockerfile.turboquant (builder-fromsource stage)
|
||||
#
|
||||
# All four files invoke this script via
|
||||
# RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
|
||||
# --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
|
||||
# bash /usr/local/sbin/install-base-deps
|
||||
#
|
||||
# so the prebuilt CI base image and the from-source local-dev path are
|
||||
# bit-equivalent by construction.
|
||||
#
|
||||
# Inputs (env, populated from Dockerfile ARG/ENV):
|
||||
# BUILD_TYPE ("cublas"|"l4t"|"hipblas"|"vulkan"|"sycl"|"clblas"|"")
|
||||
# CUDA_MAJOR_VERSION ("12" | "13" | "")
|
||||
# CUDA_MINOR_VERSION ("8" | "0" | "")
|
||||
# TARGETARCH ("amd64" | "arm64")
|
||||
# UBUNTU_VERSION ("2204" | "2404")
|
||||
# SKIP_DRIVERS ("false" | "true")
|
||||
# CMAKE_FROM_SOURCE ("false" | "true")
|
||||
# CMAKE_VERSION ("3.31.10")
|
||||
# GRPC_VERSION ("v1.65.0")
|
||||
# GRPC_MAKEFLAGS ("-j4 -Otarget")
|
||||
# APT_MIRROR / APT_PORTS_MIRROR (optional; consumed by /usr/local/sbin/apt-mirror)
|
||||
# AMDGPU_TARGETS (optional; only relevant for hipblas downstream)
|
||||
#
|
||||
# IMPORTANT: install logic is copied verbatim from the prior in-Dockerfile
|
||||
# RUN blocks. Do not paraphrase apt invocations / version pins / sed line
|
||||
# numbers / deb URLs — the bit-equivalence guarantee depends on it.
|
||||
|
||||
set -eux
|
||||
|
||||
# --- 0. apt mirror rewrite (no-op when APT_MIRROR / APT_PORTS_MIRROR unset) ---
|
||||
if [ -x /usr/local/sbin/apt-mirror ]; then
|
||||
APT_MIRROR="${APT_MIRROR:-}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR:-}" \
|
||||
sh /usr/local/sbin/apt-mirror
|
||||
fi
|
||||
|
||||
export DEBIAN_FRONTEND=noninteractive
|
||||
export MAKEFLAGS="${GRPC_MAKEFLAGS:-}"
|
||||
|
||||
# --- 1. Base apt build deps ---
|
||||
apt-get update
|
||||
apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
ccache git \
|
||||
ca-certificates \
|
||||
make \
|
||||
pkg-config libcurl4-openssl-dev \
|
||||
curl unzip \
|
||||
libssl-dev wget
|
||||
apt-get clean
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# --- 2. Vulkan SDK (BUILD_TYPE=vulkan) ---
|
||||
# NB: this block intentionally installs `cmake` via apt as part of the
|
||||
# Vulkan tooling — must run before the dedicated CMake step below.
|
||||
if [ "${BUILD_TYPE:-}" = "vulkan" ] && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
|
||||
apt-get update
|
||||
apt-get install -y --no-install-recommends \
|
||||
software-properties-common pciutils wget gpg-agent
|
||||
apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
|
||||
libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
|
||||
libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
|
||||
git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
|
||||
ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
|
||||
clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
|
||||
if [ "amd64" = "${TARGETARCH:-}" ]; then
|
||||
wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz"
|
||||
tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz
|
||||
rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz
|
||||
mkdir -p /opt/vulkan-sdk
|
||||
mv 1.4.335.0 /opt/vulkan-sdk/
|
||||
( cd /opt/vulkan-sdk/1.4.335.0 && \
|
||||
./vulkansdk --no-deps --maxjobs \
|
||||
vulkan-loader \
|
||||
vulkan-validationlayers \
|
||||
vulkan-extensionlayer \
|
||||
vulkan-tools \
|
||||
shaderc )
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/
|
||||
rm -rf /opt/vulkan-sdk
|
||||
fi
|
||||
if [ "arm64" = "${TARGETARCH:-}" ]; then
|
||||
mkdir vulkan
|
||||
( cd vulkan && \
|
||||
curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
|
||||
tar -xvf vulkan-sdk.tar.xz && \
|
||||
rm vulkan-sdk.tar.xz && \
|
||||
cd 1.4.335.0 && \
|
||||
cp -rfv aarch64/bin/* /usr/bin/ && \
|
||||
cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
|
||||
cp -rfv aarch64/include/* /usr/include/ && \
|
||||
cp -rfv aarch64/share/* /usr/share/ )
|
||||
rm -rf vulkan
|
||||
fi
|
||||
ldconfig
|
||||
apt-get clean
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
|
||||
# --- 3. CUDA toolkit (BUILD_TYPE=cublas|l4t) ---
|
||||
if { [ "${BUILD_TYPE:-}" = "cublas" ] || [ "${BUILD_TYPE:-}" = "l4t" ]; } && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
|
||||
apt-get update
|
||||
apt-get install -y --no-install-recommends \
|
||||
software-properties-common pciutils
|
||||
if [ "amd64" = "${TARGETARCH:-}" ]; then
|
||||
curl -O "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb"
|
||||
fi
|
||||
if [ "arm64" = "${TARGETARCH:-}" ]; then
|
||||
if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
|
||||
curl -O "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb"
|
||||
else
|
||||
curl -O "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb"
|
||||
fi
|
||||
fi
|
||||
dpkg -i cuda-keyring_1.1-1_all.deb
|
||||
rm -f cuda-keyring_1.1-1_all.deb
|
||||
apt-get update
|
||||
apt-get install -y --no-install-recommends \
|
||||
"cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
|
||||
"libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
|
||||
"libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
|
||||
"libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
|
||||
"libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
|
||||
"libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}"
|
||||
if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "${TARGETARCH:-}" ]; then
|
||||
apt-get install -y --no-install-recommends \
|
||||
"libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
|
||||
"libcudnn9-cuda-${CUDA_MAJOR_VERSION}" \
|
||||
"cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}" \
|
||||
"libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}"
|
||||
fi
|
||||
apt-get clean
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
|
||||
# --- 4. cuDSS / NVPL on arm64 + cublas (legacy JetPack / Tegra) ---
|
||||
# https://github.com/NVIDIA/Isaac-GR00T/issues/343
|
||||
if [ "${BUILD_TYPE:-}" = "cublas" ] && [ "${TARGETARCH:-}" = "arm64" ]; then
|
||||
wget "https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb"
|
||||
dpkg -i "cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb"
|
||||
cp /var/cudss-local-tegra-repo-ubuntu"${UBUNTU_VERSION}"-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/
|
||||
apt-get update
|
||||
apt-get -y install cudss "cudss-cuda-${CUDA_MAJOR_VERSION}"
|
||||
wget "https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb"
|
||||
dpkg -i "nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb"
|
||||
cp /var/nvpl-local-repo-ubuntu"${UBUNTU_VERSION}"-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/
|
||||
apt-get update
|
||||
apt-get install -y nvpl
|
||||
fi
|
||||
|
||||
# --- 5. clBLAS (BUILD_TYPE=clblas) ---
|
||||
# Present in variant Dockerfiles' from-source path but not in master's
|
||||
# Dockerfile.base-grpc-builder. No CI matrix entry currently uses this,
|
||||
# but keep parity so a future BUILD_TYPE=clblas build doesn't drift.
|
||||
if [ "${BUILD_TYPE:-}" = "clblas" ] && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
|
||||
apt-get update
|
||||
apt-get install -y --no-install-recommends \
|
||||
libclblast-dev
|
||||
apt-get clean
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
|
||||
# --- 6. ROCm / HIP build deps (BUILD_TYPE=hipblas) ---
|
||||
if [ "${BUILD_TYPE:-}" = "hipblas" ] && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
|
||||
apt-get update
|
||||
apt-get install -y --no-install-recommends \
|
||||
hipblas-dev \
|
||||
hipblaslt-dev \
|
||||
rocblas-dev
|
||||
apt-get clean
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
# I have no idea why, but the ROCM lib packages don't trigger ldconfig after they install,
|
||||
# which results in local-ai and others not being able to locate the libraries.
|
||||
# We run ldconfig ourselves to work around this packaging deficiency.
|
||||
ldconfig
|
||||
# Log which GPU architectures have rocBLAS kernel support
|
||||
echo "rocBLAS library data architectures:"
|
||||
(ls /opt/rocm*/lib/rocblas/library/Kernels* 2>/dev/null || ls /opt/rocm*/lib64/rocblas/library/Kernels* 2>/dev/null) | grep -oP 'gfx[0-9a-z+-]+' | sort -u || \
|
||||
echo "WARNING: No rocBLAS kernel data found"
|
||||
fi
|
||||
|
||||
echo "TARGETARCH: ${TARGETARCH:-}"
|
||||
|
||||
# --- 7. protoc (always) ---
|
||||
# The version in 22.04 is too old. We will create one as part of installing
|
||||
# the GRPC build below but that will also bring in a newer version of absl
|
||||
# which stablediffusion cannot compile with. This version of protoc is only
|
||||
# here so that we can generate the grpc code for the stablediffusion build.
|
||||
if [ "amd64" = "${TARGETARCH:-}" ]; then
|
||||
curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-x86_64.zip -o protoc.zip
|
||||
unzip -j -d /usr/local/bin protoc.zip bin/protoc
|
||||
rm protoc.zip
|
||||
fi
|
||||
if [ "arm64" = "${TARGETARCH:-}" ]; then
|
||||
curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-aarch_64.zip -o protoc.zip
|
||||
unzip -j -d /usr/local/bin protoc.zip bin/protoc
|
||||
rm protoc.zip
|
||||
fi
|
||||
|
||||
# --- 8. CMake (apt or compiled from source) ---
|
||||
# The version in 22.04 is too old. Vulkan path above already pulled cmake
|
||||
# via apt; the from-source branch here will install over it which is fine.
|
||||
if [ "${CMAKE_FROM_SOURCE:-false}" = "true" ]; then
|
||||
curl -L -s "https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz" -o cmake.tar.gz
|
||||
tar xvf cmake.tar.gz
|
||||
( cd "cmake-${CMAKE_VERSION}" && ./configure && make && make install )
|
||||
else
|
||||
apt-get update
|
||||
apt-get install -y \
|
||||
cmake
|
||||
apt-get clean
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
|
||||
# --- 9. gRPC compile + install at /opt/grpc ---
|
||||
# We install GRPC to a different prefix here so that we can copy in only
|
||||
# the build artifacts later — saves several hundred MB on the final docker
|
||||
# image size vs copying in the entire GRPC source tree and running
|
||||
# `make install` in the target container.
|
||||
#
|
||||
# The TESTONLY abseil sed patch and /opt/grpc prefix are load-bearing —
|
||||
# downstream Dockerfiles `COPY` /opt/grpc to /usr/local (or rely on the
|
||||
# prebuilt base having it at /opt/grpc).
|
||||
mkdir -p /build
|
||||
cd /build
|
||||
git clone --recurse-submodules --jobs 4 -b "${GRPC_VERSION}" --depth 1 --shallow-submodules https://github.com/grpc/grpc
|
||||
mkdir -p /build/grpc/cmake/build
|
||||
cd /build/grpc/cmake/build
|
||||
sed -i "216i\\ TESTONLY" "../../third_party/abseil-cpp/absl/container/CMakeLists.txt"
|
||||
cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX:PATH=/opt/grpc ../..
|
||||
make
|
||||
make install
|
||||
cd /
|
||||
rm -rf /build
|
||||
@@ -1,35 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared compile logic for backend/Dockerfile.llama-cpp.
|
||||
# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages.
|
||||
|
||||
set -euxo pipefail
|
||||
|
||||
export CCACHE_DIR=/root/.ccache
|
||||
ccache --max-size=5G || true
|
||||
ccache -z || true
|
||||
|
||||
export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache"
|
||||
|
||||
if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
|
||||
CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
|
||||
export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
|
||||
echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
|
||||
rm -rf /LocalAI/backend/cpp/llama-cpp-*-build
|
||||
fi
|
||||
|
||||
if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
|
||||
cd /LocalAI/backend/cpp/llama-cpp
|
||||
make llama-cpp-fallback
|
||||
make llama-cpp-grpc
|
||||
make llama-cpp-rpc-server
|
||||
else
|
||||
cd /LocalAI/backend/cpp/llama-cpp
|
||||
make llama-cpp-avx
|
||||
make llama-cpp-avx2
|
||||
make llama-cpp-avx512
|
||||
make llama-cpp-fallback
|
||||
make llama-cpp-grpc
|
||||
make llama-cpp-rpc-server
|
||||
fi
|
||||
|
||||
ccache -s || true
|
||||
@@ -1,35 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared compile logic for backend/Dockerfile.turboquant.
|
||||
# Sourced (via bind mount) from both builder-fromsource and builder-prebuilt stages.
|
||||
|
||||
set -euxo pipefail
|
||||
|
||||
export CCACHE_DIR=/root/.ccache
|
||||
ccache --max-size=5G || true
|
||||
ccache -z || true
|
||||
|
||||
export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_CUDA_COMPILER_LAUNCHER=ccache"
|
||||
|
||||
if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
|
||||
CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
|
||||
export CMAKE_ARGS="${CMAKE_ARGS} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
|
||||
echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
|
||||
rm -rf /LocalAI/backend/cpp/turboquant-*-build
|
||||
fi
|
||||
|
||||
cd /LocalAI/backend/cpp/turboquant
|
||||
|
||||
if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
|
||||
make turboquant-fallback
|
||||
make turboquant-grpc
|
||||
make turboquant-rpc-server
|
||||
else
|
||||
make turboquant-avx
|
||||
make turboquant-avx2
|
||||
make turboquant-avx512
|
||||
make turboquant-fallback
|
||||
make turboquant-grpc
|
||||
make turboquant-rpc-server
|
||||
fi
|
||||
|
||||
ccache -s || true
|
||||
65
.github/actions/free-disk-space/action.yml
vendored
65
.github/actions/free-disk-space/action.yml
vendored
@@ -1,65 +0,0 @@
|
||||
name: 'Free disk space on hosted runners'
|
||||
description: |
|
||||
Aggressively clean GitHub-hosted ubuntu-latest runners to reclaim ~6-10 GB
|
||||
of working space before docker buildx steps. Combines jlumbroso/free-disk-space
|
||||
with explicit apt purges of large packages we never use (dotnet, ghc, mono,
|
||||
android, jdk, ...).
|
||||
|
||||
No-op on self-hosted runners; pass mode=skip to force-disable.
|
||||
|
||||
inputs:
|
||||
mode:
|
||||
description: 'hosted (default — clean) or skip (no-op)'
|
||||
required: false
|
||||
default: 'hosted'
|
||||
|
||||
runs:
|
||||
using: 'composite'
|
||||
steps:
|
||||
- name: Free Disk Space (Ubuntu)
|
||||
if: inputs.mode == 'hosted' && runner.environment == 'github-hosted'
|
||||
uses: jlumbroso/free-disk-space@main
|
||||
with:
|
||||
tool-cache: true
|
||||
android: true
|
||||
dotnet: true
|
||||
haskell: true
|
||||
large-packages: true
|
||||
docker-images: true
|
||||
swap-storage: true
|
||||
|
||||
- name: Release space from worker
|
||||
if: inputs.mode == 'hosted' && runner.environment == 'github-hosted'
|
||||
shell: bash
|
||||
run: |
|
||||
echo "Listing top largest packages"
|
||||
pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
|
||||
head -n 30 <<< "${pkgs}"
|
||||
df -h
|
||||
sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
|
||||
sudo apt-get remove --auto-remove android-sdk-platform-tools snapd || true
|
||||
sudo apt-get purge --auto-remove android-sdk-platform-tools snapd || true
|
||||
sudo rm -rf /usr/local/lib/android
|
||||
sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
|
||||
sudo rm -rf /usr/share/dotnet
|
||||
sudo apt-get remove -y '^mono-.*' || true
|
||||
sudo apt-get remove -y '^ghc-.*' || true
|
||||
sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
|
||||
sudo apt-get remove -y 'php.*' || true
|
||||
sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
|
||||
sudo apt-get remove -y '^google-.*' || true
|
||||
sudo apt-get remove -y azure-cli || true
|
||||
sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
|
||||
sudo apt-get remove -y '^gfortran-.*' || true
|
||||
sudo apt-get remove -y microsoft-edge-stable || true
|
||||
sudo apt-get remove -y firefox || true
|
||||
sudo apt-get remove -y powershell || true
|
||||
sudo apt-get remove -y r-base-core || true
|
||||
sudo apt-get autoremove -y
|
||||
sudo apt-get clean
|
||||
sudo rm -rfv build || true
|
||||
sudo rm -rf /usr/share/dotnet || true
|
||||
sudo rm -rf /opt/ghc || true
|
||||
sudo rm -rf "/usr/local/share/boost" || true
|
||||
sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
|
||||
df -h
|
||||
59
.github/actions/setup-build-disk/action.yml
vendored
59
.github/actions/setup-build-disk/action.yml
vendored
@@ -1,59 +0,0 @@
|
||||
name: 'Set up build disk on hosted runners'
|
||||
description: |
|
||||
Relocate Docker's data-root to /mnt (which has ~75 GB free, vs ~20 GB
|
||||
on / after free-disk-space). Combined with the apt cleanup, gives
|
||||
~100 GB working space for buildx — enough for ROCm dev image + vLLM
|
||||
torch install + flash-attn build.
|
||||
|
||||
No-op on:
|
||||
- self-hosted runners (no /mnt expectation)
|
||||
- non-X64 runners (verify /mnt shape on ubuntu-24.04-arm separately
|
||||
before enabling there — see Task 3.2 in the migration plan)
|
||||
- mode=skip (force-disable from caller)
|
||||
|
||||
Must run after free-disk-space (which removes large packages — would
|
||||
fail mid-uninstall if Docker were stopped) and before any Docker
|
||||
operation (setup-qemu, setup-buildx, login, build) so the relocated
|
||||
data-root catches all subsequent docker activity.
|
||||
|
||||
inputs:
|
||||
mode:
|
||||
description: 'auto (default — relocate on hosted X64 only) or skip'
|
||||
required: false
|
||||
default: 'auto'
|
||||
|
||||
runs:
|
||||
using: 'composite'
|
||||
steps:
|
||||
- name: Relocate Docker data-root to /mnt
|
||||
if: inputs.mode == 'auto' && runner.environment == 'github-hosted' && runner.arch == 'X64'
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
echo "Before relocation:"
|
||||
df -h / /mnt || true
|
||||
sudo systemctl stop docker docker.socket
|
||||
sudo mkdir -p /mnt/docker-data /mnt/docker-tmp
|
||||
# buildx CLI runs as the unprivileged runner user and creates
|
||||
# config dirs under TMPDIR before binding them into the buildkit
|
||||
# container. /mnt is owned by root by default; mirror /tmp's
|
||||
# 1777 (world-writable + sticky) so non-root processes can write.
|
||||
sudo chmod 1777 /mnt/docker-tmp
|
||||
if [ -d /var/lib/docker ] && [ ! -L /var/lib/docker ]; then
|
||||
sudo rsync -a /var/lib/docker/ /mnt/docker-data/
|
||||
sudo rm -rf /var/lib/docker
|
||||
sudo ln -s /mnt/docker-data /var/lib/docker
|
||||
fi
|
||||
# daemon.json may not exist; merge data-root in or create minimal.
|
||||
if [ -f /etc/docker/daemon.json ]; then
|
||||
sudo jq '."data-root" = "/mnt/docker-data"' /etc/docker/daemon.json | sudo tee /etc/docker/daemon.json.new >/dev/null
|
||||
sudo mv /etc/docker/daemon.json.new /etc/docker/daemon.json
|
||||
else
|
||||
echo '{"data-root":"/mnt/docker-data"}' | sudo tee /etc/docker/daemon.json
|
||||
fi
|
||||
sudo systemctl start docker
|
||||
# Make TMPDIR persist for subsequent steps in the same job.
|
||||
echo "TMPDIR=/mnt/docker-tmp" >> "$GITHUB_ENV"
|
||||
echo "After relocation:"
|
||||
df -h / /mnt
|
||||
docker info | grep -i 'docker root dir' || true
|
||||
3164
.github/backend-matrix.yaml
vendored
Normal file
3164
.github/backend-matrix.yaml
vendored
Normal file
File diff suppressed because it is too large
Load Diff
3852
.github/backend-matrix.yml
vendored
3852
.github/backend-matrix.yml
vendored
File diff suppressed because it is too large
Load Diff
280
.github/workflows/backend.yml
vendored
280
.github/workflows/backend.yml
vendored
@@ -7,75 +7,61 @@ on:
|
||||
- master
|
||||
tags:
|
||||
- '*'
|
||||
schedule:
|
||||
# Weekly full-matrix rebuild to pick up upstream Python wheel updates
|
||||
# (torch, transformers, vllm, ...) which most backends pull unpinned.
|
||||
# The DEPS_REFRESH build-arg in backend_build.yml busts the install
|
||||
# layer cache on a new ISO week, but only fires when the build runs.
|
||||
# Path filtering on commit-driven pushes (scripts/changed-backends.js)
|
||||
# skips untouched backends, so without this cron those images would
|
||||
# drift on stale wheels indefinitely. C++/Go backends with pinned
|
||||
# deps cache-hit and finish fast.
|
||||
#
|
||||
# Schedule events have no event.ref / event.before, so the script's
|
||||
# changedFiles==null fallback emits the full matrix automatically —
|
||||
# no script changes needed.
|
||||
- cron: '0 6 * * 0' # Sundays 06:00 UTC
|
||||
workflow_dispatch:
|
||||
|
||||
concurrency:
|
||||
group: ci-backends-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
group: ci-backends-${{ github.head_ref || github.ref }}-${{ github.repository }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
generate-matrix:
|
||||
derive-bases:
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
matrix-singlearch: ${{ steps.set-matrix.outputs['matrix-singlearch'] }}
|
||||
matrix-multiarch: ${{ steps.set-matrix.outputs['matrix-multiarch'] }}
|
||||
matrix-darwin: ${{ steps.set-matrix.outputs['matrix-darwin'] }}
|
||||
merge-matrix-multiarch: ${{ steps.set-matrix.outputs['merge-matrix-multiarch'] }}
|
||||
merge-matrix-singlearch: ${{ steps.set-matrix.outputs['merge-matrix-singlearch'] }}
|
||||
has-backends-singlearch: ${{ steps.set-matrix.outputs['has-backends-singlearch'] }}
|
||||
has-backends-multiarch: ${{ steps.set-matrix.outputs['has-backends-multiarch'] }}
|
||||
has-backends-darwin: ${{ steps.set-matrix.outputs['has-backends-darwin'] }}
|
||||
has-merges-multiarch: ${{ steps.set-matrix.outputs['has-merges-multiarch'] }}
|
||||
has-merges-singlearch: ${{ steps.set-matrix.outputs['has-merges-singlearch'] }}
|
||||
matrix: ${{ steps.derive.outputs.matrix }}
|
||||
matrix-darwin: ${{ steps.derive.outputs.matrix-darwin }}
|
||||
bases-matrix: ${{ steps.derive.outputs.bases-matrix }}
|
||||
has-backends: ${{ steps.derive.outputs.has-backends }}
|
||||
has-backends-darwin: ${{ steps.derive.outputs.has-backends-darwin }}
|
||||
has-bases: ${{ steps.derive.outputs.has-bases }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Setup Bun
|
||||
uses: oven-sh/setup-bun@v2
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
- uses: actions/checkout@v6
|
||||
- uses: oven-sh/setup-bun@v2
|
||||
- run: |
|
||||
bun add js-yaml
|
||||
bun add @octokit/core
|
||||
|
||||
# Filter the backend matrix from .github/backend-matrix.yml against the
|
||||
# files changed by this push. Tag pushes set FORCE_ALL=true so the script
|
||||
# falls through to the full matrix (releases must rebuild everything).
|
||||
# The script splits the linux matrix into single-arch and multi-arch
|
||||
# groups so backend-merge-jobs can `needs:` only the multi-arch one —
|
||||
# see the comment block above the merge job for context.
|
||||
- name: Filter matrix for changed backends
|
||||
id: set-matrix
|
||||
- id: derive
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
GITHUB_EVENT_PATH: ${{ github.event_path }}
|
||||
FORCE_ALL: ${{ startsWith(github.ref, 'refs/tags/') && 'true' || 'false' }}
|
||||
run: bun run scripts/changed-backends.js
|
||||
|
||||
# Multi-arch backends — entries with a `platform-tag` set, paired with a
|
||||
# sibling entry sharing the same `tag-suffix` (one amd64 leg, one arm64
|
||||
# leg). Their digests are the inputs to backend-merge-jobs, so they're in
|
||||
# their own matrix to bound how long the merge waits before quay GCs the
|
||||
# untagged digests.
|
||||
backend-jobs-multiarch:
|
||||
needs: generate-matrix
|
||||
if: needs.generate-matrix.outputs['has-backends-multiarch'] == 'true'
|
||||
build-bases:
|
||||
needs: derive-bases
|
||||
if: needs.derive-bases.outputs.has-bases == 'true'
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJSON(needs.derive-bases.outputs.bases-matrix) }}
|
||||
uses: ./.github/workflows/base_images.yml
|
||||
with:
|
||||
lang: ${{ matrix.lang }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
tag-stem: ${{ matrix.tag-stem }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
secrets:
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
|
||||
backend-jobs:
|
||||
if: |
|
||||
always() && github.repository == 'mudler/LocalAI' &&
|
||||
(needs.build-bases.result == 'success' || needs.build-bases.result == 'skipped')
|
||||
needs: [derive-bases, build-bases]
|
||||
uses: ./.github/workflows/backend_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
@@ -84,9 +70,7 @@ jobs:
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
platform-tag: ${{ matrix.platform-tag || '' }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
builder-base-image: ${{ matrix.builder-base-image || '' }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
backend: ${{ matrix.backend }}
|
||||
dockerfile: ${{ matrix.dockerfile }}
|
||||
@@ -94,6 +78,10 @@ jobs:
|
||||
context: ${{ matrix.context }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
|
||||
# Set by scripts/changed-backends.js for langs that have a
|
||||
# .docker/bases/Dockerfile.<lang> recipe; '' otherwise (those run
|
||||
# the inline bootstrap in their own Dockerfile).
|
||||
base-image-prebuilt: ${{ matrix.base-image-prebuilt || '' }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
@@ -101,97 +89,18 @@ jobs:
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
max-parallel: 8
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-multiarch']) }}
|
||||
|
||||
# Single-arch backends — no `platform-tag`. Heavy ones (CUDA, ROCm, Intel
|
||||
# oneAPI, vLLM/sglang) live here. Independent of the merge job: they can
|
||||
# take their full ~6h cold without blocking manifest assembly for the
|
||||
# multi-arch backends whose per-arch digests would otherwise sit untagged
|
||||
# on quay long enough to be GC'd.
|
||||
backend-jobs-singlearch:
|
||||
needs: generate-matrix
|
||||
if: needs.generate-matrix.outputs['has-backends-singlearch'] == 'true'
|
||||
uses: ./.github/workflows/backend_build.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
platform-tag: ${{ matrix.platform-tag || '' }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
builder-base-image: ${{ matrix.builder-base-image || '' }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
backend: ${{ matrix.backend }}
|
||||
dockerfile: ${{ matrix.dockerfile }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
context: ${{ matrix.context }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
max-parallel: 8
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch']) }}
|
||||
|
||||
# Apply tags to per-arch digests via `imagetools create`. Split into two
|
||||
# jobs that mirror the build split so each merge waits ONLY on its
|
||||
# corresponding build matrix:
|
||||
#
|
||||
# - backend-merge-jobs-multiarch needs backend-jobs-multiarch (~2-3h)
|
||||
# - backend-merge-jobs-singlearch needs backend-jobs-singlearch (up to ~6h)
|
||||
#
|
||||
# If a single shared merge job depended on both, slow CUDA singlearch
|
||||
# builds would block multiarch merges long enough for quay's GC to reap
|
||||
# the multiarch per-arch digests (the bug fixed by PR #9746). Singletons
|
||||
# also need a merge step because backend_build.yml pushes by canonical
|
||||
# digest only — no tags are applied at build time.
|
||||
backend-merge-jobs-multiarch:
|
||||
needs: [generate-matrix, backend-jobs-multiarch]
|
||||
if: needs.generate-matrix.outputs['has-merges-multiarch'] == 'true'
|
||||
uses: ./.github/workflows/backend_merge.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-multiarch']) }}
|
||||
|
||||
backend-merge-jobs-singlearch:
|
||||
needs: [generate-matrix, backend-jobs-singlearch]
|
||||
if: needs.generate-matrix.outputs['has-merges-singlearch'] == 'true'
|
||||
uses: ./.github/workflows/backend_merge.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch']) }}
|
||||
|
||||
matrix: ${{ fromJSON(needs.derive-bases.outputs.matrix) }}
|
||||
backend-jobs-darwin:
|
||||
needs: generate-matrix
|
||||
if: needs.generate-matrix.outputs.has-backends-darwin == 'true'
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
needs: derive-bases
|
||||
uses: ./.github/workflows/backend_build_darwin.yml
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJSON(needs.derive-bases.outputs.matrix-darwin) }}
|
||||
with:
|
||||
backend: ${{ matrix.backend }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
go-version: "1.25.x"
|
||||
go-version: "1.24.x"
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
lang: ${{ matrix.lang || 'python' }}
|
||||
use-pip: ${{ matrix.backend == 'diffusers' }}
|
||||
@@ -201,6 +110,89 @@ jobs:
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
llama-cpp-darwin:
|
||||
runs-on: macos-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs.matrix-darwin) }}
|
||||
matrix:
|
||||
go-version: ['1.25.x']
|
||||
steps:
|
||||
- name: Clone
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
submodules: true
|
||||
- name: Setup Go ${{ matrix.go-version }}
|
||||
uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version: ${{ matrix.go-version }}
|
||||
cache: false
|
||||
# You can test your matrix by printing the current Go version
|
||||
- name: Display Go version
|
||||
run: go version
|
||||
- name: Dependencies
|
||||
run: |
|
||||
brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm
|
||||
- name: Build llama-cpp-darwin
|
||||
run: |
|
||||
make protogen-go
|
||||
make backends/llama-cpp-darwin
|
||||
- name: Upload llama-cpp.tar
|
||||
uses: actions/upload-artifact@v7
|
||||
with:
|
||||
name: llama-cpp-tar
|
||||
path: backend-images/llama-cpp.tar
|
||||
llama-cpp-darwin-publish:
|
||||
needs: llama-cpp-darwin
|
||||
if: github.event_name != 'pull_request'
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Download llama-cpp.tar
|
||||
uses: actions/download-artifact@v8
|
||||
with:
|
||||
name: llama-cpp-tar
|
||||
path: .
|
||||
- name: Install crane
|
||||
run: |
|
||||
curl -L https://github.com/google/go-containerregistry/releases/latest/download/go-containerregistry_Linux_x86_64.tar.gz | tar -xz
|
||||
sudo mv crane /usr/local/bin/
|
||||
- name: Log in to DockerHub
|
||||
run: |
|
||||
echo "${{ secrets.DOCKERHUB_PASSWORD }}" | crane auth login docker.io -u "${{ secrets.DOCKERHUB_USERNAME }}" --password-stdin
|
||||
- name: Log in to quay.io
|
||||
run: |
|
||||
echo "${{ secrets.LOCALAI_REGISTRY_PASSWORD }}" | crane auth login quay.io -u "${{ secrets.LOCALAI_REGISTRY_USERNAME }}" --password-stdin
|
||||
- name: Docker meta
|
||||
id: meta
|
||||
uses: docker/metadata-action@v6
|
||||
with:
|
||||
images: |
|
||||
localai/localai-backends
|
||||
tags: |
|
||||
type=ref,event=branch
|
||||
type=semver,pattern={{raw}}
|
||||
type=sha
|
||||
flavor: |
|
||||
latest=auto
|
||||
suffix=-metal-darwin-arm64-llama-cpp,onlatest=true
|
||||
- name: Docker meta
|
||||
id: quaymeta
|
||||
uses: docker/metadata-action@v6
|
||||
with:
|
||||
images: |
|
||||
quay.io/go-skynet/local-ai-backends
|
||||
tags: |
|
||||
type=ref,event=branch
|
||||
type=semver,pattern={{raw}}
|
||||
type=sha
|
||||
flavor: |
|
||||
latest=auto
|
||||
suffix=-metal-darwin-arm64-llama-cpp,onlatest=true
|
||||
- name: Push Docker image (DockerHub)
|
||||
run: |
|
||||
for tag in $(echo "${{ steps.meta.outputs.tags }}" | tr ',' '\n'); do
|
||||
crane push llama-cpp.tar $tag
|
||||
done
|
||||
- name: Push Docker image (Quay)
|
||||
run: |
|
||||
for tag in $(echo "${{ steps.quaymeta.outputs.tags }}" | tr ',' '\n'); do
|
||||
crane push llama-cpp.tar $tag
|
||||
done
|
||||
|
||||
137
.github/workflows/backend_build.yml
vendored
137
.github/workflows/backend_build.yml
vendored
@@ -24,17 +24,6 @@ on:
|
||||
description: 'Platforms'
|
||||
default: ''
|
||||
type: string
|
||||
platform-tag:
|
||||
description: |
|
||||
Short tag identifying the platform leg, e.g. "amd64" or "arm64".
|
||||
Used to scope the per-arch registry cache and the digest artifact name.
|
||||
Required for split-and-merge multi-arch builds; pass "amd64" for
|
||||
single-arch amd64 builds too. Optional (default '') during the
|
||||
migration to per-arch matrix expansion; will be flipped to
|
||||
required: true in Phase 6 once all callers pass an explicit value.
|
||||
required: false
|
||||
default: ''
|
||||
type: string
|
||||
tag-latest:
|
||||
description: 'Tag latest'
|
||||
default: ''
|
||||
@@ -74,12 +63,13 @@ on:
|
||||
required: false
|
||||
default: ''
|
||||
type: string
|
||||
builder-base-image:
|
||||
base-image-prebuilt:
|
||||
description: |
|
||||
Pre-built builder base image (e.g. quay.io/go-skynet/ci-cache:base-grpc-cuda-13-amd64).
|
||||
When set, the variant Dockerfile uses its `builder-prebuilt` stage which FROMs this
|
||||
image directly instead of running its own gRPC stage + apt installs. Empty for
|
||||
backends whose Dockerfile doesn't support a prebuilt base.
|
||||
Optional reference to a prebuilt accel/lang base image
|
||||
(quay.io/go-skynet/ci-cache:base-image-<stem>[-pr<N>]). When
|
||||
set, the backend Dockerfile FROMs this image instead of running
|
||||
an inline bootstrap. See .github/workflows/base_images.yml and
|
||||
.agents/ci-caching.md.
|
||||
required: false
|
||||
default: ''
|
||||
type: string
|
||||
@@ -109,13 +99,63 @@ jobs:
|
||||
id: apt_mirror
|
||||
uses: ./.github/actions/configure-apt-mirror
|
||||
|
||||
- name: Free disk space
|
||||
uses: ./.github/actions/free-disk-space
|
||||
- name: Free Disk Space (Ubuntu)
|
||||
if: inputs.runs-on == 'ubuntu-latest'
|
||||
uses: jlumbroso/free-disk-space@main
|
||||
with:
|
||||
mode: ${{ inputs.runs-on == 'ubuntu-latest' && 'hosted' || 'skip' }}
|
||||
# this might remove tools that are actually needed,
|
||||
# if set to "true" but frees about 6 GB
|
||||
tool-cache: true
|
||||
# all of these default to true, but feel free to set to
|
||||
# "false" if necessary for your workflow
|
||||
android: true
|
||||
dotnet: true
|
||||
haskell: true
|
||||
large-packages: true
|
||||
docker-images: true
|
||||
swap-storage: true
|
||||
|
||||
- name: Set up build disk
|
||||
uses: ./.github/actions/setup-build-disk
|
||||
- name: Release space from worker
|
||||
if: inputs.runs-on == 'ubuntu-latest'
|
||||
run: |
|
||||
echo "Listing top largest packages"
|
||||
pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
|
||||
head -n 30 <<< "${pkgs}"
|
||||
echo
|
||||
df -h
|
||||
echo
|
||||
sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
|
||||
sudo apt-get remove --auto-remove android-sdk-platform-tools snapd || true
|
||||
sudo apt-get purge --auto-remove android-sdk-platform-tools snapd || true
|
||||
sudo rm -rf /usr/local/lib/android
|
||||
sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
|
||||
sudo rm -rf /usr/share/dotnet
|
||||
sudo apt-get remove -y '^mono-.*' || true
|
||||
sudo apt-get remove -y '^ghc-.*' || true
|
||||
sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
|
||||
sudo apt-get remove -y 'php.*' || true
|
||||
sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
|
||||
sudo apt-get remove -y '^google-.*' || true
|
||||
sudo apt-get remove -y azure-cli || true
|
||||
sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
|
||||
sudo apt-get remove -y '^gfortran-.*' || true
|
||||
sudo apt-get remove -y microsoft-edge-stable || true
|
||||
sudo apt-get remove -y firefox || true
|
||||
sudo apt-get remove -y powershell || true
|
||||
sudo apt-get remove -y r-base-core || true
|
||||
sudo apt-get autoremove -y
|
||||
sudo apt-get clean
|
||||
echo
|
||||
echo "Listing top largest packages"
|
||||
pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
|
||||
head -n 30 <<< "${pkgs}"
|
||||
echo
|
||||
sudo rm -rfv build || true
|
||||
sudo rm -rf /usr/share/dotnet || true
|
||||
sudo rm -rf /opt/ghc || true
|
||||
sudo rm -rf "/usr/local/share/boost" || true
|
||||
sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
|
||||
df -h
|
||||
|
||||
- name: Docker meta
|
||||
id: meta
|
||||
@@ -181,8 +221,7 @@ jobs:
|
||||
id: deps_refresh
|
||||
run: echo "key=$(date -u +%Y-W%V)" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Build and push by digest
|
||||
id: build
|
||||
- name: Build and push
|
||||
uses: docker/build-push-action@v7
|
||||
if: github.event_name != 'pull_request'
|
||||
with:
|
||||
@@ -199,52 +238,17 @@ jobs:
|
||||
APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
|
||||
APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
|
||||
DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
|
||||
BUILDER_BASE_IMAGE=${{ inputs.builder-base-image }}
|
||||
BUILDER_TARGET=${{ inputs.builder-base-image != '' && 'builder-prebuilt' || 'builder-fromsource' }}
|
||||
BASE_IMAGE_PREBUILT=${{ inputs.base-image-prebuilt }}
|
||||
context: ${{ inputs.context }}
|
||||
file: ${{ inputs.dockerfile }}
|
||||
cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
|
||||
cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }},mode=max,ignore-error=true
|
||||
cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}
|
||||
cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }},mode=max,ignore-error=true
|
||||
platforms: ${{ inputs.platforms }}
|
||||
outputs: |
|
||||
type=image,name=quay.io/go-skynet/local-ai-backends,push-by-digest=true,name-canonical=true,push=true
|
||||
type=image,name=localai/localai-backends,push-by-digest=true,name-canonical=true,push=true
|
||||
# Disable provenance: with mode=max (the default for push:true)
|
||||
# buildx bundles a per-registry attestation manifest into each
|
||||
# registry's manifest list, which makes the resulting list digest
|
||||
# diverge across registries. steps.build.outputs.digest then
|
||||
# only matches one of them, and the merge job's
|
||||
# `imagetools create <reg>@sha256:<digest>` lookup fails on the
|
||||
# other. Disabling provenance keeps the digest content-only and
|
||||
# identical across both registries — required for digest-based
|
||||
# cross-registry merge.
|
||||
provenance: false
|
||||
push: ${{ github.event_name != 'pull_request' }}
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
|
||||
- name: Export digest
|
||||
if: github.event_name != 'pull_request'
|
||||
run: |
|
||||
mkdir -p /tmp/digests
|
||||
digest="${{ steps.build.outputs.digest }}"
|
||||
touch "/tmp/digests/${digest#sha256:}"
|
||||
|
||||
# Artifact name uses a `--` separator between tag-suffix and platform-tag
|
||||
# to avoid prefix collisions during the merge job's pattern-based download.
|
||||
# Tag-suffixes are not prefix-disjoint (e.g. -gpu-nvidia-cuda-12-vllm is a
|
||||
# prefix of -gpu-nvidia-cuda-12-vllm-omni); a single `-` separator plus the
|
||||
# merge-side `digests<tag-suffix>-*` glob would let one merge over-match
|
||||
# the other backend's artifacts. The `-single` placeholder for empty
|
||||
# platform-tag (single-arch entries) keeps the artifact name non-trailing.
|
||||
- name: Upload digest artifact
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: digests${{ inputs.tag-suffix }}--${{ inputs.platform-tag || 'single' }}
|
||||
path: /tmp/digests/*
|
||||
if-no-files-found: error
|
||||
retention-days: 1
|
||||
|
||||
- name: Build (PR)
|
||||
- name: Build and push (PR)
|
||||
uses: docker/build-push-action@v7
|
||||
if: github.event_name == 'pull_request'
|
||||
with:
|
||||
@@ -261,11 +265,10 @@ jobs:
|
||||
APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
|
||||
APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
|
||||
DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
|
||||
BUILDER_BASE_IMAGE=${{ inputs.builder-base-image }}
|
||||
BUILDER_TARGET=${{ inputs.builder-base-image != '' && 'builder-prebuilt' || 'builder-fromsource' }}
|
||||
BASE_IMAGE_PREBUILT=${{ inputs.base-image-prebuilt }}
|
||||
context: ${{ inputs.context }}
|
||||
file: ${{ inputs.dockerfile }}
|
||||
cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
|
||||
cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache${{ inputs.tag-suffix }}
|
||||
platforms: ${{ inputs.platforms }}
|
||||
push: ${{ env.quay_username != '' }}
|
||||
tags: ${{ steps.meta_pull_request.outputs.tags }}
|
||||
|
||||
44
.github/workflows/backend_build_darwin.yml
vendored
44
.github/workflows/backend_build_darwin.yml
vendored
@@ -93,11 +93,6 @@ jobs:
|
||||
/opt/homebrew/Cellar/libomp
|
||||
/opt/homebrew/Cellar/llvm
|
||||
/opt/homebrew/Cellar/ccache
|
||||
/opt/homebrew/Cellar/blake3
|
||||
/opt/homebrew/Cellar/fmt
|
||||
/opt/homebrew/Cellar/hiredis
|
||||
/opt/homebrew/Cellar/xxhash
|
||||
/opt/homebrew/Cellar/zstd
|
||||
key: brew-${{ runner.os }}-${{ runner.arch }}-v1-${{ hashFiles('.github/workflows/backend_build_darwin.yml') }}
|
||||
|
||||
- name: Dependencies
|
||||
@@ -105,23 +100,7 @@ jobs:
|
||||
# ccache is always installed (used by the llama-cpp variant build) so
|
||||
# the brew cache content stays stable across every backend in the
|
||||
# matrix — they all share one cache key.
|
||||
# blake3, fmt, hiredis, xxhash, zstd are ccache's runtime dylib deps.
|
||||
# Without explicitly installing them, a brew cache-hit run restores
|
||||
# ccache's Cellar dir but skips installing those transitive deps,
|
||||
# and ccache fails at runtime with `dyld: Library not loaded`.
|
||||
brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm ccache blake3 fmt hiredis xxhash zstd
|
||||
# Force-reinstall ccache so brew re-validates its full runtime-dep
|
||||
# closure on every run. This is the durable fix: when the upstream
|
||||
# ccache formula gains a new transitive dep (as it has multiple times
|
||||
# already), we don't have to chase missing dylibs one at a time.
|
||||
# The downloads cache makes the reinstall fast (~5s on a hit).
|
||||
brew reinstall ccache
|
||||
# The brew cache restores the Cellar dirs but NOT the bin symlinks
|
||||
# at /opt/homebrew/bin/*. brew install above sees the Cellar present
|
||||
# and decides "already installed" without re-linking, so on a cache-
|
||||
# hit run the formulas aren't on PATH. Force-link them; --overwrite
|
||||
# tolerates pre-existing symlinks from earlier installs.
|
||||
brew link --overwrite protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm ccache blake3 fmt hiredis xxhash zstd 2>/dev/null || true
|
||||
brew install protobuf grpc make protoc-gen-go protoc-gen-go-grpc libomp llvm ccache
|
||||
|
||||
- name: Save Homebrew cache
|
||||
if: github.event_name != 'pull_request' && steps.brew-cache.outputs.cache-hit != 'true'
|
||||
@@ -136,11 +115,6 @@ jobs:
|
||||
/opt/homebrew/Cellar/libomp
|
||||
/opt/homebrew/Cellar/llvm
|
||||
/opt/homebrew/Cellar/ccache
|
||||
/opt/homebrew/Cellar/blake3
|
||||
/opt/homebrew/Cellar/fmt
|
||||
/opt/homebrew/Cellar/hiredis
|
||||
/opt/homebrew/Cellar/xxhash
|
||||
/opt/homebrew/Cellar/zstd
|
||||
key: brew-${{ runner.os }}-${{ runner.arch }}-v1-${{ hashFiles('.github/workflows/backend_build_darwin.yml') }}
|
||||
|
||||
# ---- ccache for llama.cpp CMake builds ----
|
||||
@@ -201,23 +175,7 @@ jobs:
|
||||
restore-keys: |
|
||||
pyenv-darwin-${{ inputs.backend }}-
|
||||
|
||||
# llama-cpp on Darwin uses a bespoke build script (scripts/build/llama-cpp-darwin.sh)
|
||||
# that compiles three CMake variants from backend/cpp/llama-cpp and bundles dylibs
|
||||
# via otool — it doesn't fit the build-darwin-go-backend / build-darwin-python-backend
|
||||
# mold. Drive it via its dedicated `backends/llama-cpp-darwin` make target instead.
|
||||
- name: Build ${{ inputs.backend }}-darwin (llama-cpp)
|
||||
if: inputs.backend == 'llama-cpp'
|
||||
run: |
|
||||
make protogen-go
|
||||
make backends/llama-cpp-darwin
|
||||
|
||||
- name: Build ds4 backend (Darwin Metal)
|
||||
if: inputs.backend == 'ds4'
|
||||
run: |
|
||||
make backends/ds4-darwin
|
||||
|
||||
- name: Build ${{ inputs.backend }}-darwin
|
||||
if: inputs.backend != 'llama-cpp' && inputs.backend != 'ds4'
|
||||
run: |
|
||||
make protogen-go
|
||||
BACKEND=${{ inputs.backend }} BUILD_TYPE=${{ inputs.build-type }} USE_PIP=${{ inputs.use-pip }} make build-darwin-${{ inputs.lang }}-backend
|
||||
|
||||
137
.github/workflows/backend_merge.yml
vendored
137
.github/workflows/backend_merge.yml
vendored
@@ -1,137 +0,0 @@
|
||||
---
|
||||
name: 'merge backend manifest list (reusable)'
|
||||
|
||||
# Reusable workflow that joins per-arch digest artifacts (uploaded by
|
||||
# backend_build.yml when called with platform-tag) into a single tagged
|
||||
# multi-arch manifest list. Called once per backend by backend.yml after
|
||||
# both per-arch build jobs succeed.
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
tag-latest:
|
||||
description: 'Whether the manifest list should also be tagged latest (auto/false/true)'
|
||||
required: false
|
||||
type: string
|
||||
default: ''
|
||||
tag-suffix:
|
||||
description: 'Backend tag suffix (e.g. -cpu-faster-whisper). Used to compute the artifact pattern and the final tag suffix.'
|
||||
required: true
|
||||
type: string
|
||||
secrets:
|
||||
dockerUsername:
|
||||
required: false
|
||||
dockerPassword:
|
||||
required: false
|
||||
quayUsername:
|
||||
required: true
|
||||
quayPassword:
|
||||
required: true
|
||||
|
||||
jobs:
|
||||
merge:
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
quay_username: ${{ secrets.quayUsername }}
|
||||
steps:
|
||||
# `--` separator anchors the glob so we don't over-match sibling
|
||||
# backends whose tag-suffix happens to be a prefix of ours
|
||||
# (e.g. -cpu-vllm vs -cpu-vllm-omni). Must stay in sync with the
|
||||
# upload-artifact name in backend_build.yml.
|
||||
- name: Download digests
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
pattern: digests${{ inputs.tag-suffix }}--*
|
||||
merge-multiple: true
|
||||
path: /tmp/digests
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@master
|
||||
|
||||
- name: Login to DockerHub
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: docker/login-action@v4
|
||||
with:
|
||||
username: ${{ secrets.dockerUsername }}
|
||||
password: ${{ secrets.dockerPassword }}
|
||||
|
||||
- name: Login to Quay.io
|
||||
if: ${{ env.quay_username != '' }}
|
||||
uses: docker/login-action@v4
|
||||
with:
|
||||
registry: quay.io
|
||||
username: ${{ secrets.quayUsername }}
|
||||
password: ${{ secrets.quayPassword }}
|
||||
|
||||
- name: Docker meta
|
||||
id: meta
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: docker/metadata-action@v6
|
||||
with:
|
||||
images: |
|
||||
quay.io/go-skynet/local-ai-backends
|
||||
localai/localai-backends
|
||||
tags: |
|
||||
type=ref,event=branch
|
||||
type=semver,pattern={{raw}}
|
||||
type=sha
|
||||
flavor: |
|
||||
latest=${{ inputs.tag-latest }}
|
||||
suffix=${{ inputs.tag-suffix }},onlatest=true
|
||||
|
||||
- name: Create manifest list and push (quay)
|
||||
if: github.event_name != 'pull_request'
|
||||
working-directory: /tmp/digests
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tags=$(jq -cr '
|
||||
.tags
|
||||
| map(select(startswith("quay.io/")))
|
||||
| map("-t " + .)
|
||||
| join(" ")
|
||||
' <<< "$DOCKER_METADATA_OUTPUT_JSON")
|
||||
if [ -z "$tags" ]; then
|
||||
echo "No quay.io tags from docker/metadata-action; skipping quay merge"
|
||||
else
|
||||
# shellcheck disable=SC2086
|
||||
docker buildx imagetools create $tags \
|
||||
$(printf 'quay.io/go-skynet/local-ai-backends@sha256:%s ' *)
|
||||
fi
|
||||
|
||||
- name: Create manifest list and push (dockerhub)
|
||||
if: github.event_name != 'pull_request'
|
||||
working-directory: /tmp/digests
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tags=$(jq -cr '
|
||||
.tags
|
||||
| map(select(startswith("localai/")))
|
||||
| map("-t " + .)
|
||||
| join(" ")
|
||||
' <<< "$DOCKER_METADATA_OUTPUT_JSON")
|
||||
if [ -z "$tags" ]; then
|
||||
echo "No dockerhub tags from docker/metadata-action; skipping dockerhub merge"
|
||||
else
|
||||
# shellcheck disable=SC2086
|
||||
docker buildx imagetools create $tags \
|
||||
$(printf 'localai/localai-backends@sha256:%s ' *)
|
||||
fi
|
||||
|
||||
- name: Inspect manifest
|
||||
if: github.event_name != 'pull_request'
|
||||
run: |
|
||||
set -euo pipefail
|
||||
first_tag=$(jq -cr '.tags[0]' <<< "$DOCKER_METADATA_OUTPUT_JSON")
|
||||
if [ -n "$first_tag" ] && [ "$first_tag" != "null" ]; then
|
||||
docker buildx imagetools inspect "$first_tag"
|
||||
fi
|
||||
|
||||
- name: Job summary
|
||||
if: github.event_name != 'pull_request'
|
||||
run: |
|
||||
set -euo pipefail
|
||||
echo "Merged manifest tags:" >> "$GITHUB_STEP_SUMMARY"
|
||||
jq -r '.tags[]' <<< "$DOCKER_METADATA_OUTPUT_JSON" | sed 's/^/- /' >> "$GITHUB_STEP_SUMMARY"
|
||||
echo >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "Per-arch digests:" >> "$GITHUB_STEP_SUMMARY"
|
||||
ls -1 /tmp/digests | sed 's/^/- sha256:/' >> "$GITHUB_STEP_SUMMARY"
|
||||
145
.github/workflows/backend_pr.yml
vendored
145
.github/workflows/backend_pr.yml
vendored
@@ -4,23 +4,19 @@ on:
|
||||
pull_request:
|
||||
|
||||
concurrency:
|
||||
group: ci-backends-pr-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
group: ci-backends-pr-${{ github.head_ref || github.ref }}-${{ github.repository }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
generate-matrix:
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
matrix-singlearch: ${{ steps.set-matrix.outputs['matrix-singlearch'] }}
|
||||
matrix-multiarch: ${{ steps.set-matrix.outputs['matrix-multiarch'] }}
|
||||
matrix-darwin: ${{ steps.set-matrix.outputs['matrix-darwin'] }}
|
||||
merge-matrix-multiarch: ${{ steps.set-matrix.outputs['merge-matrix-multiarch'] }}
|
||||
merge-matrix-singlearch: ${{ steps.set-matrix.outputs['merge-matrix-singlearch'] }}
|
||||
has-backends-singlearch: ${{ steps.set-matrix.outputs['has-backends-singlearch'] }}
|
||||
has-backends-multiarch: ${{ steps.set-matrix.outputs['has-backends-multiarch'] }}
|
||||
has-backends-darwin: ${{ steps.set-matrix.outputs['has-backends-darwin'] }}
|
||||
has-merges-multiarch: ${{ steps.set-matrix.outputs['has-merges-multiarch'] }}
|
||||
has-merges-singlearch: ${{ steps.set-matrix.outputs['has-merges-singlearch'] }}
|
||||
matrix: ${{ steps.set-matrix.outputs.matrix }}
|
||||
matrix-darwin: ${{ steps.set-matrix.outputs.matrix-darwin }}
|
||||
bases-matrix: ${{ steps.set-matrix.outputs.bases-matrix }}
|
||||
has-backends: ${{ steps.set-matrix.outputs.has-backends }}
|
||||
has-backends-darwin: ${{ steps.set-matrix.outputs.has-backends-darwin }}
|
||||
has-bases: ${{ steps.set-matrix.outputs.has-bases }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
@@ -33,9 +29,8 @@ jobs:
|
||||
bun add js-yaml
|
||||
bun add @octokit/core
|
||||
|
||||
# filters the matrix in backend.yml; splits into single-arch and
|
||||
# multi-arch groups so backend-merge-jobs can `needs:` only the latter
|
||||
# (matches backend.yml's structure).
|
||||
# Filters the matrix from backend.yml against this PR's changed files
|
||||
# AND derives the deduplicated bases-matrix consumed by build-bases.
|
||||
- name: Filter matrix for changed backends
|
||||
id: set-matrix
|
||||
env:
|
||||
@@ -43,92 +38,60 @@ jobs:
|
||||
GITHUB_EVENT_PATH: ${{ github.event_path }}
|
||||
run: bun run scripts/changed-backends.js
|
||||
|
||||
backend-jobs-multiarch:
|
||||
build-bases:
|
||||
needs: generate-matrix
|
||||
uses: ./.github/workflows/backend_build.yml
|
||||
if: needs.generate-matrix.outputs['has-backends-multiarch'] == 'true'
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
platform-tag: ${{ matrix.platform-tag || '' }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
builder-base-image: ${{ matrix.builder-base-image || '' }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
backend: ${{ matrix.backend }}
|
||||
dockerfile: ${{ matrix.dockerfile }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
context: ${{ matrix.context }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
|
||||
secrets:
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: true
|
||||
max-parallel: 8
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-multiarch']) }}
|
||||
backend-jobs-singlearch:
|
||||
needs: generate-matrix
|
||||
uses: ./.github/workflows/backend_build.yml
|
||||
if: needs.generate-matrix.outputs['has-backends-singlearch'] == 'true'
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
platform-tag: ${{ matrix.platform-tag || '' }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
builder-base-image: ${{ matrix.builder-base-image || '' }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
backend: ${{ matrix.backend }}
|
||||
dockerfile: ${{ matrix.dockerfile }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
context: ${{ matrix.context }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
|
||||
secrets:
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: true
|
||||
max-parallel: 8
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['matrix-singlearch']) }}
|
||||
backend-merge-jobs-multiarch:
|
||||
needs: [generate-matrix, backend-jobs-multiarch]
|
||||
# backend_merge.yml's push-side steps are all gated on
|
||||
# github.event_name != 'pull_request', so on a PR the merge job would
|
||||
# do nothing. Skip it entirely to avoid spinning up an empty runner.
|
||||
if: github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges-multiarch'] == 'true'
|
||||
uses: ./.github/workflows/backend_merge.yml
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
secrets:
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
if: needs.generate-matrix.outputs.has-bases == 'true'
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-multiarch']) }}
|
||||
matrix: ${{ fromJSON(needs.generate-matrix.outputs.bases-matrix) }}
|
||||
uses: ./.github/workflows/base_images.yml
|
||||
with:
|
||||
lang: ${{ matrix.lang }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
tag-stem: ${{ matrix.tag-stem }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
secrets:
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
|
||||
backend-merge-jobs-singlearch:
|
||||
needs: [generate-matrix, backend-jobs-singlearch]
|
||||
if: github.event_name != 'pull_request' && needs.generate-matrix.outputs['has-merges-singlearch'] == 'true'
|
||||
uses: ./.github/workflows/backend_merge.yml
|
||||
backend-jobs:
|
||||
needs: [generate-matrix, build-bases]
|
||||
uses: ./.github/workflows/backend_build.yml
|
||||
if: |
|
||||
always() && needs.generate-matrix.outputs.has-backends == 'true' &&
|
||||
(needs.build-bases.result == 'success' || needs.build-bases.result == 'skipped')
|
||||
with:
|
||||
tag-latest: ${{ matrix.tag-latest }}
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
backend: ${{ matrix.backend }}
|
||||
dockerfile: ${{ matrix.dockerfile }}
|
||||
skip-drivers: ${{ matrix.skip-drivers }}
|
||||
context: ${{ matrix.context }}
|
||||
ubuntu-version: ${{ matrix.ubuntu-version }}
|
||||
amdgpu-targets: ${{ matrix.amdgpu-targets || 'gfx908,gfx90a,gfx942,gfx950,gfx1030,gfx1100,gfx1101,gfx1102,gfx1151,gfx1200,gfx1201' }}
|
||||
# The script annotates each filtered Python entry with the prebuilt
|
||||
# base ref it should consume; non-Python entries get '' and run their
|
||||
# own inline bootstrap.
|
||||
base-image-prebuilt: ${{ matrix.base-image-prebuilt || '' }}
|
||||
secrets:
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs['merge-matrix-singlearch']) }}
|
||||
fail-fast: true
|
||||
matrix: ${{ fromJson(needs.generate-matrix.outputs.matrix) }}
|
||||
|
||||
backend-jobs-darwin:
|
||||
needs: generate-matrix
|
||||
uses: ./.github/workflows/backend_build_darwin.yml
|
||||
@@ -136,7 +99,7 @@ jobs:
|
||||
with:
|
||||
backend: ${{ matrix.backend }}
|
||||
build-type: ${{ matrix.build-type }}
|
||||
go-version: "1.25.x"
|
||||
go-version: "1.24.x"
|
||||
tag-suffix: ${{ matrix.tag-suffix }}
|
||||
lang: ${{ matrix.lang || 'python' }}
|
||||
use-pip: ${{ matrix.backend == 'diffusers' }}
|
||||
|
||||
161
.github/workflows/base-images.yml
vendored
161
.github/workflows/base-images.yml
vendored
@@ -1,161 +0,0 @@
|
||||
---
|
||||
name: 'build base-grpc images'
|
||||
|
||||
# Builds + pushes pre-compiled builder base images that downstream
|
||||
# llama-cpp / ik-llama-cpp / turboquant variant Dockerfiles will FROM
|
||||
# (PR 2). Each base contains apt deps + protoc + cmake + gRPC at
|
||||
# /opt/grpc + (conditionally) CUDA / ROCm / Vulkan toolchains.
|
||||
#
|
||||
# Triggers:
|
||||
# - schedule (Saturdays 05:00 UTC) - picks up Ubuntu/CUDA/ROCm
|
||||
# security updates and re-runs ahead of the backend.yml weekly
|
||||
# cron (Sundays 06:00 UTC).
|
||||
# - workflow_dispatch - manual one-off rebuild.
|
||||
# - push to master that touches Dockerfile.base-grpc-builder or
|
||||
# this workflow itself - keeps bases in sync with their inputs.
|
||||
#
|
||||
# Bootstrap (one-time after this PR merges):
|
||||
# gh workflow run base-images.yml --ref master
|
||||
# Wait ~30 min for all 9 matrix variants to push to
|
||||
# quay.io/go-skynet/ci-cache:base-grpc-* before merging PR 2.
|
||||
|
||||
on:
|
||||
schedule:
|
||||
- cron: '0 5 * * 6'
|
||||
workflow_dispatch:
|
||||
push:
|
||||
branches: [master]
|
||||
paths:
|
||||
- 'backend/Dockerfile.base-grpc-builder'
|
||||
- '.github/workflows/base-images.yml'
|
||||
# The install logic and apt-mirror helper are bind-mounted into
|
||||
# Dockerfile.base-grpc-builder at build time — changes to either
|
||||
# affect the produced base images and must trigger a rebuild.
|
||||
- '.docker/install-base-deps.sh'
|
||||
- '.docker/apt-mirror.sh'
|
||||
|
||||
concurrency:
|
||||
group: ci-base-images-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
|
||||
jobs:
|
||||
build:
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- tag: 'base-grpc-amd64'
|
||||
runs-on: 'ubuntu-latest'
|
||||
base-image: 'ubuntu:24.04'
|
||||
build-type: ''
|
||||
cuda-major-version: ''
|
||||
cuda-minor-version: ''
|
||||
ubuntu-version: '2404'
|
||||
- tag: 'base-grpc-arm64'
|
||||
runs-on: 'ubuntu-24.04-arm'
|
||||
base-image: 'ubuntu:24.04'
|
||||
build-type: ''
|
||||
cuda-major-version: ''
|
||||
cuda-minor-version: ''
|
||||
ubuntu-version: '2404'
|
||||
- tag: 'base-grpc-cuda-12-amd64'
|
||||
runs-on: 'ubuntu-latest'
|
||||
base-image: 'ubuntu:24.04'
|
||||
build-type: 'cublas'
|
||||
cuda-major-version: '12'
|
||||
cuda-minor-version: '8'
|
||||
ubuntu-version: '2404'
|
||||
- tag: 'base-grpc-cuda-13-amd64'
|
||||
runs-on: 'ubuntu-latest'
|
||||
base-image: 'ubuntu:22.04'
|
||||
build-type: 'cublas'
|
||||
cuda-major-version: '13'
|
||||
cuda-minor-version: '0'
|
||||
ubuntu-version: '2204'
|
||||
- tag: 'base-grpc-cuda-13-arm64'
|
||||
runs-on: 'ubuntu-24.04-arm'
|
||||
base-image: 'ubuntu:24.04'
|
||||
build-type: 'cublas'
|
||||
cuda-major-version: '13'
|
||||
cuda-minor-version: '0'
|
||||
ubuntu-version: '2404'
|
||||
- tag: 'base-grpc-rocm-amd64'
|
||||
runs-on: 'ubuntu-latest'
|
||||
base-image: 'rocm/dev-ubuntu-24.04:7.2.1'
|
||||
build-type: 'hipblas'
|
||||
cuda-major-version: ''
|
||||
cuda-minor-version: ''
|
||||
ubuntu-version: '2404'
|
||||
- tag: 'base-grpc-vulkan-amd64'
|
||||
runs-on: 'ubuntu-latest'
|
||||
base-image: 'ubuntu:24.04'
|
||||
build-type: 'vulkan'
|
||||
cuda-major-version: ''
|
||||
cuda-minor-version: ''
|
||||
ubuntu-version: '2404'
|
||||
- tag: 'base-grpc-vulkan-arm64'
|
||||
runs-on: 'ubuntu-24.04-arm'
|
||||
base-image: 'ubuntu:24.04'
|
||||
build-type: 'vulkan'
|
||||
cuda-major-version: ''
|
||||
cuda-minor-version: ''
|
||||
ubuntu-version: '2404'
|
||||
- tag: 'base-grpc-intel-amd64'
|
||||
runs-on: 'ubuntu-latest'
|
||||
base-image: 'intel/oneapi-basekit:2025.3.2-0-devel-ubuntu24.04'
|
||||
build-type: 'sycl'
|
||||
cuda-major-version: ''
|
||||
cuda-minor-version: ''
|
||||
ubuntu-version: '2404'
|
||||
# Legacy JetPack r36.4.0 base for older Jetson devices (CUDA 12).
|
||||
# Distinct from base-grpc-cuda-13-arm64 (Ubuntu 24.04 + CUDA 13 sbsa)
|
||||
# which targets newer Jetsons. Some matrix entries
|
||||
# (-nvidia-l4t-arm64-llama-cpp / -turboquant) still build against
|
||||
# the JetPack image, so we need a matching base.
|
||||
- tag: 'base-grpc-l4t-cuda-12-arm64'
|
||||
runs-on: 'ubuntu-24.04-arm'
|
||||
base-image: 'nvcr.io/nvidia/l4t-jetpack:r36.4.0'
|
||||
build-type: 'l4t'
|
||||
cuda-major-version: '12'
|
||||
cuda-minor-version: '0'
|
||||
ubuntu-version: '2204'
|
||||
# JetPack r36.4.0 already ships CUDA preinstalled at /usr/local/cuda;
|
||||
# apt-installing cuda-nvcc-12-0 from the public repos fails because
|
||||
# those packages aren't published for the JetPack apt feed. Match
|
||||
# the original l4t matrix entry which set skip-drivers: 'true'.
|
||||
skip-drivers: 'true'
|
||||
steps:
|
||||
- uses: actions/checkout@v6
|
||||
with:
|
||||
submodules: false
|
||||
- name: Free disk space
|
||||
uses: ./.github/actions/free-disk-space
|
||||
- name: Set up build disk
|
||||
uses: ./.github/actions/setup-build-disk
|
||||
- uses: docker/setup-qemu-action@master
|
||||
with:
|
||||
platforms: all
|
||||
- uses: docker/setup-buildx-action@master
|
||||
- uses: docker/login-action@v4
|
||||
with:
|
||||
registry: quay.io
|
||||
username: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
password: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
- uses: docker/build-push-action@v7
|
||||
with:
|
||||
context: .
|
||||
file: ./backend/Dockerfile.base-grpc-builder
|
||||
build-args: |
|
||||
BASE_IMAGE=${{ matrix.base-image }}
|
||||
BUILD_TYPE=${{ matrix.build-type }}
|
||||
CUDA_MAJOR_VERSION=${{ matrix.cuda-major-version }}
|
||||
CUDA_MINOR_VERSION=${{ matrix.cuda-minor-version }}
|
||||
UBUNTU_VERSION=${{ matrix.ubuntu-version }}
|
||||
SKIP_DRIVERS=${{ matrix.skip-drivers || 'false' }}
|
||||
cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-${{ matrix.tag }}
|
||||
cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache-${{ matrix.tag }},mode=max,ignore-error=true
|
||||
provenance: false
|
||||
tags: quay.io/go-skynet/ci-cache:${{ matrix.tag }}
|
||||
push: true
|
||||
152
.github/workflows/base_images.yml
vendored
Normal file
152
.github/workflows/base_images.yml
vendored
Normal file
@@ -0,0 +1,152 @@
|
||||
---
|
||||
name: 'build base image (reusable)'
|
||||
|
||||
# Builds and pushes one (lang, accel, arch, ubuntu, cuda) base image flavour
|
||||
# to quay.io/go-skynet/ci-cache:base-image-<stem>[-pr<N>]. Consumed by
|
||||
# backend builds via the BASE_IMAGE_PREBUILT build-arg. PR builds tag with
|
||||
# `-pr${PR_NUMBER}` so the same PR's backend matrix can opt-in to the
|
||||
# freshly-built base; master builds overwrite the unsuffixed tag for
|
||||
# downstream consumption. The image lives in the same ci-cache repo as the
|
||||
# buildkit cache (under a `base-image-` prefix that doesn't collide with
|
||||
# the `base-<stem>` cache prefix), so no separate quay repo + grant is
|
||||
# needed. See .agents/ci-caching.md for the full tagging scheme.
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
lang:
|
||||
description: 'Language toolchain (matches .docker/bases/Dockerfile.<lang>)'
|
||||
required: true
|
||||
type: string
|
||||
base-image:
|
||||
description: 'Upstream base image (ubuntu:24.04, rocm/dev-ubuntu-24.04:..., etc.)'
|
||||
required: true
|
||||
type: string
|
||||
build-type:
|
||||
description: 'BUILD_TYPE: empty for CPU, cublas, hipblas, vulkan, l4t, ...'
|
||||
default: ''
|
||||
type: string
|
||||
cuda-major-version:
|
||||
description: 'CUDA major version (only meaningful for cublas/l4t)'
|
||||
default: '12'
|
||||
type: string
|
||||
cuda-minor-version:
|
||||
description: 'CUDA minor version'
|
||||
default: '9'
|
||||
type: string
|
||||
ubuntu-version:
|
||||
description: 'Ubuntu version code (2204, 2404)'
|
||||
default: '2404'
|
||||
type: string
|
||||
platforms:
|
||||
description: 'Single platform per call (linux/amd64 or linux/arm64)'
|
||||
required: true
|
||||
type: string
|
||||
runs-on:
|
||||
description: 'Runner label'
|
||||
required: true
|
||||
type: string
|
||||
tag-stem:
|
||||
description: 'Stable portion of the image tag (e.g. python-cpu-amd64-2404)'
|
||||
required: true
|
||||
type: string
|
||||
skip-drivers:
|
||||
description: 'Pass-through to the base Dockerfile'
|
||||
default: 'false'
|
||||
type: string
|
||||
secrets:
|
||||
quayUsername:
|
||||
required: false
|
||||
quayPassword:
|
||||
required: false
|
||||
outputs:
|
||||
image-ref:
|
||||
description: 'Full image reference of the built base'
|
||||
value: ${{ jobs.base-build.outputs.image-ref }}
|
||||
|
||||
jobs:
|
||||
base-build:
|
||||
runs-on: ${{ inputs.runs-on }}
|
||||
env:
|
||||
quay_username: ${{ secrets.quayUsername }}
|
||||
outputs:
|
||||
image-ref: ${{ steps.compute_ref.outputs.ref }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@v6
|
||||
|
||||
- name: Configure apt mirror on runner
|
||||
id: apt_mirror
|
||||
uses: ./.github/actions/configure-apt-mirror
|
||||
|
||||
- name: Free Disk Space (Ubuntu)
|
||||
if: inputs.runs-on == 'ubuntu-latest'
|
||||
uses: jlumbroso/free-disk-space@main
|
||||
with:
|
||||
tool-cache: true
|
||||
android: true
|
||||
dotnet: true
|
||||
haskell: true
|
||||
large-packages: true
|
||||
docker-images: true
|
||||
swap-storage: true
|
||||
|
||||
- name: Compute image ref
|
||||
id: compute_ref
|
||||
run: |
|
||||
stem='${{ inputs.tag-stem }}'
|
||||
if [ "${{ github.event_name }}" = "pull_request" ]; then
|
||||
tag="${stem}-pr${{ github.event.number }}"
|
||||
else
|
||||
tag="${stem}"
|
||||
fi
|
||||
echo "tag=${tag}" >> "$GITHUB_OUTPUT"
|
||||
# Published into the existing ci-cache repo (the CI robot already
|
||||
# has write access there) under a distinct `base-image-` prefix so
|
||||
# the OCI image tags coexist with the buildkit cache tags
|
||||
# (`base-<stem>`, `cache<tag-suffix>`, `cache-localai<tag-suffix>`).
|
||||
echo "ref=quay.io/go-skynet/ci-cache:base-image-${tag}" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Set up QEMU
|
||||
uses: docker/setup-qemu-action@master
|
||||
with:
|
||||
platforms: all
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
id: buildx
|
||||
uses: docker/setup-buildx-action@master
|
||||
|
||||
- name: Login to Quay.io
|
||||
if: ${{ env.quay_username != '' }}
|
||||
uses: docker/login-action@v4
|
||||
with:
|
||||
registry: quay.io
|
||||
username: ${{ secrets.quayUsername }}
|
||||
password: ${{ secrets.quayPassword }}
|
||||
|
||||
- name: Build and push base image
|
||||
uses: docker/build-push-action@v7
|
||||
with:
|
||||
builder: ${{ steps.buildx.outputs.name }}
|
||||
context: .
|
||||
file: ./.docker/bases/Dockerfile.${{ inputs.lang }}
|
||||
build-args: |
|
||||
BUILD_TYPE=${{ inputs.build-type }}
|
||||
CUDA_MAJOR_VERSION=${{ inputs.cuda-major-version }}
|
||||
CUDA_MINOR_VERSION=${{ inputs.cuda-minor-version }}
|
||||
BASE_IMAGE=${{ inputs.base-image }}
|
||||
UBUNTU_VERSION=${{ inputs.ubuntu-version }}
|
||||
SKIP_DRIVERS=${{ inputs.skip-drivers }}
|
||||
APT_MIRROR=${{ steps.apt_mirror.outputs.effective-mirror }}
|
||||
APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
|
||||
platforms: ${{ inputs.platforms }}
|
||||
# Push on PRs as well (if creds present) so the PR's backend matrix
|
||||
# can opt-in to the freshly-built base via -pr${N} tag.
|
||||
push: ${{ env.quay_username != '' }}
|
||||
tags: ${{ steps.compute_ref.outputs.ref }}
|
||||
cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:base-${{ inputs.tag-stem }}
|
||||
cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:base-${{ inputs.tag-stem }},mode=max,ignore-error=true
|
||||
|
||||
- name: job summary
|
||||
run: |
|
||||
echo "Built base image: ${{ steps.compute_ref.outputs.ref }}" >> "$GITHUB_STEP_SUMMARY"
|
||||
6
.github/workflows/bump_deps.yaml
vendored
6
.github/workflows/bump_deps.yaml
vendored
@@ -22,10 +22,6 @@ jobs:
|
||||
variable: "TURBOQUANT_VERSION"
|
||||
branch: "feature/turboquant-kv-cache"
|
||||
file: "backend/cpp/turboquant/Makefile"
|
||||
- repository: "antirez/ds4"
|
||||
variable: "DS4_VERSION"
|
||||
branch: "main"
|
||||
file: "backend/cpp/ds4/Makefile"
|
||||
- repository: "ggml-org/whisper.cpp"
|
||||
variable: "WHISPER_CPP_VERSION"
|
||||
branch: "master"
|
||||
@@ -54,7 +50,7 @@ jobs:
|
||||
variable: "QWEN3TTS_CPP_VERSION"
|
||||
branch: "main"
|
||||
file: "backend/go/qwen3-tts-cpp/Makefile"
|
||||
- repository: "localai-org/vibevoice.cpp"
|
||||
- repository: "mudler/vibevoice.cpp"
|
||||
variable: "VIBEVOICE_CPP_VERSION"
|
||||
branch: "master"
|
||||
file: "backend/go/vibevoice-cpp/Makefile"
|
||||
|
||||
4
.github/workflows/generate_intel_image.yaml
vendored
4
.github/workflows/generate_intel_image.yaml
vendored
@@ -7,8 +7,8 @@ on:
|
||||
- master
|
||||
|
||||
concurrency:
|
||||
group: intel-cache-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
group: intel-cache-${{ github.head_ref || github.ref }}-${{ github.repository }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
generate_caches:
|
||||
|
||||
17
.github/workflows/image-pr.yml
vendored
17
.github/workflows/image-pr.yml
vendored
@@ -5,8 +5,8 @@
|
||||
pull_request:
|
||||
|
||||
concurrency:
|
||||
group: ci-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
group: ci-${{ github.head_ref || github.ref }}-${{ github.repository }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
image-build:
|
||||
@@ -18,7 +18,6 @@
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
platform-tag: ${{ matrix.platform-tag || '' }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
makeflags: ${{ matrix.makeflags }}
|
||||
@@ -72,23 +71,13 @@
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
ubuntu-version: '2404'
|
||||
- build-type: 'vulkan'
|
||||
platforms: 'linux/amd64'
|
||||
platform-tag: 'amd64'
|
||||
platforms: 'linux/amd64,linux/arm64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-vulkan-core'
|
||||
runs-on: 'ubuntu-latest'
|
||||
base-image: "ubuntu:24.04"
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
ubuntu-version: '2404'
|
||||
- build-type: 'vulkan'
|
||||
platforms: 'linux/arm64'
|
||||
platform-tag: 'arm64'
|
||||
tag-latest: 'false'
|
||||
tag-suffix: '-vulkan-core'
|
||||
runs-on: 'ubuntu-24.04-arm'
|
||||
base-image: "ubuntu:24.04"
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
ubuntu-version: '2404'
|
||||
- build-type: 'cublas'
|
||||
cuda-major-version: "13"
|
||||
cuda-minor-version: "0"
|
||||
|
||||
59
.github/workflows/image.yml
vendored
59
.github/workflows/image.yml
vendored
@@ -9,8 +9,8 @@
|
||||
- '*'
|
||||
|
||||
concurrency:
|
||||
group: ci-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
group: ci-${{ github.head_ref || github.ref }}-${{ github.repository }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
hipblas-jobs:
|
||||
@@ -56,7 +56,6 @@
|
||||
cuda-major-version: ${{ matrix.cuda-major-version }}
|
||||
cuda-minor-version: ${{ matrix.cuda-minor-version }}
|
||||
platforms: ${{ matrix.platforms }}
|
||||
platform-tag: ${{ matrix.platform-tag || '' }}
|
||||
runs-on: ${{ matrix.runs-on }}
|
||||
base-image: ${{ matrix.base-image }}
|
||||
makeflags: ${{ matrix.makeflags }}
|
||||
@@ -73,8 +72,7 @@
|
||||
matrix:
|
||||
include:
|
||||
- build-type: ''
|
||||
platforms: 'linux/amd64'
|
||||
platform-tag: 'amd64'
|
||||
platforms: 'linux/amd64,linux/arm64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: ''
|
||||
base-image: "ubuntu:24.04"
|
||||
@@ -83,17 +81,6 @@
|
||||
skip-drivers: 'false'
|
||||
ubuntu-version: '2404'
|
||||
ubuntu-codename: 'noble'
|
||||
- build-type: ''
|
||||
platforms: 'linux/arm64'
|
||||
platform-tag: 'arm64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: ''
|
||||
base-image: "ubuntu:24.04"
|
||||
runs-on: 'ubuntu-24.04-arm'
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
skip-drivers: 'false'
|
||||
ubuntu-version: '2404'
|
||||
ubuntu-codename: 'noble'
|
||||
- build-type: 'cublas'
|
||||
cuda-major-version: "12"
|
||||
cuda-minor-version: "8"
|
||||
@@ -119,8 +106,7 @@
|
||||
ubuntu-version: '2404'
|
||||
ubuntu-codename: 'noble'
|
||||
- build-type: 'vulkan'
|
||||
platforms: 'linux/amd64'
|
||||
platform-tag: 'amd64'
|
||||
platforms: 'linux/amd64,linux/arm64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-gpu-vulkan'
|
||||
runs-on: 'ubuntu-latest'
|
||||
@@ -129,17 +115,6 @@
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
ubuntu-version: '2404'
|
||||
ubuntu-codename: 'noble'
|
||||
- build-type: 'vulkan'
|
||||
platforms: 'linux/arm64'
|
||||
platform-tag: 'arm64'
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-gpu-vulkan'
|
||||
runs-on: 'ubuntu-24.04-arm'
|
||||
base-image: "ubuntu:24.04"
|
||||
skip-drivers: 'false'
|
||||
makeflags: "--jobs=4 --output-sync=target"
|
||||
ubuntu-version: '2404'
|
||||
ubuntu-codename: 'noble'
|
||||
- build-type: 'intel'
|
||||
platforms: 'linux/amd64'
|
||||
tag-latest: 'auto'
|
||||
@@ -149,32 +124,6 @@
|
||||
makeflags: "--jobs=3 --output-sync=target"
|
||||
ubuntu-version: '2404'
|
||||
ubuntu-codename: 'noble'
|
||||
|
||||
core-image-merge:
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
needs: core-image-build
|
||||
uses: ./.github/workflows/image_merge.yml
|
||||
with:
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: ''
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
|
||||
gpu-vulkan-image-merge:
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
needs: core-image-build
|
||||
uses: ./.github/workflows/image_merge.yml
|
||||
with:
|
||||
tag-latest: 'auto'
|
||||
tag-suffix: '-gpu-vulkan'
|
||||
secrets:
|
||||
dockerUsername: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
dockerPassword: ${{ secrets.DOCKERHUB_PASSWORD }}
|
||||
quayUsername: ${{ secrets.LOCALAI_REGISTRY_USERNAME }}
|
||||
quayPassword: ${{ secrets.LOCALAI_REGISTRY_PASSWORD }}
|
||||
|
||||
gh-runner:
|
||||
if: github.repository == 'mudler/LocalAI'
|
||||
|
||||
103
.github/workflows/image_build.yml
vendored
103
.github/workflows/image_build.yml
vendored
@@ -24,15 +24,6 @@ on:
|
||||
description: 'Platforms'
|
||||
default: ''
|
||||
type: string
|
||||
platform-tag:
|
||||
description: |
|
||||
Short tag identifying the platform leg, e.g. "amd64" or "arm64".
|
||||
Used to scope the per-arch registry cache and the digest artifact name.
|
||||
Optional during the migration; will be flipped to required: true once
|
||||
every caller passes an explicit value.
|
||||
required: false
|
||||
default: ''
|
||||
type: string
|
||||
tag-latest:
|
||||
description: 'Tag latest'
|
||||
default: ''
|
||||
@@ -86,13 +77,63 @@ jobs:
|
||||
id: apt_mirror
|
||||
uses: ./.github/actions/configure-apt-mirror
|
||||
|
||||
- name: Free disk space
|
||||
uses: ./.github/actions/free-disk-space
|
||||
- name: Free Disk Space (Ubuntu)
|
||||
if: inputs.runs-on == 'ubuntu-latest'
|
||||
uses: jlumbroso/free-disk-space@main
|
||||
with:
|
||||
mode: ${{ inputs.runs-on == 'ubuntu-latest' && 'hosted' || 'skip' }}
|
||||
# this might remove tools that are actually needed,
|
||||
# if set to "true" but frees about 6 GB
|
||||
tool-cache: true
|
||||
# all of these default to true, but feel free to set to
|
||||
# "false" if necessary for your workflow
|
||||
android: true
|
||||
dotnet: true
|
||||
haskell: true
|
||||
large-packages: true
|
||||
docker-images: true
|
||||
swap-storage: true
|
||||
|
||||
- name: Set up build disk
|
||||
uses: ./.github/actions/setup-build-disk
|
||||
- name: Release space from worker
|
||||
if: inputs.runs-on == 'ubuntu-latest'
|
||||
run: |
|
||||
echo "Listing top largest packages"
|
||||
pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
|
||||
head -n 30 <<< "${pkgs}"
|
||||
echo
|
||||
df -h
|
||||
echo
|
||||
sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
|
||||
sudo apt-get remove --auto-remove android-sdk-platform-tools snapd || true
|
||||
sudo apt-get purge --auto-remove android-sdk-platform-tools snapd || true
|
||||
sudo rm -rf /usr/local/lib/android
|
||||
sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
|
||||
sudo rm -rf /usr/share/dotnet
|
||||
sudo apt-get remove -y '^mono-.*' || true
|
||||
sudo apt-get remove -y '^ghc-.*' || true
|
||||
sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
|
||||
sudo apt-get remove -y 'php.*' || true
|
||||
sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
|
||||
sudo apt-get remove -y '^google-.*' || true
|
||||
sudo apt-get remove -y azure-cli || true
|
||||
sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
|
||||
sudo apt-get remove -y '^gfortran-.*' || true
|
||||
sudo apt-get remove -y microsoft-edge-stable || true
|
||||
sudo apt-get remove -y firefox || true
|
||||
sudo apt-get remove -y powershell || true
|
||||
sudo apt-get remove -y r-base-core || true
|
||||
sudo apt-get autoremove -y
|
||||
sudo apt-get clean
|
||||
echo
|
||||
echo "Listing top largest packages"
|
||||
pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
|
||||
head -n 30 <<< "${pkgs}"
|
||||
echo
|
||||
sudo rm -rfv build || true
|
||||
sudo rm -rf /usr/share/dotnet || true
|
||||
sudo rm -rf /opt/ghc || true
|
||||
sudo rm -rf "/usr/local/share/boost" || true
|
||||
sudo rm -rf "$AGENT_TOOLSDIRECTORY" || true
|
||||
df -h
|
||||
|
||||
- name: Docker meta
|
||||
id: meta
|
||||
@@ -147,8 +188,7 @@ jobs:
|
||||
username: ${{ secrets.quayUsername }}
|
||||
password: ${{ secrets.quayPassword }}
|
||||
|
||||
- name: Build and push by digest
|
||||
id: build
|
||||
- name: Build and push
|
||||
uses: docker/build-push-action@v7
|
||||
if: github.event_name != 'pull_request'
|
||||
with:
|
||||
@@ -166,33 +206,12 @@ jobs:
|
||||
APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
|
||||
context: .
|
||||
file: ./Dockerfile
|
||||
cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
|
||||
cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}-${{ inputs.platform-tag }},mode=max,ignore-error=true
|
||||
cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}
|
||||
cache-to: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }},mode=max,ignore-error=true
|
||||
platforms: ${{ inputs.platforms }}
|
||||
outputs: |
|
||||
type=image,name=quay.io/go-skynet/local-ai,push-by-digest=true,name-canonical=true,push=true
|
||||
type=image,name=localai/localai,push-by-digest=true,name-canonical=true,push=true
|
||||
# See backend_build.yml for the rationale — provenance=mode=max
|
||||
# diverges the manifest-list digest per registry, breaking the
|
||||
# downstream imagetools create lookup.
|
||||
provenance: false
|
||||
push: ${{ github.event_name != 'pull_request' }}
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
|
||||
- name: Export digest
|
||||
if: github.event_name != 'pull_request'
|
||||
run: |
|
||||
mkdir -p /tmp/digests
|
||||
digest="${{ steps.build.outputs.digest }}"
|
||||
touch "/tmp/digests/${digest#sha256:}"
|
||||
|
||||
- name: Upload digest artifact
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: digests-localai${{ inputs.tag-suffix == '' && '-core' || inputs.tag-suffix }}-${{ inputs.platform-tag }}
|
||||
path: /tmp/digests/*
|
||||
if-no-files-found: error
|
||||
retention-days: 1
|
||||
### Start testing image
|
||||
- name: Build and push
|
||||
uses: docker/build-push-action@v7
|
||||
@@ -212,7 +231,7 @@ jobs:
|
||||
APT_PORTS_MIRROR=${{ steps.apt_mirror.outputs.effective-ports-mirror }}
|
||||
context: .
|
||||
file: ./Dockerfile
|
||||
cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}-${{ inputs.platform-tag }}
|
||||
cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}
|
||||
platforms: ${{ inputs.platforms }}
|
||||
#push: true
|
||||
tags: ${{ steps.meta_pull_request.outputs.tags }}
|
||||
|
||||
117
.github/workflows/image_merge.yml
vendored
117
.github/workflows/image_merge.yml
vendored
@@ -1,117 +0,0 @@
|
||||
---
|
||||
name: 'merge LocalAI image manifest list (reusable)'
|
||||
|
||||
# Reusable workflow that joins per-arch digest artifacts (uploaded by
|
||||
# image_build.yml when called with platform-tag) into a single tagged
|
||||
# multi-arch manifest list.
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
tag-latest:
|
||||
description: 'Whether the manifest list should also be tagged latest (auto/false/true)'
|
||||
required: false
|
||||
type: string
|
||||
default: ''
|
||||
tag-suffix:
|
||||
description: 'Image tag suffix (empty for core image). Used in artifact pattern with a -core placeholder for empty.'
|
||||
required: true
|
||||
type: string
|
||||
secrets:
|
||||
dockerUsername:
|
||||
required: false
|
||||
dockerPassword:
|
||||
required: false
|
||||
quayUsername:
|
||||
required: true
|
||||
quayPassword:
|
||||
required: true
|
||||
|
||||
jobs:
|
||||
merge:
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
quay_username: ${{ secrets.quayUsername }}
|
||||
steps:
|
||||
- name: Download digests
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
pattern: digests-localai${{ inputs.tag-suffix == '' && '-core' || inputs.tag-suffix }}-*
|
||||
merge-multiple: true
|
||||
path: /tmp/digests
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@master
|
||||
|
||||
- name: Login to DockerHub
|
||||
if: github.event_name != 'pull_request'
|
||||
uses: docker/login-action@v4
|
||||
with:
|
||||
username: ${{ secrets.dockerUsername }}
|
||||
password: ${{ secrets.dockerPassword }}
|
||||
|
||||
- name: Login to Quay.io
|
||||
uses: docker/login-action@v4
|
||||
with:
|
||||
registry: quay.io
|
||||
username: ${{ secrets.quayUsername }}
|
||||
password: ${{ secrets.quayPassword }}
|
||||
|
||||
- name: Docker meta
|
||||
id: meta
|
||||
uses: docker/metadata-action@v6
|
||||
with:
|
||||
images: |
|
||||
quay.io/go-skynet/local-ai
|
||||
localai/localai
|
||||
tags: |
|
||||
type=ref,event=branch
|
||||
type=semver,pattern={{raw}}
|
||||
type=sha
|
||||
flavor: |
|
||||
latest=${{ inputs.tag-latest }}
|
||||
suffix=${{ inputs.tag-suffix }},onlatest=true
|
||||
|
||||
- name: Create manifest list and push (quay)
|
||||
working-directory: /tmp/digests
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tags=$(jq -cr '.tags | map(select(startswith("quay.io/"))) | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON")
|
||||
if [ -z "$tags" ]; then
|
||||
echo "No quay.io tags from docker/metadata-action; skipping quay merge"
|
||||
else
|
||||
# shellcheck disable=SC2086
|
||||
docker buildx imagetools create $tags \
|
||||
$(printf 'quay.io/go-skynet/local-ai@sha256:%s ' *)
|
||||
fi
|
||||
|
||||
- name: Create manifest list and push (dockerhub)
|
||||
if: github.event_name != 'pull_request'
|
||||
working-directory: /tmp/digests
|
||||
run: |
|
||||
set -euo pipefail
|
||||
tags=$(jq -cr '.tags | map(select(startswith("localai/"))) | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON")
|
||||
if [ -z "$tags" ]; then
|
||||
echo "No dockerhub tags from docker/metadata-action; skipping dockerhub merge"
|
||||
else
|
||||
# shellcheck disable=SC2086
|
||||
docker buildx imagetools create $tags \
|
||||
$(printf 'localai/localai@sha256:%s ' *)
|
||||
fi
|
||||
|
||||
- name: Inspect manifest
|
||||
run: |
|
||||
set -euo pipefail
|
||||
first_tag=$(jq -cr '.tags[0]' <<< "$DOCKER_METADATA_OUTPUT_JSON")
|
||||
if [ -n "$first_tag" ] && [ "$first_tag" != "null" ]; then
|
||||
docker buildx imagetools inspect "$first_tag"
|
||||
fi
|
||||
|
||||
- name: Job summary
|
||||
run: |
|
||||
set -euo pipefail
|
||||
echo "Merged manifest tags:" >> "$GITHUB_STEP_SUMMARY"
|
||||
jq -r '.tags[]' <<< "$DOCKER_METADATA_OUTPUT_JSON" | sed 's/^/- /' >> "$GITHUB_STEP_SUMMARY"
|
||||
echo >> "$GITHUB_STEP_SUMMARY"
|
||||
echo "Per-arch digests:" >> "$GITHUB_STEP_SUMMARY"
|
||||
ls -1 /tmp/digests | sed 's/^/- sha256:/' >> "$GITHUB_STEP_SUMMARY"
|
||||
4
.github/workflows/lint.yml
vendored
4
.github/workflows/lint.yml
vendored
@@ -13,8 +13,8 @@ on:
|
||||
- master
|
||||
|
||||
concurrency:
|
||||
group: ci-lint-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
group: ci-lint-${{ github.head_ref || github.ref }}-${{ github.repository }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
golangci-lint:
|
||||
|
||||
26
.github/workflows/test-extra.yml
vendored
26
.github/workflows/test-extra.yml
vendored
@@ -10,8 +10,8 @@ on:
|
||||
- '*'
|
||||
|
||||
concurrency:
|
||||
group: ci-tests-extra-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
group: ci-tests-extra-${{ github.head_ref || github.ref }}-${{ github.repository }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
detect-changes:
|
||||
@@ -43,7 +43,6 @@ jobs:
|
||||
insightface: ${{ steps.detect.outputs.insightface }}
|
||||
speaker-recognition: ${{ steps.detect.outputs.speaker-recognition }}
|
||||
sherpa-onnx: ${{ steps.detect.outputs.sherpa-onnx }}
|
||||
whisper: ${{ steps.detect.outputs.whisper }}
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v6
|
||||
@@ -584,27 +583,6 @@ jobs:
|
||||
- name: Build sherpa-onnx backend image and run streaming ASR gRPC e2e tests
|
||||
run: |
|
||||
make test-extra-backend-sherpa-onnx-transcription
|
||||
# End-to-end transcription via the e2e-backends gRPC harness against
|
||||
# the whisper.cpp backend. Drives AudioTranscription (offline) and
|
||||
# AudioTranscriptionStream (real, segment-callback-driven deltas) on
|
||||
# ggml-base.en + the JFK 11s clip.
|
||||
tests-whisper-grpc-transcription:
|
||||
needs: detect-changes
|
||||
if: needs.detect-changes.outputs.whisper == 'true' || needs.detect-changes.outputs.run-all == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 90
|
||||
steps:
|
||||
- name: Clone
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
submodules: true
|
||||
- name: Setup Go
|
||||
uses: actions/setup-go@v5
|
||||
with:
|
||||
go-version: '1.25.4'
|
||||
- name: Build whisper backend image and run transcription gRPC e2e tests
|
||||
run: |
|
||||
make test-extra-backend-whisper-transcription
|
||||
# VITS TTS via the sherpa-onnx backend. Drives both TTS (file write) and
|
||||
# TTSStream (PCM chunks) on the e2e-backends harness.
|
||||
tests-sherpa-onnx-grpc-tts:
|
||||
|
||||
58
.github/workflows/test.yml
vendored
58
.github/workflows/test.yml
vendored
@@ -3,6 +3,12 @@ name: 'tests'
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths-ignore:
|
||||
- 'docs/**'
|
||||
- 'examples/**'
|
||||
- 'README.md'
|
||||
- '**/*.md'
|
||||
- 'backend/**'
|
||||
push:
|
||||
branches:
|
||||
- master
|
||||
@@ -10,8 +16,8 @@ on:
|
||||
- '*'
|
||||
|
||||
concurrency:
|
||||
group: ci-tests-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
group: ci-tests-${{ github.head_ref || github.ref }}-${{ github.repository }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
tests-linux:
|
||||
@@ -20,12 +26,56 @@ jobs:
|
||||
matrix:
|
||||
go-version: ['1.26.x']
|
||||
steps:
|
||||
- name: Free Disk Space (Ubuntu)
|
||||
uses: jlumbroso/free-disk-space@main
|
||||
with:
|
||||
# this might remove tools that are actually needed,
|
||||
# if set to "true" but frees about 6 GB
|
||||
tool-cache: true
|
||||
# all of these default to true, but feel free to set to
|
||||
# "false" if necessary for your workflow
|
||||
android: true
|
||||
dotnet: true
|
||||
haskell: true
|
||||
large-packages: true
|
||||
docker-images: true
|
||||
swap-storage: true
|
||||
- name: Release space from worker
|
||||
run: |
|
||||
echo "Listing top largest packages"
|
||||
pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
|
||||
head -n 30 <<< "${pkgs}"
|
||||
echo
|
||||
df -h
|
||||
echo
|
||||
sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
|
||||
sudo apt-get remove --auto-remove android-sdk-platform-tools || true
|
||||
sudo apt-get purge --auto-remove android-sdk-platform-tools || true
|
||||
sudo rm -rf /usr/local/lib/android
|
||||
sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
|
||||
sudo rm -rf /usr/share/dotnet
|
||||
sudo apt-get remove -y '^mono-.*' || true
|
||||
sudo apt-get remove -y '^ghc-.*' || true
|
||||
sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
|
||||
sudo apt-get remove -y 'php.*' || true
|
||||
sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
|
||||
sudo apt-get remove -y '^google-.*' || true
|
||||
sudo apt-get remove -y azure-cli || true
|
||||
sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
|
||||
sudo apt-get remove -y '^gfortran-.*' || true
|
||||
sudo apt-get autoremove -y
|
||||
sudo apt-get clean
|
||||
echo
|
||||
echo "Listing top largest packages"
|
||||
pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
|
||||
head -n 30 <<< "${pkgs}"
|
||||
echo
|
||||
sudo rm -rfv build || true
|
||||
df -h
|
||||
- name: Clone
|
||||
uses: actions/checkout@v6
|
||||
with:
|
||||
submodules: true
|
||||
- name: Free disk space
|
||||
uses: ./.github/actions/free-disk-space
|
||||
- name: Setup Go ${{ matrix.go-version }}
|
||||
uses: actions/setup-go@v5
|
||||
with:
|
||||
|
||||
4
.github/workflows/tests-aio.yml
vendored
4
.github/workflows/tests-aio.yml
vendored
@@ -22,8 +22,8 @@ on:
|
||||
- '*'
|
||||
|
||||
concurrency:
|
||||
group: ci-tests-aio-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
group: ci-tests-aio-${{ github.head_ref || github.ref }}-${{ github.repository }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
tests-aio:
|
||||
|
||||
10
.github/workflows/tests-e2e.yml
vendored
10
.github/workflows/tests-e2e.yml
vendored
@@ -3,6 +3,12 @@ name: 'E2E Backend Tests'
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths-ignore:
|
||||
- 'docs/**'
|
||||
- 'examples/**'
|
||||
- 'README.md'
|
||||
- '**/*.md'
|
||||
- 'backend/**'
|
||||
push:
|
||||
branches:
|
||||
- master
|
||||
@@ -10,8 +16,8 @@ on:
|
||||
- '*'
|
||||
|
||||
concurrency:
|
||||
group: ci-tests-e2e-backend-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
group: ci-tests-e2e-backend-${{ github.head_ref || github.ref }}-${{ github.repository }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
tests-e2e-backend:
|
||||
|
||||
4
.github/workflows/tests-ui-e2e.yml
vendored
4
.github/workflows/tests-ui-e2e.yml
vendored
@@ -12,8 +12,8 @@ on:
|
||||
- master
|
||||
|
||||
concurrency:
|
||||
group: ci-tests-ui-e2e-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
group: ci-tests-ui-e2e-${{ github.head_ref || github.ref }}-${{ github.repository }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
tests-ui-e2e:
|
||||
|
||||
@@ -19,13 +19,11 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
|
||||
|------|-------------|
|
||||
| [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md) | Policy for AI-assisted contributions — licensing, DCO, attribution |
|
||||
| [.agents/building-and-testing.md](.agents/building-and-testing.md) | Building the project, running tests, Docker builds for specific platforms |
|
||||
| [.agents/ci-caching.md](.agents/ci-caching.md) | CI build cache layout (registry-backed BuildKit cache on quay.io/go-skynet/ci-cache, per-arch keys), `DEPS_REFRESH` weekly cache-buster for unpinned Python deps, prebuilt `base-grpc-*` images for llama.cpp variants, per-arch native + manifest-merge pattern, `setup-build-disk` `/mnt` relocation, path filter on master push, manual eviction |
|
||||
| [.agents/ci-caching.md](.agents/ci-caching.md) | CI build cache layout (registry-backed BuildKit cache on quay.io/go-skynet/ci-cache), `DEPS_REFRESH` weekly cache-buster for unpinned Python deps, manual eviction |
|
||||
| [.agents/adding-backends.md](.agents/adding-backends.md) | Adding a new backend (Python, Go, or C++) — full step-by-step checklist, including importer integration (the `/import-model` dropdown is server-driven from `GET /backends/known`) |
|
||||
| [.agents/coding-style.md](.agents/coding-style.md) | Code style, editorconfig, logging, documentation conventions |
|
||||
| [.agents/llama-cpp-backend.md](.agents/llama-cpp-backend.md) | Working on the llama.cpp backend — architecture, updating, tool call parsing |
|
||||
| [.agents/vllm-backend.md](.agents/vllm-backend.md) | Working on the vLLM / vLLM-omni backends — native parsers, ChatDelta, CPU build, libnuma packaging, backend hooks |
|
||||
| [.agents/sglang-backend.md](.agents/sglang-backend.md) | Working on the SGLang backend — `engine_args` validation against ServerArgs, speculative-decoding (EAGLE/EAGLE3/DFLASH/MTP) recipes, parser handling |
|
||||
| [.agents/ds4-backend.md](.agents/ds4-backend.md) | Working on the ds4 backend - DSML state machine, thinking modes, KV cache, Metal+CUDA matrix |
|
||||
| [.agents/testing-mcp-apps.md](.agents/testing-mcp-apps.md) | Testing MCP Apps (interactive tool UIs) in the React UI |
|
||||
| [.agents/api-endpoints-and-auth.md](.agents/api-endpoints-and-auth.md) | Adding API endpoints, auth middleware, feature permissions, user access control |
|
||||
| [.agents/debugging-backends.md](.agents/debugging-backends.md) | Debugging runtime backend failures, dependency conflicts, rebuilding backends |
|
||||
|
||||
117
Makefile
117
Makefile
@@ -1,5 +1,5 @@
|
||||
# Disable parallel execution for backend builds
|
||||
.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin
|
||||
.NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/outetts backends/piper backends/stablediffusion-ggml backends/whisper backends/faster-whisper backends/silero-vad backends/local-store backends/huggingface backends/rfdetr backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx
|
||||
|
||||
GOCMD=go
|
||||
GOTEST=$(GOCMD) test
|
||||
@@ -594,7 +594,6 @@ test-extra-backend-llama-cpp-transcription: docker-build-llama-cpp
|
||||
BACKEND_TEST_MMPROJ_URL=https://huggingface.co/ggml-org/Qwen3-ASR-0.6B-GGUF/resolve/main/mmproj-Qwen3-ASR-0.6B-Q8_0.gguf \
|
||||
BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
|
||||
BACKEND_TEST_CAPS=health,load,transcription \
|
||||
BACKEND_TEST_CTX_SIZE=2048 \
|
||||
$(MAKE) test-extra-backend
|
||||
|
||||
## vllm is resolved from a HuggingFace model id (no file download) and
|
||||
@@ -897,18 +896,6 @@ test-extra-backend-vibevoice-cpp-transcription: docker-build-vibevoice-cpp
|
||||
BACKEND_TEST_CAPS=health,load,transcription \
|
||||
$(MAKE) test-extra-backend
|
||||
|
||||
## Audio transcription wrapper for the whisper.cpp backend.
|
||||
## Drives the AudioTranscription / AudioTranscriptionStream RPCs against
|
||||
## ggml-base.en (~145 MB) using the JFK 11s clip. The streaming spec
|
||||
## asserts len(deltas) >= 1 and concat(deltas) == final.Text - whisper-
|
||||
## specific multi-segment assertions live in backend/go/whisper/gowhisper_test.go.
|
||||
test-extra-backend-whisper-transcription: docker-build-whisper
|
||||
BACKEND_IMAGE=local-ai-backend:whisper \
|
||||
BACKEND_TEST_MODEL_URL=https://huggingface.co/ggerganov/whisper.cpp/resolve/main/ggml-base.en.bin \
|
||||
BACKEND_TEST_AUDIO_URL=https://github.com/ggml-org/whisper.cpp/raw/master/samples/jfk.wav \
|
||||
BACKEND_TEST_CAPS=health,load,transcription \
|
||||
$(MAKE) test-extra-backend
|
||||
|
||||
## LocalVQE audio transform (joint AEC + noise suppression + dereverb).
|
||||
## Exercises the audio_transform capability end-to-end: batch transform
|
||||
## of a real WAV fixture and bidi streaming of synthetic silent frames.
|
||||
@@ -1009,10 +996,6 @@ backends/llama-cpp-darwin: build
|
||||
bash ./scripts/build/llama-cpp-darwin.sh
|
||||
./local-ai backends install "ocifile://$(abspath ./backend-images/llama-cpp.tar)"
|
||||
|
||||
backends/ds4-darwin: build
|
||||
bash ./scripts/build/ds4-darwin.sh
|
||||
./local-ai backends install "ocifile://$(abspath ./backend-images/ds4.tar)"
|
||||
|
||||
build-darwin-python-backend: build
|
||||
bash ./scripts/build/python-darwin.sh
|
||||
|
||||
@@ -1054,10 +1037,6 @@ BACKEND_IK_LLAMA_CPP = ik-llama-cpp|ik-llama-cpp|.|false|false
|
||||
# turboquant is a llama.cpp fork with TurboQuant KV-cache quantization.
|
||||
# Reuses backend/cpp/llama-cpp grpc-server sources via a thin wrapper Makefile.
|
||||
BACKEND_TURBOQUANT = turboquant|turboquant|.|false|false
|
||||
# ds4 is antirez/ds4, a DeepSeek V4 Flash-specific inference engine.
|
||||
# Single-model; hardware-only validation lives at tests/e2e-backends/
|
||||
# (BACKEND_BINARY mode); see docs/superpowers/plans/2026-05-11-ds4-backend.md.
|
||||
BACKEND_DS4 = ds4|ds4|.|false|false
|
||||
|
||||
# Golang backends
|
||||
BACKEND_PIPER = piper|golang|.|false|true
|
||||
@@ -1115,6 +1094,90 @@ BACKEND_KOKOROS = kokoros|rust|.|false|true
|
||||
# C++ backends (Go wrapper with purego)
|
||||
BACKEND_SAM3_CPP = sam3-cpp|golang|.|false|true
|
||||
|
||||
# Tag stem for the local prebuilt base images. Mirrors tagStem() in
|
||||
# scripts/changed-backends.js and the inline expression in
|
||||
# .github/workflows/backend.yml, so a `make docker-build-X` produces the
|
||||
# same FROM ref shape that CI uses.
|
||||
LOCAL_BASE_BUILD_TYPE := $(or $(BUILD_TYPE),cpu)
|
||||
LOCAL_BASE_UBUNTU_VERSION := $(or $(UBUNTU_VERSION),2404)
|
||||
LOCAL_BASE_CUDA_SUFFIX := $(if $(filter cublas l4t,$(BUILD_TYPE)),-cuda$(CUDA_MAJOR_VERSION).$(CUDA_MINOR_VERSION))
|
||||
LOCAL_BASE_PYTHON_TAG := localai-base:python-$(LOCAL_BASE_BUILD_TYPE)-$(LOCAL_BASE_UBUNTU_VERSION)$(LOCAL_BASE_CUDA_SUFFIX)
|
||||
LOCAL_BASE_GOLANG_TAG := localai-base:golang-$(LOCAL_BASE_BUILD_TYPE)-$(LOCAL_BASE_UBUNTU_VERSION)$(LOCAL_BASE_CUDA_SUFFIX)
|
||||
LOCAL_BASE_CPP_TAG := localai-base:cpp-$(LOCAL_BASE_BUILD_TYPE)-$(LOCAL_BASE_UBUNTU_VERSION)$(LOCAL_BASE_CUDA_SUFFIX)
|
||||
LOCAL_BASE_RUST_TAG := localai-base:rust-$(LOCAL_BASE_BUILD_TYPE)-$(LOCAL_BASE_UBUNTU_VERSION)
|
||||
|
||||
# Per-(lang) base image build targets. Each backend's docker-build-X target
|
||||
# depends on the matching base via generate-docker-build-target below.
|
||||
# PHONY so docker handles its own layer caching.
|
||||
.PHONY: docker-build-python-base docker-build-golang-base docker-build-cpp-base docker-build-rust-base
|
||||
|
||||
docker-build-python-base:
|
||||
docker build \
|
||||
--build-arg BUILD_TYPE=$(BUILD_TYPE) \
|
||||
--build-arg BASE_IMAGE=$(or $(BASE_IMAGE),ubuntu:24.04) \
|
||||
--build-arg CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) \
|
||||
--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
|
||||
--build-arg UBUNTU_VERSION=$(LOCAL_BASE_UBUNTU_VERSION) \
|
||||
--build-arg APT_MIRROR=$(APT_MIRROR) \
|
||||
--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
|
||||
$(if $(SKIP_DRIVERS),--build-arg SKIP_DRIVERS=$(SKIP_DRIVERS)) \
|
||||
-t $(LOCAL_BASE_PYTHON_TAG) \
|
||||
-f .docker/bases/Dockerfile.python \
|
||||
.
|
||||
|
||||
docker-build-golang-base:
|
||||
docker build \
|
||||
--build-arg BUILD_TYPE=$(BUILD_TYPE) \
|
||||
--build-arg BASE_IMAGE=$(or $(BASE_IMAGE),ubuntu:24.04) \
|
||||
--build-arg CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) \
|
||||
--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
|
||||
--build-arg UBUNTU_VERSION=$(LOCAL_BASE_UBUNTU_VERSION) \
|
||||
--build-arg APT_MIRROR=$(APT_MIRROR) \
|
||||
--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
|
||||
$(if $(SKIP_DRIVERS),--build-arg SKIP_DRIVERS=$(SKIP_DRIVERS)) \
|
||||
-t $(LOCAL_BASE_GOLANG_TAG) \
|
||||
-f .docker/bases/Dockerfile.golang \
|
||||
.
|
||||
|
||||
docker-build-cpp-base:
|
||||
docker build \
|
||||
--build-arg BUILD_TYPE=$(BUILD_TYPE) \
|
||||
--build-arg BASE_IMAGE=$(or $(BASE_IMAGE),ubuntu:24.04) \
|
||||
--build-arg CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) \
|
||||
--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
|
||||
--build-arg UBUNTU_VERSION=$(LOCAL_BASE_UBUNTU_VERSION) \
|
||||
--build-arg APT_MIRROR=$(APT_MIRROR) \
|
||||
--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
|
||||
$(if $(SKIP_DRIVERS),--build-arg SKIP_DRIVERS=$(SKIP_DRIVERS)) \
|
||||
-t $(LOCAL_BASE_CPP_TAG) \
|
||||
-f .docker/bases/Dockerfile.cpp \
|
||||
.
|
||||
|
||||
docker-build-rust-base:
|
||||
docker build \
|
||||
--build-arg BASE_IMAGE=$(or $(BASE_IMAGE),ubuntu:24.04) \
|
||||
--build-arg UBUNTU_VERSION=$(LOCAL_BASE_UBUNTU_VERSION) \
|
||||
--build-arg APT_MIRROR=$(APT_MIRROR) \
|
||||
--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
|
||||
-t $(LOCAL_BASE_RUST_TAG) \
|
||||
-f .docker/bases/Dockerfile.rust \
|
||||
.
|
||||
|
||||
# Map a consumer dockerfile-type to the base-image tag it should consume.
|
||||
# Mirrors langOf() in scripts/changed-backends.js: the C++ trio
|
||||
# (llama-cpp/ik-llama-cpp/turboquant) all consume the shared cpp base.
|
||||
local-base-tag = $(strip \
|
||||
$(if $(filter python,$(1)),$(LOCAL_BASE_PYTHON_TAG), \
|
||||
$(if $(filter golang,$(1)),$(LOCAL_BASE_GOLANG_TAG), \
|
||||
$(if $(filter llama-cpp ik-llama-cpp turboquant,$(1)),$(LOCAL_BASE_CPP_TAG), \
|
||||
$(if $(filter rust,$(1)),$(LOCAL_BASE_RUST_TAG))))))
|
||||
|
||||
local-base-target = $(strip \
|
||||
$(if $(filter python,$(1)),docker-build-python-base, \
|
||||
$(if $(filter golang,$(1)),docker-build-golang-base, \
|
||||
$(if $(filter llama-cpp ik-llama-cpp turboquant,$(1)),docker-build-cpp-base, \
|
||||
$(if $(filter rust,$(1)),docker-build-rust-base)))))
|
||||
|
||||
# Helper function to build docker image for a backend
|
||||
# Usage: $(call docker-build-backend,BACKEND_NAME,DOCKERFILE_TYPE,BUILD_CONTEXT,PROGRESS_FLAG,NEEDS_BACKEND_ARG)
|
||||
define docker-build-backend
|
||||
@@ -1127,15 +1190,18 @@ define docker-build-backend
|
||||
--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
|
||||
--build-arg APT_MIRROR=$(APT_MIRROR) \
|
||||
--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
|
||||
$(if $(call local-base-tag,$(2)),--build-arg BASE_IMAGE_PREBUILT=$(call local-base-tag,$(2))) \
|
||||
$(if $(FROM_SOURCE),--build-arg FROM_SOURCE=$(FROM_SOURCE)) \
|
||||
$(if $(AMDGPU_TARGETS),--build-arg AMDGPU_TARGETS=$(AMDGPU_TARGETS)) \
|
||||
$(if $(filter true,$(5)),--build-arg BACKEND=$(1)) \
|
||||
-t local-ai-backend:$(1) -f backend/Dockerfile.$(2) $(3)
|
||||
endef
|
||||
|
||||
# Generate docker-build targets from backend definitions
|
||||
# Generate docker-build targets from backend definitions. Each consumer
|
||||
# gets the matching layered base as a prerequisite so the FROM in the
|
||||
# slimmed Dockerfile resolves locally. The map lives in local-base-target.
|
||||
define generate-docker-build-target
|
||||
docker-build-$(word 1,$(subst |, ,$(1))):
|
||||
docker-build-$(word 1,$(subst |, ,$(1))): $(call local-base-target,$(word 2,$(subst |, ,$(1))))
|
||||
$$(call docker-build-backend,$(word 1,$(subst |, ,$(1))),$(word 2,$(subst |, ,$(1))),$(word 3,$(subst |, ,$(1))),$(word 4,$(subst |, ,$(1))),$(word 5,$(subst |, ,$(1))))
|
||||
endef
|
||||
|
||||
@@ -1143,7 +1209,6 @@ endef
|
||||
$(eval $(call generate-docker-build-target,$(BACKEND_LLAMA_CPP)))
|
||||
$(eval $(call generate-docker-build-target,$(BACKEND_IK_LLAMA_CPP)))
|
||||
$(eval $(call generate-docker-build-target,$(BACKEND_TURBOQUANT)))
|
||||
$(eval $(call generate-docker-build-target,$(BACKEND_DS4)))
|
||||
$(eval $(call generate-docker-build-target,$(BACKEND_PIPER)))
|
||||
$(eval $(call generate-docker-build-target,$(BACKEND_LOCAL_STORE)))
|
||||
$(eval $(call generate-docker-build-target,$(BACKEND_HUGGINGFACE)))
|
||||
@@ -1197,7 +1262,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_SHERPA_ONNX)))
|
||||
docker-save-%: backend-images
|
||||
docker save local-ai-backend:$* -o backend-images/$*.tar
|
||||
|
||||
docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-qwen3-tts-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx
|
||||
docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-qwen3-tts-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx
|
||||
|
||||
########################################################
|
||||
### Mock Backend for E2E Tests
|
||||
|
||||
@@ -1,98 +0,0 @@
|
||||
# syntax=docker/dockerfile:1.7
|
||||
#
|
||||
# Pre-built builder base image for LocalAI's C++ backends.
|
||||
#
|
||||
# This Dockerfile is the source of truth for the
|
||||
# `quay.io/go-skynet/ci-cache:base-grpc-*` images that
|
||||
# `.github/workflows/base-images.yml` builds and pushes. The output of a
|
||||
# build is a fully-prepped builder layer containing:
|
||||
#
|
||||
# - apt build deps (build-essential, ccache, git, make, pkg-config,
|
||||
# libcurl4-openssl-dev, libssl-dev, curl, unzip, wget, ca-certificates)
|
||||
# - cmake (apt or, when CMAKE_FROM_SOURCE=true, compiled from
|
||||
# ${CMAKE_VERSION})
|
||||
# - protoc v27.1 at /usr/local/bin/protoc
|
||||
# - gRPC ${GRPC_VERSION} compiled and installed at /opt/grpc
|
||||
# - Conditional CUDA toolkit (BUILD_TYPE=cublas|l4t, SKIP_DRIVERS=false)
|
||||
# including the cuda-13 + arm64 cudss/nvpl special case
|
||||
# - Conditional ROCm/HIP build deps (BUILD_TYPE=hipblas)
|
||||
# - Conditional Vulkan SDK 1.4.335.0 (BUILD_TYPE=vulkan)
|
||||
#
|
||||
# Variants built by the workflow (matrix in base-images.yml):
|
||||
#
|
||||
# base-grpc-amd64 ubuntu:24.04, CPU-only
|
||||
# base-grpc-arm64 ubuntu:24.04, CPU-only
|
||||
# base-grpc-cuda-12-amd64 ubuntu:24.04 + CUDA 12.8
|
||||
# base-grpc-cuda-13-amd64 ubuntu:22.04 + CUDA 13.0
|
||||
# base-grpc-cuda-13-arm64 ubuntu:24.04 + CUDA 13.0 (sbsa)
|
||||
# base-grpc-l4t-cuda-12-arm64 ubuntu:22.04 + CUDA 12.x (legacy JetPack)
|
||||
# base-grpc-rocm-amd64 rocm/dev-ubuntu-24.04:7.2.1 + hipblas
|
||||
# base-grpc-vulkan-amd64 ubuntu:24.04 + Vulkan SDK 1.4.335
|
||||
# base-grpc-vulkan-arm64 ubuntu:24.04 + Vulkan SDK ARM 1.4.335
|
||||
# base-grpc-intel-amd64 intel/oneapi-basekit:2025.3.2 (sycl)
|
||||
#
|
||||
# This is a SINGLE-stage Dockerfile by design: the final image IS the
|
||||
# builder base. The intermediate gRPC compile happens inside this same
|
||||
# stage so consumer Dockerfiles in PR 2 can simply
|
||||
# `FROM quay.io/go-skynet/ci-cache:base-grpc-<variant>` without needing a
|
||||
# COPY --from=grpc step. /opt/grpc is the canonical install prefix and
|
||||
# downstream builds will add it to CMAKE_PREFIX_PATH (or copy to
|
||||
# /usr/local) the same way Dockerfile.llama-cpp does today.
|
||||
#
|
||||
# Install logic lives in .docker/install-base-deps.sh, which is also
|
||||
# bind-mounted by the variant Dockerfiles' builder-fromsource stage.
|
||||
# This guarantees bit-equivalence between the prebuilt CI base and the
|
||||
# from-source local-dev path — both invoke the same script with the
|
||||
# same env inputs.
|
||||
|
||||
ARG BASE_IMAGE=ubuntu:24.04
|
||||
|
||||
FROM ${BASE_IMAGE}
|
||||
|
||||
ARG BASE_IMAGE=ubuntu:24.04
|
||||
ARG BUILD_TYPE=""
|
||||
ARG CUDA_MAJOR_VERSION=""
|
||||
ARG CUDA_MINOR_VERSION=""
|
||||
ARG CMAKE_FROM_SOURCE=false
|
||||
# CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain
|
||||
# detection / arch table issues.
|
||||
ARG CMAKE_VERSION=3.31.10
|
||||
ARG GRPC_VERSION=v1.65.0
|
||||
ARG GRPC_MAKEFLAGS="-j4 -Otarget"
|
||||
ARG SKIP_DRIVERS=false
|
||||
ARG TARGETARCH
|
||||
ARG UBUNTU_VERSION=2404
|
||||
ARG APT_MIRROR=""
|
||||
ARG APT_PORTS_MIRROR=""
|
||||
ARG AMDGPU_TARGETS=""
|
||||
|
||||
ENV BUILD_TYPE=${BUILD_TYPE} \
|
||||
CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
|
||||
CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
|
||||
CMAKE_FROM_SOURCE=${CMAKE_FROM_SOURCE} \
|
||||
CMAKE_VERSION=${CMAKE_VERSION} \
|
||||
GRPC_VERSION=${GRPC_VERSION} \
|
||||
GRPC_MAKEFLAGS=${GRPC_MAKEFLAGS} \
|
||||
SKIP_DRIVERS=${SKIP_DRIVERS} \
|
||||
TARGETARCH=${TARGETARCH} \
|
||||
UBUNTU_VERSION=${UBUNTU_VERSION} \
|
||||
APT_MIRROR=${APT_MIRROR} \
|
||||
APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
|
||||
AMDGPU_TARGETS=${AMDGPU_TARGETS} \
|
||||
MAKEFLAGS=${GRPC_MAKEFLAGS} \
|
||||
DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
# CUDA on PATH (no-op when CUDA isn't installed)
|
||||
ENV PATH=/usr/local/cuda/bin:${PATH}
|
||||
# HipBLAS / ROCm on PATH (no-op when ROCm isn't installed)
|
||||
ENV PATH=/opt/rocm/bin:${PATH}
|
||||
|
||||
WORKDIR /build
|
||||
|
||||
# Single RUN that delegates to .docker/install-base-deps.sh — the same
|
||||
# script the variant Dockerfiles' builder-fromsource stage runs.
|
||||
RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
|
||||
--mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
|
||||
bash /usr/local/sbin/install-base-deps
|
||||
|
||||
WORKDIR /
|
||||
@@ -1,41 +0,0 @@
|
||||
ARG BASE_IMAGE=ubuntu:24.04
|
||||
ARG APT_MIRROR=""
|
||||
ARG APT_PORTS_MIRROR=""
|
||||
|
||||
# BASE_IMAGE is either ubuntu:24.04 (for cpu builds) or nvidia/cuda:13.0.0-devel-ubuntu24.04
|
||||
# (for cublas builds). Both ship apt + Ubuntu Noble packages; the nvidia/cuda base
|
||||
# additionally provides /usr/local/cuda. Darwin (Metal) builds bypass this Dockerfile
|
||||
# entirely via scripts/build/ds4-darwin.sh.
|
||||
FROM ${BASE_IMAGE} AS builder
|
||||
ARG BUILD_TYPE
|
||||
ARG TARGETARCH
|
||||
ARG TARGETVARIANT
|
||||
|
||||
ENV BUILD_TYPE=${BUILD_TYPE} \
|
||||
DEBIAN_FRONTEND=noninteractive \
|
||||
PATH=/usr/local/cuda/bin:${PATH}
|
||||
|
||||
WORKDIR /build
|
||||
|
||||
# Install build-time deps via plain apt - install-base-deps.sh's full pipeline
|
||||
# (CUDA keyring + from-source gRPC) is unnecessary here:
|
||||
# - CUDA: when BASE_IMAGE=nvidia/cuda:*, /usr/local/cuda is already populated;
|
||||
# for the cpu build we don't need CUDA at all.
|
||||
# - gRPC/Protobuf: system apt packages are sufficient; ds4's wrapper only links
|
||||
# against them, it doesn't ship the gRPC source tree.
|
||||
# - nlohmann-json: dsml_renderer's only third-party dep.
|
||||
RUN apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
git cmake build-essential pkg-config ca-certificates \
|
||||
libgrpc++-dev libprotobuf-dev protobuf-compiler protobuf-compiler-grpc \
|
||||
nlohmann-json3-dev && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
COPY . /LocalAI
|
||||
|
||||
RUN --mount=type=cache,target=/root/.ccache,id=ds4-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
|
||||
make -C /LocalAI/backend/cpp/ds4 BUILD_TYPE=${BUILD_TYPE} NATIVE=false grpc-server package
|
||||
|
||||
FROM scratch
|
||||
COPY --from=builder /LocalAI/backend/cpp/ds4/package/. ./
|
||||
@@ -1,210 +1,37 @@
|
||||
ARG BASE_IMAGE=ubuntu:24.04
|
||||
ARG APT_MIRROR=""
|
||||
ARG APT_PORTS_MIRROR=""
|
||||
# Builds a single Go backend on top of the shared
|
||||
# .docker/bases/Dockerfile.golang base. The base bakes in apt + GPU SDK +
|
||||
# Go toolchain + protoc + grpc tooling, so this stage only carries the
|
||||
# per-backend opus-dev install + COPY + `make build`.
|
||||
#
|
||||
# CI orchestration (.github/workflows/backend.yml + backend_pr.yml) builds
|
||||
# the right base flavour automatically via scripts/changed-backends.js
|
||||
# and passes BASE_IMAGE_PREBUILT here. For local builds, run:
|
||||
# make backend-image-base LANG=golang BUILD_TYPE=<...>
|
||||
# make backend-image BACKEND=<...> BUILD_TYPE=<...>
|
||||
# See .agents/ci-caching.md.
|
||||
|
||||
ARG BASE_IMAGE_PREBUILT
|
||||
|
||||
FROM ${BASE_IMAGE_PREBUILT} AS builder
|
||||
|
||||
FROM ${BASE_IMAGE} AS builder
|
||||
ARG BACKEND=rerankers
|
||||
ARG BUILD_TYPE
|
||||
ENV BUILD_TYPE=${BUILD_TYPE}
|
||||
ARG CUDA_MAJOR_VERSION
|
||||
ARG CUDA_MINOR_VERSION
|
||||
ARG SKIP_DRIVERS=false
|
||||
ENV CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION}
|
||||
ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
ARG TARGETARCH
|
||||
ARG TARGETVARIANT
|
||||
ARG GO_VERSION=1.25.4
|
||||
ARG UBUNTU_VERSION=2404
|
||||
ARG AMDGPU_TARGETS
|
||||
ENV AMDGPU_TARGETS=${AMDGPU_TARGETS}
|
||||
ARG APT_MIRROR
|
||||
ARG APT_PORTS_MIRROR
|
||||
|
||||
# gcc-14 is the default on noble (ubuntu:24.04) but absent from jammy
|
||||
# (the L4T jetpack r36.4.0 base). LocalVQE specifically needs it; the
|
||||
# other Go backends compile fine with the default gcc shipped via
|
||||
# build-essential. So: try gcc-14 from the configured repos, fall back
|
||||
# gracefully when it's not available so jammy-based builds don't fail
|
||||
# at the apt step.
|
||||
RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
|
||||
APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
git ccache \
|
||||
ca-certificates \
|
||||
make cmake wget libopenblas-dev \
|
||||
curl unzip \
|
||||
libssl-dev && \
|
||||
if apt-cache show gcc-14 >/dev/null 2>&1 && apt-cache show g++-14 >/dev/null 2>&1; then \
|
||||
apt-get install -y --no-install-recommends gcc-14 g++-14 && \
|
||||
update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-14 100 \
|
||||
--slave /usr/bin/g++ g++ /usr/bin/g++-14 \
|
||||
--slave /usr/bin/gcov gcov /usr/bin/gcov-14; \
|
||||
fi && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
|
||||
# Cuda
|
||||
ENV PATH=/usr/local/cuda/bin:${PATH}
|
||||
|
||||
# HipBLAS requirements
|
||||
ENV PATH=/opt/rocm/bin:${PATH}
|
||||
|
||||
|
||||
# Vulkan requirements
|
||||
RUN <<EOT bash
|
||||
if [ "${BUILD_TYPE}" = "vulkan" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
software-properties-common pciutils wget gpg-agent && \
|
||||
apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
|
||||
libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
|
||||
libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
|
||||
git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
|
||||
ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
|
||||
clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
|
||||
if [ "amd64" = "$TARGETARCH" ]; then
|
||||
wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz" && \
|
||||
tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
|
||||
rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
|
||||
mkdir -p /opt/vulkan-sdk && \
|
||||
mv 1.4.335.0 /opt/vulkan-sdk/ && \
|
||||
cd /opt/vulkan-sdk/1.4.335.0 && \
|
||||
./vulkansdk --no-deps --maxjobs \
|
||||
vulkan-loader \
|
||||
vulkan-validationlayers \
|
||||
vulkan-extensionlayer \
|
||||
vulkan-tools \
|
||||
shaderc && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/ && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/ && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/ && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/ && \
|
||||
rm -rf /opt/vulkan-sdk
|
||||
fi
|
||||
if [ "arm64" = "$TARGETARCH" ]; then
|
||||
mkdir vulkan && cd vulkan && \
|
||||
curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
|
||||
tar -xvf vulkan-sdk.tar.xz && \
|
||||
rm vulkan-sdk.tar.xz && \
|
||||
cd 1.4.335.0 && \
|
||||
cp -rfv aarch64/bin/* /usr/bin/ && \
|
||||
cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
|
||||
cp -rfv aarch64/include/* /usr/include/ && \
|
||||
cp -rfv aarch64/share/* /usr/share/ && \
|
||||
cd ../.. && \
|
||||
rm -rf vulkan
|
||||
fi
|
||||
ldconfig && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
EOT
|
||||
|
||||
# CuBLAS requirements
|
||||
RUN <<EOT bash
|
||||
if ( [ "${BUILD_TYPE}" = "cublas" ] || [ "${BUILD_TYPE}" = "l4t" ] ) && [ "${SKIP_DRIVERS}" = "false" ]; then
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
software-properties-common pciutils
|
||||
if [ "amd64" = "$TARGETARCH" ]; then
|
||||
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb
|
||||
fi
|
||||
if [ "arm64" = "$TARGETARCH" ]; then
|
||||
if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
|
||||
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb
|
||||
else
|
||||
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb
|
||||
fi
|
||||
fi
|
||||
dpkg -i cuda-keyring_1.1-1_all.deb && \
|
||||
rm -f cuda-keyring_1.1-1_all.deb && \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
|
||||
if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "$TARGETARCH" ]; then
|
||||
apt-get install -y --no-install-recommends \
|
||||
libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libcudnn9-cuda-${CUDA_MAJOR_VERSION} cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
|
||||
fi
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
EOT
|
||||
|
||||
|
||||
# https://github.com/NVIDIA/Isaac-GR00T/issues/343
|
||||
RUN <<EOT bash
|
||||
if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "arm64" ]; then
|
||||
wget https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
|
||||
dpkg -i cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
|
||||
cp /var/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/ && \
|
||||
apt-get update && apt-get -y install cudss cudss-cuda-${CUDA_MAJOR_VERSION} && \
|
||||
wget https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
|
||||
dpkg -i nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
|
||||
cp /var/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/ && \
|
||||
apt-get update && apt-get install -y nvpl
|
||||
fi
|
||||
EOT
|
||||
|
||||
# If we are building with clblas support, we need the libraries for the builds
|
||||
RUN if [ "${BUILD_TYPE}" = "clblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
libclblast-dev && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/* \
|
||||
; fi
|
||||
|
||||
RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
hipblas-dev \
|
||||
hipblaslt-dev \
|
||||
rocblas-dev && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/* && \
|
||||
# I have no idea why, but the ROCM lib packages don't trigger ldconfig after they install, which results in local-ai and others not being able
|
||||
# to locate the libraries. We run ldconfig ourselves to work around this packaging deficiency
|
||||
ldconfig \
|
||||
; fi
|
||||
|
||||
# Install Go
|
||||
RUN curl -L -s https://go.dev/dl/go${GO_VERSION}.linux-${TARGETARCH}.tar.gz | tar -C /usr/local -xz
|
||||
ENV PATH=$PATH:/root/go/bin:/usr/local/go/bin:/usr/local/bin
|
||||
|
||||
# Install grpc compilers
|
||||
RUN go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2 && \
|
||||
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
|
||||
RUN echo "TARGETARCH: $TARGETARCH"
|
||||
|
||||
# We need protoc installed, and the version in 22.04 is too old. We will create one as part installing the GRPC build below
|
||||
# but that will also being in a newer version of absl which stablediffusion cannot compile with. This version of protoc is only
|
||||
# here so that we can generate the grpc code for the stablediffusion build
|
||||
RUN <<EOT bash
|
||||
if [ "amd64" = "$TARGETARCH" ]; then
|
||||
curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-x86_64.zip -o protoc.zip && \
|
||||
unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
|
||||
rm protoc.zip
|
||||
fi
|
||||
if [ "arm64" = "$TARGETARCH" ]; then
|
||||
curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-aarch_64.zip -o protoc.zip && \
|
||||
unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
|
||||
rm protoc.zip
|
||||
fi
|
||||
EOT
|
||||
|
||||
# opus-dev is only needed for the opus backend; install on demand to keep
|
||||
# every other golang backend's base image lean.
|
||||
RUN if [ "${BACKEND}" = "opus" ]; then \
|
||||
apt-get update && apt-get install -y --no-install-recommends libopus-dev pkg-config && \
|
||||
apt-get clean && rm -rf /var/lib/apt/lists/*; \
|
||||
fi
|
||||
apt-get update && apt-get install -y --no-install-recommends libopus-dev pkg-config && \
|
||||
apt-get clean && rm -rf /var/lib/apt/lists/*; \
|
||||
fi
|
||||
|
||||
COPY . /LocalAI
|
||||
|
||||
|
||||
@@ -1,151 +1,54 @@
|
||||
ARG BASE_IMAGE=ubuntu:24.04
|
||||
# BUILDER_BASE_IMAGE defaults to BASE_IMAGE so the Dockerfile parses even
|
||||
# when no prebuilt base is supplied. The builder-prebuilt stage is only
|
||||
# entered when BUILDER_TARGET=builder-prebuilt, so a "wrong" fallback
|
||||
# content here is harmless — BuildKit prunes the unreferenced builder.
|
||||
ARG BUILDER_BASE_IMAGE=${BASE_IMAGE}
|
||||
# BUILDER_TARGET selects which builder stage the final scratch image copies
|
||||
# package output from. Declared at global scope (before any FROM) so it's
|
||||
# usable in `FROM ${BUILDER_TARGET}` below. Default keeps local
|
||||
# `make backends/ik-llama-cpp` on the from-source path.
|
||||
ARG BUILDER_TARGET=builder-fromsource
|
||||
ARG APT_MIRROR=""
|
||||
ARG APT_PORTS_MIRROR=""
|
||||
# Builds the ik-llama-cpp backend on top of the shared
|
||||
# .docker/bases/Dockerfile.cpp base (shared with llama-cpp/turboquant).
|
||||
# See backend/Dockerfile.llama-cpp for the rationale; this consumer differs
|
||||
# only in the make targets at the end.
|
||||
|
||||
ARG BASE_IMAGE_PREBUILT
|
||||
|
||||
# ============================================================================
|
||||
# Stage: builder-fromsource — self-contained build path.
|
||||
# Runs .docker/install-base-deps.sh (apt deps + cmake + protoc + gRPC +
|
||||
# conditional CUDA/ROCm/Vulkan), copies /opt/grpc to /usr/local, then
|
||||
# compiles the variant. Used when BUILDER_TARGET=builder-fromsource (the
|
||||
# default; local `make backends/ik-llama-cpp`).
|
||||
#
|
||||
# The install script is the same one that backend/Dockerfile.base-grpc-builder
|
||||
# runs, so the result is bit-equivalent to the prebuilt-base path
|
||||
# (builder-prebuilt below).
|
||||
# ============================================================================
|
||||
FROM ${BASE_IMAGE} AS builder-fromsource
|
||||
ARG BUILD_TYPE
|
||||
ARG CUDA_MAJOR_VERSION
|
||||
ARG CUDA_MINOR_VERSION
|
||||
ARG CMAKE_FROM_SOURCE=false
|
||||
# CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
|
||||
ARG CMAKE_VERSION=3.31.10
|
||||
ARG GRPC_VERSION=v1.65.0
|
||||
ARG GRPC_MAKEFLAGS="-j4 -Otarget"
|
||||
ARG SKIP_DRIVERS=false
|
||||
ARG TARGETARCH
|
||||
ARG TARGETVARIANT
|
||||
ARG GO_VERSION=1.25.4
|
||||
ARG UBUNTU_VERSION=2404
|
||||
ARG APT_MIRROR
|
||||
ARG APT_PORTS_MIRROR
|
||||
ARG AMDGPU_TARGETS=""
|
||||
ARG BACKEND=rerankers
|
||||
# CUDA target archs, e.g. --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
|
||||
ARG CUDA_DOCKER_ARCH
|
||||
ARG CMAKE_ARGS
|
||||
FROM ${BASE_IMAGE_PREBUILT} AS builder
|
||||
|
||||
ENV BUILD_TYPE=${BUILD_TYPE} \
|
||||
CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
|
||||
CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
|
||||
CMAKE_FROM_SOURCE=${CMAKE_FROM_SOURCE} \
|
||||
CMAKE_VERSION=${CMAKE_VERSION} \
|
||||
GRPC_VERSION=${GRPC_VERSION} \
|
||||
GRPC_MAKEFLAGS=${GRPC_MAKEFLAGS} \
|
||||
SKIP_DRIVERS=${SKIP_DRIVERS} \
|
||||
TARGETARCH=${TARGETARCH} \
|
||||
UBUNTU_VERSION=${UBUNTU_VERSION} \
|
||||
APT_MIRROR=${APT_MIRROR} \
|
||||
APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
|
||||
AMDGPU_TARGETS=${AMDGPU_TARGETS} \
|
||||
CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH} \
|
||||
CMAKE_ARGS=${CMAKE_ARGS} \
|
||||
DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
# CUDA on PATH (no-op when CUDA isn't installed)
|
||||
ENV PATH=/usr/local/cuda/bin:${PATH}
|
||||
# HipBLAS / ROCm on PATH (no-op when ROCm isn't installed)
|
||||
ENV PATH=/opt/rocm/bin:${PATH}
|
||||
|
||||
WORKDIR /build
|
||||
|
||||
# Install everything via the shared script — the same one that
|
||||
# backend/Dockerfile.base-grpc-builder runs, so the prebuilt CI base and
|
||||
# this from-source path are bit-equivalent.
|
||||
RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
|
||||
--mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
|
||||
bash /usr/local/sbin/install-base-deps
|
||||
|
||||
# Mirror builder-prebuilt: copy gRPC from /opt/grpc to /usr/local so
|
||||
# CMake's find_package finds it at the canonical prefix the Makefile expects.
|
||||
RUN cp -a /opt/grpc/. /usr/local/
|
||||
|
||||
COPY . /LocalAI
|
||||
|
||||
# BuildKit cache mount for ccache. See Dockerfile.llama-cpp (commit 9228e5b4)
|
||||
# for the rationale. Distinct mount id so ik-llama-cpp's cache doesn't
|
||||
# overlap with llama-cpp's — ik_llama.cpp is a different fork with
|
||||
# different source.
|
||||
#
|
||||
# The compile body is shared with builder-prebuilt via .docker/ik-llama-cpp-compile.sh.
|
||||
RUN --mount=type=bind,source=.docker/ik-llama-cpp-compile.sh,target=/usr/local/sbin/compile.sh \
|
||||
--mount=type=cache,target=/root/.ccache,id=ik-llama-cpp-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
|
||||
bash /usr/local/sbin/compile.sh
|
||||
|
||||
|
||||
# Copy libraries using a script to handle architecture differences
|
||||
RUN make -BC /LocalAI/backend/cpp/ik-llama-cpp package
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Stage: builder-prebuilt — uses the pre-built base from
|
||||
# quay.io/go-skynet/ci-cache:base-grpc-* (built by .github/workflows/base-images.yml).
|
||||
# That image already has gRPC at /opt/grpc + apt deps + CUDA/ROCm/Vulkan
|
||||
# pre-installed, so we just copy gRPC to /usr/local and compile. Used when
|
||||
# BUILDER_TARGET=builder-prebuilt (CI when the matrix entry sets
|
||||
# builder-base-image).
|
||||
# ============================================================================
|
||||
FROM ${BUILDER_BASE_IMAGE} AS builder-prebuilt
|
||||
|
||||
ARG BUILD_TYPE
|
||||
ENV BUILD_TYPE=${BUILD_TYPE}
|
||||
ARG CUDA_DOCKER_ARCH
|
||||
ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
|
||||
ARG CMAKE_ARGS
|
||||
ENV CMAKE_ARGS=${CMAKE_ARGS}
|
||||
ARG BACKEND=ik-llama-cpp
|
||||
ARG BUILD_TYPE
|
||||
ENV BUILD_TYPE=${BUILD_TYPE}
|
||||
ARG CUDA_MAJOR_VERSION
|
||||
ARG CUDA_MINOR_VERSION
|
||||
ENV CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION}
|
||||
ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
|
||||
ARG TARGETARCH
|
||||
ARG TARGETVARIANT
|
||||
|
||||
# The base-grpc-* image installs gRPC to /opt/grpc but doesn't copy it to
|
||||
# /usr/local. Mirror what the from-source path does so the compile step
|
||||
# can find gRPC at the canonical prefix the Makefile expects.
|
||||
RUN cp -a /opt/grpc/. /usr/local/
|
||||
|
||||
COPY . /LocalAI
|
||||
|
||||
RUN --mount=type=bind,source=.docker/ik-llama-cpp-compile.sh,target=/usr/local/sbin/compile.sh \
|
||||
--mount=type=cache,target=/root/.ccache,id=ik-llama-cpp-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
|
||||
bash /usr/local/sbin/compile.sh
|
||||
RUN <<'EOT' bash
|
||||
set -euxo pipefail
|
||||
|
||||
if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
|
||||
CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
|
||||
export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
|
||||
echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
|
||||
rm -rf /LocalAI/backend/cpp/ik-llama-cpp-*-build
|
||||
fi
|
||||
|
||||
cd /LocalAI/backend/cpp/ik-llama-cpp
|
||||
|
||||
if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
|
||||
# ARM64 / ROCm: build without x86 SIMD
|
||||
make ik-llama-cpp-fallback
|
||||
else
|
||||
# ik_llama.cpp's IQK kernels require at least AVX2
|
||||
make ik-llama-cpp-avx2
|
||||
fi
|
||||
EOT
|
||||
|
||||
|
||||
RUN make -BC /LocalAI/backend/cpp/ik-llama-cpp package
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Final stage — copies package output from one of the two builders.
|
||||
# BUILDER_TARGET selects which one. BuildKit prunes the unreferenced builder.
|
||||
#
|
||||
# BuildKit doesn't support variable expansion in `COPY --from=` directly,
|
||||
# so we resolve the ARG by aliasing the chosen builder to a fixed stage
|
||||
# name via `FROM ${BUILDER_TARGET} AS builder` and then COPY --from=builder.
|
||||
# BUILDER_TARGET itself is declared as a global ARG at the top of this
|
||||
# file (required for use in FROM), so we just re-import it into this
|
||||
# stage's scope before the FROM directive.
|
||||
# ============================================================================
|
||||
FROM ${BUILDER_TARGET} AS builder
|
||||
|
||||
FROM scratch
|
||||
|
||||
|
||||
# Copy all available binaries (the build process only creates the appropriate ones for the target architecture)
|
||||
COPY --from=builder /LocalAI/backend/cpp/ik-llama-cpp/package/. ./
|
||||
|
||||
@@ -1,155 +1,65 @@
|
||||
ARG BASE_IMAGE=ubuntu:24.04
|
||||
# BUILDER_BASE_IMAGE defaults to BASE_IMAGE so the Dockerfile parses even
|
||||
# when no prebuilt base is supplied. The builder-prebuilt stage is only
|
||||
# entered when BUILDER_TARGET=builder-prebuilt, so a "wrong" fallback
|
||||
# content here is harmless — BuildKit prunes the unreferenced builder.
|
||||
ARG BUILDER_BASE_IMAGE=${BASE_IMAGE}
|
||||
# BUILDER_TARGET selects which builder stage the final scratch image copies
|
||||
# package output from. Declared at global scope (before any FROM) so it's
|
||||
# usable in `FROM ${BUILDER_TARGET}` below. Default keeps local
|
||||
# `make backends/llama-cpp` on the from-source path.
|
||||
ARG BUILDER_TARGET=builder-fromsource
|
||||
ARG APT_MIRROR=""
|
||||
ARG APT_PORTS_MIRROR=""
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Stage: builder-fromsource — self-contained build path.
|
||||
# Runs .docker/install-base-deps.sh (apt deps + cmake + protoc + gRPC +
|
||||
# conditional CUDA/ROCm/Vulkan), copies /opt/grpc to /usr/local, then
|
||||
# compiles the variant. Used when BUILDER_TARGET=builder-fromsource (the
|
||||
# default; local `make backends/llama-cpp`).
|
||||
# Builds the llama-cpp backend on top of the shared
|
||||
# .docker/bases/Dockerfile.cpp base. The base bakes in apt + GPU SDK +
|
||||
# protoc + cmake + GRPC, so this stage only carries the COPY + `make`
|
||||
# invocations and the final scratch-stage package.
|
||||
#
|
||||
# The install script is the same one that backend/Dockerfile.base-grpc-builder
|
||||
# runs, so the result is bit-equivalent to the prebuilt-base path
|
||||
# (builder-prebuilt below).
|
||||
# ============================================================================
|
||||
FROM ${BASE_IMAGE} AS builder-fromsource
|
||||
ARG BUILD_TYPE
|
||||
ARG CUDA_MAJOR_VERSION
|
||||
ARG CUDA_MINOR_VERSION
|
||||
ARG CMAKE_FROM_SOURCE=false
|
||||
# CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
|
||||
ARG CMAKE_VERSION=3.31.10
|
||||
ARG GRPC_VERSION=v1.65.0
|
||||
ARG GRPC_MAKEFLAGS="-j4 -Otarget"
|
||||
ARG SKIP_DRIVERS=false
|
||||
ARG TARGETARCH
|
||||
ARG TARGETVARIANT
|
||||
ARG GO_VERSION=1.25.4
|
||||
ARG UBUNTU_VERSION=2404
|
||||
ARG APT_MIRROR
|
||||
ARG APT_PORTS_MIRROR
|
||||
ARG AMDGPU_TARGETS
|
||||
# CUDA target archs, e.g. --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
|
||||
ARG CUDA_DOCKER_ARCH
|
||||
ARG CMAKE_ARGS
|
||||
# CI orchestration (.github/workflows/backend.yml + backend_pr.yml) passes
|
||||
# BASE_IMAGE_PREBUILT. See .agents/ci-caching.md.
|
||||
|
||||
ENV BUILD_TYPE=${BUILD_TYPE} \
|
||||
CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
|
||||
CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
|
||||
CMAKE_FROM_SOURCE=${CMAKE_FROM_SOURCE} \
|
||||
CMAKE_VERSION=${CMAKE_VERSION} \
|
||||
GRPC_VERSION=${GRPC_VERSION} \
|
||||
GRPC_MAKEFLAGS=${GRPC_MAKEFLAGS} \
|
||||
SKIP_DRIVERS=${SKIP_DRIVERS} \
|
||||
TARGETARCH=${TARGETARCH} \
|
||||
UBUNTU_VERSION=${UBUNTU_VERSION} \
|
||||
APT_MIRROR=${APT_MIRROR} \
|
||||
APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
|
||||
AMDGPU_TARGETS=${AMDGPU_TARGETS} \
|
||||
CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH} \
|
||||
CMAKE_ARGS=${CMAKE_ARGS} \
|
||||
DEBIAN_FRONTEND=noninteractive
|
||||
ARG BASE_IMAGE_PREBUILT
|
||||
|
||||
# CUDA on PATH (no-op when CUDA isn't installed)
|
||||
ENV PATH=/usr/local/cuda/bin:${PATH}
|
||||
# HipBLAS / ROCm on PATH (no-op when ROCm isn't installed)
|
||||
ENV PATH=/opt/rocm/bin:${PATH}
|
||||
FROM ${BASE_IMAGE_PREBUILT} AS builder
|
||||
|
||||
WORKDIR /build
|
||||
|
||||
# Install everything via the shared script — the same one that
|
||||
# backend/Dockerfile.base-grpc-builder runs, so the prebuilt CI base and
|
||||
# this from-source path are bit-equivalent.
|
||||
RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
|
||||
--mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
|
||||
bash /usr/local/sbin/install-base-deps
|
||||
|
||||
# Mirror builder-prebuilt: copy gRPC from /opt/grpc to /usr/local so
|
||||
# CMake's find_package finds it at the canonical prefix the Makefile expects.
|
||||
RUN cp -a /opt/grpc/. /usr/local/
|
||||
|
||||
COPY . /LocalAI
|
||||
|
||||
# BuildKit cache mount for ccache. Persists compiler outputs across builds
|
||||
# via the registry cache (cache-to: type=registry,mode=max in CI). On a
|
||||
# LLAMA_VERSION bump most TUs are byte-identical to the previous version's
|
||||
# preprocessed source — ccache returns the previous .o file and skips the
|
||||
# real compile. Same for LocalAI source changes that don't touch llama.cpp.
|
||||
# CMAKE_*_COMPILER_LAUNCHER threads ccache through CMake to wrap gcc/g++/nvcc.
|
||||
# sharing=locked serializes concurrent writes if multiple matrix variants
|
||||
# share the same cache mount id.
|
||||
#
|
||||
# The compile body is shared with builder-prebuilt via .docker/llama-cpp-compile.sh.
|
||||
RUN --mount=type=bind,source=.docker/llama-cpp-compile.sh,target=/usr/local/sbin/compile.sh \
|
||||
--mount=type=cache,target=/root/.ccache,id=llama-cpp-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
|
||||
bash /usr/local/sbin/compile.sh
|
||||
|
||||
|
||||
# Copy libraries using a script to handle architecture differences
|
||||
RUN make -BC /LocalAI/backend/cpp/llama-cpp package
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Stage: builder-prebuilt — uses the pre-built base from
|
||||
# quay.io/go-skynet/ci-cache:base-grpc-* (built by .github/workflows/base-images.yml).
|
||||
# That image already has gRPC at /opt/grpc + apt deps + CUDA/ROCm/Vulkan
|
||||
# pre-installed, so we just copy gRPC to /usr/local and compile. Used when
|
||||
# BUILDER_TARGET=builder-prebuilt (CI when the matrix entry sets
|
||||
# builder-base-image).
|
||||
# ============================================================================
|
||||
FROM ${BUILDER_BASE_IMAGE} AS builder-prebuilt
|
||||
|
||||
ARG BUILD_TYPE
|
||||
ENV BUILD_TYPE=${BUILD_TYPE}
|
||||
# We can target specific CUDA ARCHITECTURES like --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
|
||||
ARG CUDA_DOCKER_ARCH
|
||||
ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
|
||||
ARG CMAKE_ARGS
|
||||
ENV CMAKE_ARGS=${CMAKE_ARGS}
|
||||
ARG AMDGPU_TARGETS
|
||||
ENV AMDGPU_TARGETS=${AMDGPU_TARGETS}
|
||||
ARG BACKEND=llama-cpp
|
||||
ARG BUILD_TYPE
|
||||
ENV BUILD_TYPE=${BUILD_TYPE}
|
||||
ARG CUDA_MAJOR_VERSION
|
||||
ARG CUDA_MINOR_VERSION
|
||||
ENV CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION}
|
||||
ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
|
||||
ARG TARGETARCH
|
||||
ARG TARGETVARIANT
|
||||
|
||||
# The base-grpc-* image installs gRPC to /opt/grpc but doesn't copy it to
|
||||
# /usr/local. The variant Dockerfile's from-source path does that too;
|
||||
# mirror it here so the compile step can find gRPC at the canonical
|
||||
# prefix the Makefile expects.
|
||||
RUN cp -a /opt/grpc/. /usr/local/
|
||||
|
||||
COPY . /LocalAI
|
||||
|
||||
RUN --mount=type=bind,source=.docker/llama-cpp-compile.sh,target=/usr/local/sbin/compile.sh \
|
||||
--mount=type=cache,target=/root/.ccache,id=llama-cpp-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
|
||||
bash /usr/local/sbin/compile.sh
|
||||
RUN <<'EOT' bash
|
||||
set -euxo pipefail
|
||||
|
||||
if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
|
||||
CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
|
||||
export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
|
||||
echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
|
||||
rm -rf /LocalAI/backend/cpp/llama-cpp-*-build
|
||||
fi
|
||||
|
||||
if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
|
||||
cd /LocalAI/backend/cpp/llama-cpp
|
||||
make llama-cpp-fallback
|
||||
make llama-cpp-grpc
|
||||
make llama-cpp-rpc-server
|
||||
else
|
||||
cd /LocalAI/backend/cpp/llama-cpp
|
||||
make llama-cpp-avx
|
||||
make llama-cpp-avx2
|
||||
make llama-cpp-avx512
|
||||
make llama-cpp-fallback
|
||||
make llama-cpp-grpc
|
||||
make llama-cpp-rpc-server
|
||||
fi
|
||||
EOT
|
||||
|
||||
|
||||
# Copy libraries using a script to handle architecture differences
|
||||
RUN make -BC /LocalAI/backend/cpp/llama-cpp package
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Final stage — copies package output from one of the two builders.
|
||||
# BUILDER_TARGET selects which one. BuildKit prunes the unreferenced builder.
|
||||
#
|
||||
# BuildKit doesn't support variable expansion in `COPY --from=` directly,
|
||||
# so we resolve the ARG by aliasing the chosen builder to a fixed stage
|
||||
# name via `FROM ${BUILDER_TARGET} AS builder` and then COPY --from=builder.
|
||||
# BUILDER_TARGET itself is declared as a global ARG at the top of this
|
||||
# file (required for use in FROM), so we just re-import it into this
|
||||
# stage's scope before the FROM directive.
|
||||
# ============================================================================
|
||||
FROM ${BUILDER_TARGET} AS builder
|
||||
|
||||
FROM scratch
|
||||
|
||||
|
||||
|
||||
@@ -1,202 +1,26 @@
|
||||
ARG BASE_IMAGE=ubuntu:24.04
|
||||
ARG APT_MIRROR=""
|
||||
ARG APT_PORTS_MIRROR=""
|
||||
# Builds a single Python backend on top of the shared
|
||||
# .docker/bases/Dockerfile.python base. The base bakes in apt-update + GPU
|
||||
# SDK install + python toolchain (uv, pip, rustup, grpcio-tools), so this
|
||||
# stage only carries the per-backend source COPY + `make`.
|
||||
#
|
||||
# CI orchestration (.github/workflows/backend.yml + backend_pr.yml) builds
|
||||
# the right base flavour automatically via scripts/derive-build-matrix.js
|
||||
# and passes BASE_IMAGE_PREBUILT here. For local builds, run:
|
||||
# make backend-image-base BUILD_TYPE=<...> # build the base
|
||||
# make backend-image BACKEND=<...> BUILD_TYPE=<...>
|
||||
# See .agents/ci-caching.md.
|
||||
|
||||
ARG BASE_IMAGE_PREBUILT
|
||||
|
||||
FROM ${BASE_IMAGE_PREBUILT} AS builder
|
||||
|
||||
FROM ${BASE_IMAGE} AS builder
|
||||
ARG BACKEND=rerankers
|
||||
ARG BUILD_TYPE
|
||||
ENV BUILD_TYPE=${BUILD_TYPE}
|
||||
ARG CUDA_MAJOR_VERSION
|
||||
ARG CUDA_MINOR_VERSION
|
||||
ARG SKIP_DRIVERS=false
|
||||
ENV CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION}
|
||||
ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
ARG TARGETARCH
|
||||
ARG TARGETVARIANT
|
||||
ARG UBUNTU_VERSION=2404
|
||||
ARG APT_MIRROR
|
||||
ARG APT_PORTS_MIRROR
|
||||
|
||||
RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
|
||||
APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
ccache \
|
||||
ca-certificates \
|
||||
espeak-ng \
|
||||
curl \
|
||||
libssl-dev \
|
||||
git wget \
|
||||
git-lfs \
|
||||
unzip clang \
|
||||
upx-ucl \
|
||||
curl python3-pip \
|
||||
python-is-python3 \
|
||||
python3-dev llvm \
|
||||
libnuma1 libgomp1 \
|
||||
python3-venv make cmake && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN <<EOT bash
|
||||
if [ "${UBUNTU_VERSION}" = "2404" ]; then
|
||||
pip install --break-system-packages --user --upgrade pip
|
||||
else
|
||||
pip install --upgrade pip
|
||||
fi
|
||||
EOT
|
||||
|
||||
|
||||
# Cuda
|
||||
ENV PATH=/usr/local/cuda/bin:${PATH}
|
||||
|
||||
# HipBLAS requirements
|
||||
ENV PATH=/opt/rocm/bin:${PATH}
|
||||
|
||||
# Vulkan requirements
|
||||
RUN <<EOT bash
|
||||
if [ "${BUILD_TYPE}" = "vulkan" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
software-properties-common pciutils wget gpg-agent && \
|
||||
apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
|
||||
libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
|
||||
libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
|
||||
git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
|
||||
ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
|
||||
clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
|
||||
if [ "amd64" = "$TARGETARCH" ]; then
|
||||
wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz" && \
|
||||
tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
|
||||
rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
|
||||
mkdir -p /opt/vulkan-sdk && \
|
||||
mv 1.4.335.0 /opt/vulkan-sdk/ && \
|
||||
cd /opt/vulkan-sdk/1.4.335.0 && \
|
||||
./vulkansdk --no-deps --maxjobs \
|
||||
vulkan-loader \
|
||||
vulkan-validationlayers \
|
||||
vulkan-extensionlayer \
|
||||
vulkan-tools \
|
||||
shaderc && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/ && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/ && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/ && \
|
||||
cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/ && \
|
||||
rm -rf /opt/vulkan-sdk
|
||||
fi
|
||||
if [ "arm64" = "$TARGETARCH" ]; then
|
||||
mkdir vulkan && cd vulkan && \
|
||||
curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
|
||||
tar -xvf vulkan-sdk.tar.xz && \
|
||||
rm vulkan-sdk.tar.xz && \
|
||||
cd 1.4.335.0 && \
|
||||
cp -rfv aarch64/bin/* /usr/bin/ && \
|
||||
cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
|
||||
cp -rfv aarch64/include/* /usr/include/ && \
|
||||
cp -rfv aarch64/share/* /usr/share/ && \
|
||||
cd ../.. && \
|
||||
rm -rf vulkan
|
||||
fi
|
||||
ldconfig && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
EOT
|
||||
|
||||
# CuBLAS requirements
|
||||
RUN <<EOT bash
|
||||
if ( [ "${BUILD_TYPE}" = "cublas" ] || [ "${BUILD_TYPE}" = "l4t" ] ) && [ "${SKIP_DRIVERS}" = "false" ]; then
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
software-properties-common pciutils
|
||||
if [ "amd64" = "$TARGETARCH" ]; then
|
||||
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb
|
||||
fi
|
||||
if [ "arm64" = "$TARGETARCH" ]; then
|
||||
if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
|
||||
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb
|
||||
else
|
||||
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb
|
||||
fi
|
||||
fi
|
||||
dpkg -i cuda-keyring_1.1-1_all.deb && \
|
||||
rm -f cuda-keyring_1.1-1_all.deb && \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
|
||||
libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
|
||||
if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "$TARGETARCH" ]; then
|
||||
apt-get install -y --no-install-recommends \
|
||||
libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libcudnn9-cuda-${CUDA_MAJOR_VERSION} cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
|
||||
fi
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
fi
|
||||
EOT
|
||||
|
||||
|
||||
# https://github.com/NVIDIA/Isaac-GR00T/issues/343
|
||||
RUN <<EOT bash
|
||||
if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "arm64" ]; then
|
||||
wget https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
|
||||
dpkg -i cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
|
||||
cp /var/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/ && \
|
||||
apt-get update && apt-get -y install cudss cudss-cuda-${CUDA_MAJOR_VERSION} && \
|
||||
wget https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
|
||||
dpkg -i nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
|
||||
cp /var/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/ && \
|
||||
apt-get update && apt-get install -y nvpl
|
||||
fi
|
||||
EOT
|
||||
|
||||
# If we are building with clblas support, we need the libraries for the builds
|
||||
RUN if [ "${BUILD_TYPE}" = "clblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
libclblast-dev && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/* \
|
||||
; fi
|
||||
|
||||
RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
hipblas-dev \
|
||||
hipblaslt-dev \
|
||||
rocblas-dev && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/* && \
|
||||
# I have no idea why, but the ROCM lib packages don't trigger ldconfig after they install, which results in local-ai and others not being able
|
||||
# to locate the libraries. We run ldconfig ourselves to work around this packaging deficiency
|
||||
ldconfig \
|
||||
; fi
|
||||
|
||||
RUN if [ "${BUILD_TYPE}" = "hipblas" ]; then \
|
||||
ln -s /opt/rocm-**/lib/llvm/lib/libomp.so /usr/lib/libomp.so \
|
||||
; fi
|
||||
|
||||
# Install uv as a system package
|
||||
RUN curl -LsSf https://astral.sh/uv/install.sh | UV_INSTALL_DIR=/usr/bin sh
|
||||
ENV PATH="/root/.cargo/bin:${PATH}"
|
||||
# Increase timeout for uv installs behind slow networks
|
||||
ENV UV_HTTP_TIMEOUT=180
|
||||
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
|
||||
|
||||
# Install grpcio-tools (the version in 22.04 is too old)
|
||||
RUN <<EOT bash
|
||||
if [ "${UBUNTU_VERSION}" = "2404" ]; then
|
||||
pip install --break-system-packages --user grpcio-tools==1.71.0 grpcio==1.71.0
|
||||
else
|
||||
pip install grpcio-tools==1.71.0 grpcio==1.71.0
|
||||
fi
|
||||
EOT
|
||||
|
||||
|
||||
COPY backend/python/${BACKEND} /${BACKEND}
|
||||
COPY backend/backend.proto /${BACKEND}/backend.proto
|
||||
|
||||
@@ -1,37 +1,15 @@
|
||||
ARG BASE_IMAGE=ubuntu:24.04
|
||||
ARG APT_MIRROR=""
|
||||
ARG APT_PORTS_MIRROR=""
|
||||
# Builds a single Rust backend on top of the shared
|
||||
# .docker/bases/Dockerfile.rust base. The base bakes in apt + Rust +
|
||||
# protobuf-compiler + audio dev libs (espeak/sonic/pcaudio/opus), so this
|
||||
# stage only carries the per-backend COPY + `make build`.
|
||||
#
|
||||
# CI orchestration (.github/workflows/backend.yml + backend_pr.yml) passes
|
||||
# BASE_IMAGE_PREBUILT. See .agents/ci-caching.md.
|
||||
|
||||
FROM ${BASE_IMAGE} AS builder
|
||||
ARG BASE_IMAGE_PREBUILT
|
||||
|
||||
FROM ${BASE_IMAGE_PREBUILT} AS builder
|
||||
ARG BACKEND=kokoros
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
ARG TARGETARCH
|
||||
ARG TARGETVARIANT
|
||||
ARG APT_MIRROR
|
||||
ARG APT_PORTS_MIRROR
|
||||
|
||||
RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
|
||||
APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
|
||||
apt-get update && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
build-essential \
|
||||
git ccache \
|
||||
ca-certificates \
|
||||
make cmake wget \
|
||||
curl unzip \
|
||||
clang \
|
||||
pkg-config \
|
||||
libssl-dev \
|
||||
espeak-ng libespeak-ng-dev \
|
||||
libsonic-dev libpcaudio-dev \
|
||||
libopus-dev \
|
||||
protobuf-compiler && \
|
||||
apt-get clean && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install Rust
|
||||
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
|
||||
ENV PATH="/root/.cargo/bin:${PATH}"
|
||||
|
||||
COPY . /LocalAI
|
||||
|
||||
|
||||
@@ -1,152 +1,59 @@
|
||||
ARG BASE_IMAGE=ubuntu:24.04
|
||||
# BUILDER_BASE_IMAGE defaults to BASE_IMAGE so the Dockerfile parses even
|
||||
# when no prebuilt base is supplied. The builder-prebuilt stage is only
|
||||
# entered when BUILDER_TARGET=builder-prebuilt, so a "wrong" fallback
|
||||
# content here is harmless — BuildKit prunes the unreferenced builder.
|
||||
ARG BUILDER_BASE_IMAGE=${BASE_IMAGE}
|
||||
# BUILDER_TARGET selects which builder stage the final scratch image copies
|
||||
# package output from. Declared at global scope (before any FROM) so it's
|
||||
# usable in `FROM ${BUILDER_TARGET}` below. Default keeps local
|
||||
# `make backends/turboquant` on the from-source path.
|
||||
ARG BUILDER_TARGET=builder-fromsource
|
||||
ARG APT_MIRROR=""
|
||||
ARG APT_PORTS_MIRROR=""
|
||||
# Builds the turboquant backend on top of the shared
|
||||
# .docker/bases/Dockerfile.cpp base (shared with llama-cpp/ik-llama-cpp).
|
||||
# See backend/Dockerfile.llama-cpp for the rationale; this consumer differs
|
||||
# only in the make targets at the end.
|
||||
|
||||
ARG BASE_IMAGE_PREBUILT
|
||||
|
||||
# ============================================================================
|
||||
# Stage: builder-fromsource — self-contained build path.
|
||||
# Runs .docker/install-base-deps.sh (apt deps + cmake + protoc + gRPC +
|
||||
# conditional CUDA/ROCm/Vulkan), copies /opt/grpc to /usr/local, then
|
||||
# compiles the variant. Used when BUILDER_TARGET=builder-fromsource (the
|
||||
# default; local `make backends/turboquant`).
|
||||
#
|
||||
# The install script is the same one that backend/Dockerfile.base-grpc-builder
|
||||
# runs, so the result is bit-equivalent to the prebuilt-base path
|
||||
# (builder-prebuilt below).
|
||||
# ============================================================================
|
||||
FROM ${BASE_IMAGE} AS builder-fromsource
|
||||
FROM ${BASE_IMAGE_PREBUILT} AS builder
|
||||
|
||||
ARG CUDA_DOCKER_ARCH
|
||||
ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
|
||||
ARG CMAKE_ARGS
|
||||
ENV CMAKE_ARGS=${CMAKE_ARGS}
|
||||
ARG BACKEND=turboquant
|
||||
ARG BUILD_TYPE
|
||||
ENV BUILD_TYPE=${BUILD_TYPE}
|
||||
ARG CUDA_MAJOR_VERSION
|
||||
ARG CUDA_MINOR_VERSION
|
||||
ARG CMAKE_FROM_SOURCE=false
|
||||
# CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
|
||||
ARG CMAKE_VERSION=3.31.10
|
||||
ARG GRPC_VERSION=v1.65.0
|
||||
ARG GRPC_MAKEFLAGS="-j4 -Otarget"
|
||||
ARG SKIP_DRIVERS=false
|
||||
ENV CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION}
|
||||
ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
|
||||
ARG TARGETARCH
|
||||
ARG TARGETVARIANT
|
||||
ARG GO_VERSION=1.25.4
|
||||
ARG UBUNTU_VERSION=2404
|
||||
ARG APT_MIRROR
|
||||
ARG APT_PORTS_MIRROR
|
||||
ARG AMDGPU_TARGETS=""
|
||||
ARG BACKEND=rerankers
|
||||
# CUDA target archs, e.g. --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
|
||||
ARG CUDA_DOCKER_ARCH
|
||||
ARG CMAKE_ARGS
|
||||
|
||||
ENV BUILD_TYPE=${BUILD_TYPE} \
|
||||
CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION} \
|
||||
CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION} \
|
||||
CMAKE_FROM_SOURCE=${CMAKE_FROM_SOURCE} \
|
||||
CMAKE_VERSION=${CMAKE_VERSION} \
|
||||
GRPC_VERSION=${GRPC_VERSION} \
|
||||
GRPC_MAKEFLAGS=${GRPC_MAKEFLAGS} \
|
||||
SKIP_DRIVERS=${SKIP_DRIVERS} \
|
||||
TARGETARCH=${TARGETARCH} \
|
||||
UBUNTU_VERSION=${UBUNTU_VERSION} \
|
||||
APT_MIRROR=${APT_MIRROR} \
|
||||
APT_PORTS_MIRROR=${APT_PORTS_MIRROR} \
|
||||
AMDGPU_TARGETS=${AMDGPU_TARGETS} \
|
||||
CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH} \
|
||||
CMAKE_ARGS=${CMAKE_ARGS} \
|
||||
DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
# CUDA on PATH (no-op when CUDA isn't installed)
|
||||
ENV PATH=/usr/local/cuda/bin:${PATH}
|
||||
# HipBLAS / ROCm on PATH (no-op when ROCm isn't installed)
|
||||
ENV PATH=/opt/rocm/bin:${PATH}
|
||||
|
||||
WORKDIR /build
|
||||
|
||||
# Install everything via the shared script — the same one that
|
||||
# backend/Dockerfile.base-grpc-builder runs, so the prebuilt CI base and
|
||||
# this from-source path are bit-equivalent.
|
||||
RUN --mount=type=bind,source=.docker/install-base-deps.sh,target=/usr/local/sbin/install-base-deps \
|
||||
--mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
|
||||
bash /usr/local/sbin/install-base-deps
|
||||
|
||||
# Mirror builder-prebuilt: copy gRPC from /opt/grpc to /usr/local so
|
||||
# CMake's find_package finds it at the canonical prefix the Makefile expects.
|
||||
RUN cp -a /opt/grpc/. /usr/local/
|
||||
|
||||
COPY . /LocalAI
|
||||
|
||||
# BuildKit cache mount for ccache. See Dockerfile.llama-cpp (commit 9228e5b4)
|
||||
# for rationale. turboquant is a llama.cpp fork that reuses
|
||||
# backend/cpp/llama-cpp source via a thin wrapper Makefile, so MOST TUs
|
||||
# are content-identical to the upstream llama-cpp build. Sharing a cache
|
||||
# id with llama-cpp could give cross-fork hits — but for now keep them
|
||||
# separate so a regression in one doesn't poison the other. Revisit
|
||||
# sharing after measuring the actual hit rate.
|
||||
#
|
||||
# The compile body is shared with builder-prebuilt via .docker/turboquant-compile.sh.
|
||||
RUN --mount=type=bind,source=.docker/turboquant-compile.sh,target=/usr/local/sbin/compile.sh \
|
||||
--mount=type=cache,target=/root/.ccache,id=turboquant-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
|
||||
bash /usr/local/sbin/compile.sh
|
||||
RUN <<'EOT' bash
|
||||
set -euxo pipefail
|
||||
|
||||
if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
|
||||
CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
|
||||
export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
|
||||
echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
|
||||
rm -rf /LocalAI/backend/cpp/turboquant-*-build
|
||||
fi
|
||||
|
||||
cd /LocalAI/backend/cpp/turboquant
|
||||
|
||||
if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
|
||||
make turboquant-fallback
|
||||
make turboquant-grpc
|
||||
make turboquant-rpc-server
|
||||
else
|
||||
make turboquant-avx
|
||||
make turboquant-avx2
|
||||
make turboquant-avx512
|
||||
make turboquant-fallback
|
||||
make turboquant-grpc
|
||||
make turboquant-rpc-server
|
||||
fi
|
||||
EOT
|
||||
|
||||
|
||||
# Copy libraries using a script to handle architecture differences
|
||||
RUN make -BC /LocalAI/backend/cpp/turboquant package
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Stage: builder-prebuilt — uses the pre-built base from
|
||||
# quay.io/go-skynet/ci-cache:base-grpc-* (built by .github/workflows/base-images.yml).
|
||||
# That image already has gRPC at /opt/grpc + apt deps + CUDA/ROCm/Vulkan
|
||||
# pre-installed, so we just copy gRPC to /usr/local and compile. Used when
|
||||
# BUILDER_TARGET=builder-prebuilt (CI when the matrix entry sets
|
||||
# builder-base-image).
|
||||
# ============================================================================
|
||||
FROM ${BUILDER_BASE_IMAGE} AS builder-prebuilt
|
||||
|
||||
ARG BUILD_TYPE
|
||||
ENV BUILD_TYPE=${BUILD_TYPE}
|
||||
ARG CUDA_DOCKER_ARCH
|
||||
ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
|
||||
ARG CMAKE_ARGS
|
||||
ENV CMAKE_ARGS=${CMAKE_ARGS}
|
||||
ARG TARGETARCH
|
||||
ARG TARGETVARIANT
|
||||
|
||||
# The base-grpc-* image installs gRPC to /opt/grpc but doesn't copy it to
|
||||
# /usr/local. Mirror what the from-source path does so the compile step
|
||||
# can find gRPC at the canonical prefix the Makefile expects.
|
||||
RUN cp -a /opt/grpc/. /usr/local/
|
||||
|
||||
COPY . /LocalAI
|
||||
|
||||
RUN --mount=type=bind,source=.docker/turboquant-compile.sh,target=/usr/local/sbin/compile.sh \
|
||||
--mount=type=cache,target=/root/.ccache,id=turboquant-ccache-${TARGETARCH}-${BUILD_TYPE},sharing=locked \
|
||||
bash /usr/local/sbin/compile.sh
|
||||
|
||||
RUN make -BC /LocalAI/backend/cpp/turboquant package
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Final stage — copies package output from one of the two builders.
|
||||
# BUILDER_TARGET selects which one. BuildKit prunes the unreferenced builder.
|
||||
#
|
||||
# BuildKit doesn't support variable expansion in `COPY --from=` directly,
|
||||
# so we resolve the ARG by aliasing the chosen builder to a fixed stage
|
||||
# name via `FROM ${BUILDER_TARGET} AS builder` and then COPY --from=builder.
|
||||
# BUILDER_TARGET itself is declared as a global ARG at the top of this
|
||||
# file (required for use in FROM), so we just re-import it into this
|
||||
# stage's scope before the FROM directive.
|
||||
# ============================================================================
|
||||
FROM ${BUILDER_TARGET} AS builder
|
||||
|
||||
FROM scratch
|
||||
|
||||
|
||||
|
||||
9
backend/cpp/ds4/.gitignore
vendored
9
backend/cpp/ds4/.gitignore
vendored
@@ -1,9 +0,0 @@
|
||||
ds4/
|
||||
build/
|
||||
package/
|
||||
grpc-server
|
||||
*.o
|
||||
backend.pb.cc
|
||||
backend.pb.h
|
||||
backend.grpc.pb.cc
|
||||
backend.grpc.pb.h
|
||||
@@ -1,101 +0,0 @@
|
||||
cmake_minimum_required(VERSION 3.15)
|
||||
project(ds4-grpc-server LANGUAGES CXX C)
|
||||
|
||||
set(CMAKE_CXX_STANDARD 17)
|
||||
set(CMAKE_CXX_STANDARD_REQUIRED ON)
|
||||
set(TARGET grpc-server)
|
||||
|
||||
option(DS4_NATIVE "Compile with -march=native / -mcpu=native" ON)
|
||||
set(DS4_GPU "cpu" CACHE STRING "GPU backend: cpu, cuda, or metal")
|
||||
set(DS4_DIR "${CMAKE_CURRENT_SOURCE_DIR}/ds4" CACHE PATH "Path to cloned ds4 source")
|
||||
|
||||
find_package(Threads REQUIRED)
|
||||
find_package(Protobuf CONFIG QUIET)
|
||||
if(NOT Protobuf_FOUND)
|
||||
find_package(Protobuf REQUIRED)
|
||||
endif()
|
||||
find_package(gRPC CONFIG QUIET)
|
||||
if(NOT gRPC_FOUND)
|
||||
# Ubuntu's apt-installed grpc++ does not ship a CMake config - fall back.
|
||||
find_library(GRPCPP_LIB grpc++ REQUIRED)
|
||||
find_library(GRPCPP_REFLECTION_LIB grpc++_reflection REQUIRED)
|
||||
add_library(gRPC::grpc++ INTERFACE IMPORTED)
|
||||
set_target_properties(gRPC::grpc++ PROPERTIES INTERFACE_LINK_LIBRARIES "${GRPCPP_LIB}")
|
||||
add_library(gRPC::grpc++_reflection INTERFACE IMPORTED)
|
||||
set_target_properties(gRPC::grpc++_reflection PROPERTIES INTERFACE_LINK_LIBRARIES "${GRPCPP_REFLECTION_LIB}")
|
||||
endif()
|
||||
|
||||
find_program(_PROTOC NAMES protoc REQUIRED)
|
||||
find_program(_GRPC_CPP_PLUGIN NAMES grpc_cpp_plugin REQUIRED)
|
||||
|
||||
get_filename_component(HW_PROTO "${CMAKE_CURRENT_SOURCE_DIR}/../../backend.proto" ABSOLUTE)
|
||||
get_filename_component(HW_PROTO_PATH "${HW_PROTO}" PATH)
|
||||
|
||||
set(HW_PROTO_SRCS "${CMAKE_CURRENT_BINARY_DIR}/backend.pb.cc")
|
||||
set(HW_PROTO_HDRS "${CMAKE_CURRENT_BINARY_DIR}/backend.pb.h")
|
||||
set(HW_GRPC_SRCS "${CMAKE_CURRENT_BINARY_DIR}/backend.grpc.pb.cc")
|
||||
set(HW_GRPC_HDRS "${CMAKE_CURRENT_BINARY_DIR}/backend.grpc.pb.h")
|
||||
|
||||
add_custom_command(
|
||||
OUTPUT "${HW_PROTO_SRCS}" "${HW_PROTO_HDRS}" "${HW_GRPC_SRCS}" "${HW_GRPC_HDRS}"
|
||||
COMMAND ${_PROTOC}
|
||||
ARGS --grpc_out "${CMAKE_CURRENT_BINARY_DIR}"
|
||||
--cpp_out "${CMAKE_CURRENT_BINARY_DIR}"
|
||||
-I "${HW_PROTO_PATH}"
|
||||
--plugin=protoc-gen-grpc="${_GRPC_CPP_PLUGIN}"
|
||||
"${HW_PROTO}"
|
||||
DEPENDS "${HW_PROTO}")
|
||||
|
||||
add_library(hw_grpc_proto STATIC
|
||||
${HW_GRPC_SRCS} ${HW_GRPC_HDRS}
|
||||
${HW_PROTO_SRCS} ${HW_PROTO_HDRS})
|
||||
target_include_directories(hw_grpc_proto PUBLIC ${CMAKE_CURRENT_BINARY_DIR})
|
||||
|
||||
set(DS4_OBJS "${DS4_DIR}/ds4.o")
|
||||
if(DS4_GPU STREQUAL "cuda")
|
||||
list(APPEND DS4_OBJS "${DS4_DIR}/ds4_cuda.o")
|
||||
elseif(DS4_GPU STREQUAL "metal")
|
||||
list(APPEND DS4_OBJS "${DS4_DIR}/ds4_metal.o")
|
||||
elseif(DS4_GPU STREQUAL "cpu")
|
||||
set(DS4_OBJS "${DS4_DIR}/ds4_cpu.o")
|
||||
endif()
|
||||
|
||||
add_executable(${TARGET}
|
||||
grpc-server.cpp
|
||||
dsml_parser.cpp
|
||||
dsml_renderer.cpp
|
||||
kv_cache.cpp)
|
||||
|
||||
target_include_directories(${TARGET} PRIVATE ${DS4_DIR})
|
||||
|
||||
foreach(obj ${DS4_OBJS})
|
||||
target_sources(${TARGET} PRIVATE ${obj})
|
||||
set_source_files_properties(${obj} PROPERTIES EXTERNAL_OBJECT TRUE GENERATED TRUE)
|
||||
endforeach()
|
||||
|
||||
target_link_libraries(${TARGET} PRIVATE
|
||||
hw_grpc_proto
|
||||
gRPC::grpc++
|
||||
gRPC::grpc++_reflection
|
||||
protobuf::libprotobuf
|
||||
Threads::Threads
|
||||
m)
|
||||
|
||||
if(DS4_GPU STREQUAL "cuda")
|
||||
find_package(CUDAToolkit REQUIRED)
|
||||
target_link_libraries(${TARGET} PRIVATE CUDA::cudart CUDA::cublas)
|
||||
elseif(DS4_GPU STREQUAL "metal")
|
||||
find_library(FOUNDATION_LIB Foundation REQUIRED)
|
||||
find_library(METAL_LIB Metal REQUIRED)
|
||||
target_link_libraries(${TARGET} PRIVATE ${FOUNDATION_LIB} ${METAL_LIB})
|
||||
elseif(DS4_GPU STREQUAL "cpu")
|
||||
target_compile_definitions(${TARGET} PRIVATE DS4_NO_GPU)
|
||||
endif()
|
||||
|
||||
if(DS4_NATIVE)
|
||||
if(APPLE)
|
||||
target_compile_options(${TARGET} PRIVATE -mcpu=native)
|
||||
else()
|
||||
target_compile_options(${TARGET} PRIVATE -march=native)
|
||||
endif()
|
||||
endif()
|
||||
@@ -1,78 +0,0 @@
|
||||
# ds4 backend Makefile.
|
||||
#
|
||||
# Upstream pin lives below as DS4_VERSION?= so the bump-deps bot
|
||||
# (.github/bump_deps.sh) can find and update it - matches the
|
||||
# llama-cpp / ik-llama-cpp / turboquant convention.
|
||||
|
||||
DS4_VERSION?=ae302c2fa18cc6d9aefc021d0f27ae03c9ad2fc0
|
||||
DS4_REPO?=https://github.com/antirez/ds4
|
||||
|
||||
CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
|
||||
BUILD_DIR := build
|
||||
|
||||
BUILD_TYPE ?=
|
||||
NATIVE ?= false
|
||||
JOBS ?= $(shell nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4)
|
||||
|
||||
UNAME_S := $(shell uname -s)
|
||||
|
||||
CMAKE_ARGS ?= -DCMAKE_BUILD_TYPE=Release
|
||||
|
||||
ifeq ($(BUILD_TYPE),cublas)
|
||||
CMAKE_ARGS += -DDS4_GPU=cuda
|
||||
DS4_OBJ_TARGET := ds4.o ds4_cuda.o
|
||||
else ifeq ($(UNAME_S),Darwin)
|
||||
CMAKE_ARGS += -DDS4_GPU=metal
|
||||
DS4_OBJ_TARGET := ds4.o ds4_metal.o
|
||||
else
|
||||
# CPU reference path (Linux only - macOS CPU path is broken by VM bug per ds4 README).
|
||||
CMAKE_ARGS += -DDS4_GPU=cpu
|
||||
DS4_OBJ_TARGET := ds4_cpu.o
|
||||
endif
|
||||
|
||||
ifneq ($(NATIVE),true)
|
||||
CMAKE_ARGS += -DDS4_NATIVE=OFF
|
||||
endif
|
||||
|
||||
.PHONY: grpc-server package clean purge test all
|
||||
all: grpc-server
|
||||
|
||||
# Clone the upstream ds4 source at the pinned commit. Directory acts as the
|
||||
# target so make only re-clones when missing. After a DS4_VERSION bump,
|
||||
# run 'make purge && make' to refetch (or rely on CI's clean build).
|
||||
ds4:
|
||||
mkdir -p ds4
|
||||
cd ds4 && \
|
||||
git init -q && \
|
||||
git remote add origin $(DS4_REPO) && \
|
||||
git fetch --depth 1 origin $(DS4_VERSION) && \
|
||||
git checkout FETCH_HEAD
|
||||
|
||||
# Build ds4's engine object files via its own Makefile, which already encodes
|
||||
# the right per-platform compile flags (Objective-C/Metal on Darwin, nvcc on Linux+CUDA).
|
||||
ds4/ds4.o: ds4
|
||||
ifeq ($(BUILD_TYPE),cublas)
|
||||
+$(MAKE) -C ds4 ds4.o ds4_cuda.o
|
||||
else ifeq ($(UNAME_S),Darwin)
|
||||
+$(MAKE) -C ds4 ds4.o ds4_metal.o
|
||||
else
|
||||
+$(MAKE) -C ds4 ds4_cpu.o
|
||||
endif
|
||||
|
||||
grpc-server: ds4/ds4.o
|
||||
mkdir -p $(BUILD_DIR)
|
||||
cd $(BUILD_DIR) && cmake $(CMAKE_ARGS) $(CURRENT_MAKEFILE_DIR) && cmake --build . --config Release -j $(JOBS)
|
||||
cp $(BUILD_DIR)/grpc-server grpc-server
|
||||
|
||||
package: grpc-server
|
||||
bash package.sh
|
||||
|
||||
test:
|
||||
@echo "ds4 backend: e2e coverage at tests/e2e-backends/ (BACKEND_BINARY mode)"
|
||||
|
||||
clean:
|
||||
rm -rf $(BUILD_DIR) grpc-server package
|
||||
if [ -d ds4 ]; then $(MAKE) -C ds4 clean; fi
|
||||
|
||||
purge: clean
|
||||
rm -rf ds4
|
||||
@@ -1,359 +0,0 @@
|
||||
#include "dsml_parser.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <cstdio>
|
||||
#include <cstring>
|
||||
#include <chrono>
|
||||
#include <random>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
namespace ds4cpp {
|
||||
|
||||
namespace {
|
||||
|
||||
constexpr const char *kThinkOpen = "<think>";
|
||||
constexpr const char *kThinkClose = "</think>";
|
||||
constexpr const char *kToolsOpen = "<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "tool_calls>"; // <｜DSML｜tool_calls>
|
||||
constexpr const char *kToolsClose = "</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "tool_calls>"; // </｜DSML｜tool_calls>
|
||||
constexpr const char *kInvokeOpenPfx = "<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "invoke name=\""; // <｜DSML｜invoke name="
|
||||
constexpr const char *kInvokeClose = "</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "invoke>"; // </｜DSML｜invoke>
|
||||
constexpr const char *kParamOpenPfx = "<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "parameter name=\""; // <｜DSML｜parameter name="
|
||||
constexpr const char *kParamClose = "</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "parameter>"; // </｜DSML｜parameter>
|
||||
|
||||
// All structural markers the parser might encounter - used to detect "buf
|
||||
// might be a partial marker, don't drain yet" conditions.
|
||||
const std::vector<std::string> &all_markers() {
|
||||
static const std::vector<std::string> v = {
|
||||
kThinkOpen, kThinkClose,
|
||||
kToolsOpen, kToolsClose,
|
||||
kInvokeOpenPfx, kInvokeClose,
|
||||
kParamOpenPfx, kParamClose,
|
||||
};
|
||||
return v;
|
||||
}
|
||||
|
||||
// Returns true if `buf` could be a *prefix* of any marker (i.e., we should
|
||||
// wait for more text before draining as plain content). The marker-prefix
|
||||
// loop handles fixed markers exactly. For markers with variable-length
|
||||
// internal data (kInvokeOpenPfx, kParamOpenPfx have an open quote, then the
|
||||
// tool/param name, then a closing quote and `>`), we also wait while buf
|
||||
// starts with `<` and has not yet seen a `>`: the leading `<` could be the
|
||||
// start of one of those open markers, or a literal that we can confirm only
|
||||
// once we know what follows. Anything after the first `>` arrives is either
|
||||
// consumed by TryConsumeMarker or emitted as a literal `<` by the caller.
|
||||
bool looks_like_prefix(const std::string &buf) {
|
||||
for (const auto &m : all_markers()) {
|
||||
if (m.size() > buf.size() && m.compare(0, buf.size(), buf) == 0) return true;
|
||||
}
|
||||
if (!buf.empty() && buf[0] == '<' && buf.find('>') == std::string::npos) {
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
bool consume_literal(std::string &buf, const std::string &lit) {
|
||||
if (buf.compare(0, lit.size(), lit) == 0) {
|
||||
buf.erase(0, lit.size());
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
// Find the next '<' in buf starting at offset; returns std::string::npos if none.
|
||||
size_t next_tag(const std::string &buf, size_t off = 0) {
|
||||
return buf.find('<', off);
|
||||
}
|
||||
|
||||
std::string json_escape(const std::string &in) {
|
||||
std::string out;
|
||||
out.reserve(in.size() + 2);
|
||||
for (char c : in) {
|
||||
switch (c) {
|
||||
case '"': out += "\\\""; break;
|
||||
case '\\': out += "\\\\"; break;
|
||||
case '\b': out += "\\b"; break;
|
||||
case '\f': out += "\\f"; break;
|
||||
case '\n': out += "\\n"; break;
|
||||
case '\r': out += "\\r"; break;
|
||||
case '\t': out += "\\t"; break;
|
||||
default:
|
||||
if (static_cast<unsigned char>(c) < 0x20) {
|
||||
char tmp[8];
|
||||
std::snprintf(tmp, sizeof(tmp), "\\u%04x", c);
|
||||
out += tmp;
|
||||
} else {
|
||||
out += c;
|
||||
}
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
DsmlParser::DsmlParser() = default;
|
||||
|
||||
bool DsmlParser::IsInDsmlStructural() const {
|
||||
switch (state_) {
|
||||
case State::TOOL_CALLS:
|
||||
case State::INVOKE:
|
||||
return true;
|
||||
case State::PARAM_VALUE: // payload bytes; user sampling applies
|
||||
case State::TEXT:
|
||||
case State::THINK:
|
||||
return false;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
void DsmlParser::EmitArgsChunk(const std::string &chunk, std::vector<ParserEvent> &out) {
|
||||
if (chunk.empty()) return;
|
||||
ParserEvent e;
|
||||
e.type = ParserEvent::TOOL_ARGS;
|
||||
e.text = chunk;
|
||||
e.index = tool_index_;
|
||||
out.push_back(std::move(e));
|
||||
}
|
||||
|
||||
void DsmlParser::FinishCurrentToolCall(std::vector<ParserEvent> &out) {
|
||||
if (tool_index_ < 0) return;
|
||||
// Close the JSON object that was opened on the first parameter.
|
||||
if (args_emitted_open_brace_) {
|
||||
EmitArgsChunk("}", out);
|
||||
} else {
|
||||
EmitArgsChunk("{}", out);
|
||||
}
|
||||
ParserEvent e;
|
||||
e.type = ParserEvent::TOOL_END;
|
||||
e.index = tool_index_;
|
||||
out.push_back(std::move(e));
|
||||
current_tool_name_.clear();
|
||||
args_emitted_open_brace_ = false;
|
||||
args_param_count_ = 0;
|
||||
}
|
||||
|
||||
bool DsmlParser::TryConsumeMarker(std::vector<ParserEvent> &out) {
|
||||
switch (state_) {
|
||||
case State::TEXT: {
|
||||
if (consume_literal(buf_, kThinkOpen)) { state_ = State::THINK; return true; }
|
||||
if (consume_literal(buf_, kToolsOpen)) { state_ = State::TOOL_CALLS; return true; }
|
||||
return false;
|
||||
}
|
||||
case State::THINK: {
|
||||
if (consume_literal(buf_, kThinkClose)) { state_ = State::TEXT; return true; }
|
||||
return false;
|
||||
}
|
||||
case State::TOOL_CALLS: {
|
||||
if (consume_literal(buf_, kToolsClose)) { state_ = State::TEXT; return true; }
|
||||
// <｜DSML｜invoke name="X">
|
||||
if (buf_.compare(0, std::strlen(kInvokeOpenPfx), kInvokeOpenPfx) == 0) {
|
||||
size_t close_q = buf_.find('"', std::strlen(kInvokeOpenPfx));
|
||||
if (close_q == std::string::npos) return false; // need more bytes
|
||||
size_t close_gt = buf_.find('>', close_q);
|
||||
if (close_gt == std::string::npos) return false;
|
||||
current_tool_name_ = buf_.substr(std::strlen(kInvokeOpenPfx),
|
||||
close_q - std::strlen(kInvokeOpenPfx));
|
||||
tool_index_++;
|
||||
buf_.erase(0, close_gt + 1);
|
||||
ParserEvent e;
|
||||
e.type = ParserEvent::TOOL_START;
|
||||
e.tool_name = current_tool_name_;
|
||||
e.tool_id = RandomToolId();
|
||||
e.index = tool_index_;
|
||||
out.push_back(std::move(e));
|
||||
args_emitted_open_brace_ = false;
|
||||
args_param_count_ = 0;
|
||||
state_ = State::INVOKE;
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
case State::INVOKE: {
|
||||
if (consume_literal(buf_, kInvokeClose)) {
|
||||
FinishCurrentToolCall(out);
|
||||
state_ = State::TOOL_CALLS;
|
||||
return true;
|
||||
}
|
||||
// <｜DSML｜parameter name="K" string="true|false">
|
||||
if (buf_.compare(0, std::strlen(kParamOpenPfx), kParamOpenPfx) == 0) {
|
||||
size_t close_q = buf_.find('"', std::strlen(kParamOpenPfx));
|
||||
if (close_q == std::string::npos) return false;
|
||||
size_t string_attr = buf_.find("string=\"", close_q);
|
||||
if (string_attr == std::string::npos) return false;
|
||||
size_t string_q = buf_.find('"', string_attr + 8);
|
||||
if (string_q == std::string::npos) return false;
|
||||
size_t close_gt = buf_.find('>', string_q);
|
||||
if (close_gt == std::string::npos) return false;
|
||||
param_name_ = buf_.substr(std::strlen(kParamOpenPfx),
|
||||
close_q - std::strlen(kParamOpenPfx));
|
||||
std::string string_val = buf_.substr(string_attr + 8,
|
||||
string_q - (string_attr + 8));
|
||||
param_is_string_ = (string_val == "true");
|
||||
param_value_.clear();
|
||||
buf_.erase(0, close_gt + 1);
|
||||
// Emit args JSON opener / separator.
|
||||
std::string opener;
|
||||
if (!args_emitted_open_brace_) { opener = "{"; args_emitted_open_brace_ = true; }
|
||||
else { opener = ","; }
|
||||
opener += "\"" + json_escape(param_name_) + "\":";
|
||||
if (param_is_string_) opener += "\"";
|
||||
EmitArgsChunk(opener, out);
|
||||
args_param_count_++;
|
||||
state_ = State::PARAM_VALUE;
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
case State::PARAM_VALUE: {
|
||||
if (consume_literal(buf_, kParamClose)) {
|
||||
if (param_is_string_) EmitArgsChunk("\"", out);
|
||||
state_ = State::INVOKE;
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
void DsmlParser::DrainPlain(std::vector<ParserEvent> &out) {
|
||||
// Drain everything up to the next '<' that *might* start a marker.
|
||||
// Anything before the next '<' is safe to emit; the '<...' tail stays buffered.
|
||||
while (!buf_.empty()) {
|
||||
size_t lt = next_tag(buf_, 0);
|
||||
if (lt == std::string::npos) {
|
||||
// No tag at all - emit (or accumulate) the whole buffer.
|
||||
ParserEvent e;
|
||||
if (state_ == State::PARAM_VALUE) {
|
||||
std::string esc = param_is_string_ ? json_escape(buf_) : buf_;
|
||||
EmitArgsChunk(esc, out);
|
||||
} else if (state_ == State::THINK) {
|
||||
e.type = ParserEvent::REASONING;
|
||||
e.text = buf_;
|
||||
out.push_back(std::move(e));
|
||||
} else if (state_ == State::TEXT) {
|
||||
e.type = ParserEvent::CONTENT;
|
||||
e.text = buf_;
|
||||
out.push_back(std::move(e));
|
||||
}
|
||||
// Inside INVOKE / TOOL_CALLS with no marker, raw bytes are
|
||||
// structural whitespace - discard.
|
||||
buf_.clear();
|
||||
return;
|
||||
}
|
||||
if (lt > 0) {
|
||||
std::string chunk = buf_.substr(0, lt);
|
||||
buf_.erase(0, lt);
|
||||
ParserEvent e;
|
||||
if (state_ == State::PARAM_VALUE) {
|
||||
std::string esc = param_is_string_ ? json_escape(chunk) : chunk;
|
||||
EmitArgsChunk(esc, out);
|
||||
} else if (state_ == State::THINK) {
|
||||
e.type = ParserEvent::REASONING;
|
||||
e.text = chunk;
|
||||
out.push_back(std::move(e));
|
||||
} else if (state_ == State::TEXT) {
|
||||
e.type = ParserEvent::CONTENT;
|
||||
e.text = chunk;
|
||||
out.push_back(std::move(e));
|
||||
}
|
||||
}
|
||||
// buf_[0] == '<' - try consuming a marker. If we consumed one, loop again.
|
||||
if (!TryConsumeMarker(out)) {
|
||||
// Could be a partial marker - wait for more bytes.
|
||||
if (looks_like_prefix(buf_)) return;
|
||||
// Otherwise this '<' is a literal - emit one char and continue.
|
||||
std::string one(1, buf_[0]);
|
||||
buf_.erase(0, 1);
|
||||
ParserEvent e;
|
||||
if (state_ == State::PARAM_VALUE) {
|
||||
std::string esc = param_is_string_ ? json_escape(one) : one;
|
||||
EmitArgsChunk(esc, out);
|
||||
} else if (state_ == State::THINK) {
|
||||
e.type = ParserEvent::REASONING;
|
||||
e.text = one;
|
||||
out.push_back(std::move(e));
|
||||
} else if (state_ == State::TEXT) {
|
||||
e.type = ParserEvent::CONTENT;
|
||||
e.text = one;
|
||||
out.push_back(std::move(e));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
void DsmlParser::Feed(const std::string &chunk, std::vector<ParserEvent> &out) {
|
||||
buf_ += chunk;
|
||||
DrainPlain(out);
|
||||
}
|
||||
|
||||
void DsmlParser::Flush(std::vector<ParserEvent> &out) {
|
||||
// At flush time we no longer wait for marker completion - drain everything
|
||||
// (the trailing bytes won't grow). Mirror DrainPlain's state-aware
|
||||
// classification: PARAM_VALUE bytes become TOOL_ARGS, THINK bytes become
|
||||
// REASONING, TEXT bytes become CONTENT, and INVOKE/TOOL_CALLS bytes are
|
||||
// structural whitespace (discarded).
|
||||
auto emit_plain = [&](const std::string &chunk) {
|
||||
if (chunk.empty()) return;
|
||||
if (state_ == State::PARAM_VALUE) {
|
||||
std::string esc = param_is_string_ ? json_escape(chunk) : chunk;
|
||||
EmitArgsChunk(esc, out);
|
||||
return;
|
||||
}
|
||||
if (state_ == State::THINK) {
|
||||
ParserEvent e;
|
||||
e.type = ParserEvent::REASONING;
|
||||
e.text = chunk;
|
||||
out.push_back(std::move(e));
|
||||
return;
|
||||
}
|
||||
if (state_ == State::TEXT) {
|
||||
ParserEvent e;
|
||||
e.type = ParserEvent::CONTENT;
|
||||
e.text = chunk;
|
||||
out.push_back(std::move(e));
|
||||
return;
|
||||
}
|
||||
// INVOKE / TOOL_CALLS: structural whitespace, discard.
|
||||
};
|
||||
while (!buf_.empty()) {
|
||||
size_t lt = next_tag(buf_, 0);
|
||||
if (lt == std::string::npos) {
|
||||
emit_plain(buf_);
|
||||
buf_.clear();
|
||||
return;
|
||||
}
|
||||
if (lt > 0) {
|
||||
std::string chunk = buf_.substr(0, lt);
|
||||
buf_.erase(0, lt);
|
||||
emit_plain(chunk);
|
||||
}
|
||||
if (!TryConsumeMarker(out)) {
|
||||
// Definitely a literal '<' now (no chance of more bytes arriving).
|
||||
std::string one(1, buf_[0]);
|
||||
buf_.erase(0, 1);
|
||||
emit_plain(one);
|
||||
}
|
||||
}
|
||||
// If we ended mid-tool-call (model truncated), close it cleanly.
|
||||
if (state_ == State::INVOKE || state_ == State::PARAM_VALUE) {
|
||||
if (state_ == State::PARAM_VALUE && param_is_string_) EmitArgsChunk("\"", out);
|
||||
FinishCurrentToolCall(out);
|
||||
state_ = State::TEXT;
|
||||
}
|
||||
}
|
||||
|
||||
std::string RandomToolId() {
|
||||
static thread_local std::mt19937_64 rng{
|
||||
static_cast<uint64_t>(std::chrono::system_clock::now().time_since_epoch().count())};
|
||||
const char *alphabet =
|
||||
"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
|
||||
std::string out = "call_";
|
||||
for (int i = 0; i < 16; ++i) {
|
||||
out += alphabet[rng() % 62];
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
} // namespace ds4cpp
|
||||
@@ -1,77 +0,0 @@
|
||||
#pragma once
|
||||
#include <functional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
namespace ds4cpp {
|
||||
|
||||
struct ParserEvent {
|
||||
enum Type { CONTENT, REASONING, TOOL_START, TOOL_ARGS, TOOL_END };
|
||||
Type type;
|
||||
std::string text; // CONTENT, REASONING, TOOL_ARGS
|
||||
std::string tool_name; // TOOL_START
|
||||
std::string tool_id; // TOOL_START (caller-assigned)
|
||||
int index = 0; // TOOL_START / TOOL_ARGS / TOOL_END
|
||||
};
|
||||
|
||||
// Streaming parser. Stateless across instances; one per Predict call.
|
||||
class DsmlParser {
|
||||
public:
|
||||
DsmlParser();
|
||||
|
||||
// Feed a chunk of raw model-emitted text. Appends classified events to
|
||||
// `out`. May buffer the tail of `chunk` internally if it looks like a
|
||||
// marker prefix.
|
||||
void Feed(const std::string &chunk, std::vector<ParserEvent> &out);
|
||||
|
||||
// Flush any remaining buffered text as CONTENT (called at generation end).
|
||||
void Flush(std::vector<ParserEvent> &out);
|
||||
|
||||
// True when the parser is inside a DSML structural position - that is,
|
||||
// tags/markers between tool-call boundaries where the model is expected
|
||||
// to emit protocol bytes verbatim. Mirrors ds4_server.c's "force
|
||||
// temperature=0 unless dsml_decode_state_uses_payload_sampling" rule:
|
||||
//
|
||||
// TEXT / THINK -> false (user sampling applies)
|
||||
// PARAM_VALUE -> false (payload uses user sampling)
|
||||
// TOOL_CALLS / INVOKE -> true (structural; force greedy)
|
||||
//
|
||||
// Callers should use this BEFORE the next sample() call to pick the
|
||||
// effective temperature; the parser's state reflects what's already
|
||||
// been consumed, so it predicts the next token's classification.
|
||||
bool IsInDsmlStructural() const;
|
||||
|
||||
private:
|
||||
enum class State { TEXT, THINK, TOOL_CALLS, INVOKE, PARAM_VALUE };
|
||||
State state_ = State::TEXT;
|
||||
std::string buf_;
|
||||
std::string current_tool_name_;
|
||||
int tool_index_ = -1;
|
||||
// While parsing a parameter value:
|
||||
std::string param_name_;
|
||||
bool param_is_string_ = true;
|
||||
std::string param_value_;
|
||||
// Incrementally-built arguments JSON for the active tool call.
|
||||
std::string args_json_so_far_;
|
||||
bool args_emitted_open_brace_ = false;
|
||||
int args_param_count_ = 0;
|
||||
|
||||
// Try to consume one structural marker starting at buf_[0]. Returns true
|
||||
// and advances state if a complete marker was consumed; false if the
|
||||
// buffer is ambiguous (could be a marker prefix).
|
||||
bool TryConsumeMarker(std::vector<ParserEvent> &out);
|
||||
|
||||
// Drain plain text from buf_ as far as we're sure it's not a marker prefix.
|
||||
// Emits CONTENT or REASONING depending on current state.
|
||||
void DrainPlain(std::vector<ParserEvent> &out);
|
||||
|
||||
// Emit the next chunk of arguments JSON to the consumer.
|
||||
void EmitArgsChunk(const std::string &chunk, std::vector<ParserEvent> &out);
|
||||
void FinishCurrentToolCall(std::vector<ParserEvent> &out);
|
||||
};
|
||||
|
||||
// Generate a random tool call ID (e.g. "call_AbCdEf"). Used by the gRPC layer
|
||||
// when assigning IDs to streamed tool calls.
|
||||
std::string RandomToolId();
|
||||
|
||||
} // namespace ds4cpp
|
||||
@@ -1,140 +0,0 @@
|
||||
#include "dsml_renderer.h"
|
||||
|
||||
// We accept either nlohmann::json (if available) or fall back to a tiny
|
||||
// hand-rolled parser. The LocalAI tree already has nlohmann/json bundled
|
||||
// in vendor paths; we use the apt-installed nlohmann-json3-dev (installed
|
||||
// in Task 11 step 1) when present, otherwise the bundled copy.
|
||||
#if __has_include(<nlohmann/json.hpp>)
|
||||
#include <nlohmann/json.hpp>
|
||||
using json = nlohmann::json;
|
||||
#else
|
||||
#error "nlohmann/json.hpp not found; install nlohmann-json3-dev"
|
||||
#endif
|
||||
|
||||
#include <sstream>
|
||||
|
||||
namespace ds4cpp {
|
||||
|
||||
namespace {
|
||||
|
||||
void render_param(std::ostringstream &os, const std::string &name,
|
||||
const json &value) {
|
||||
bool is_string = value.is_string();
|
||||
os << "<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "parameter name=\"" << name
|
||||
<< "\" string=\"" << (is_string ? "true" : "false") << "\">";
|
||||
if (is_string) {
|
||||
os << value.get<std::string>();
|
||||
} else {
|
||||
os << value.dump();
|
||||
}
|
||||
os << "</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "parameter>\n";
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
std::string RenderAssistantToolCalls(const std::string &tool_calls_json) {
|
||||
if (tool_calls_json.empty()) return "";
|
||||
json arr;
|
||||
try {
|
||||
arr = json::parse(tool_calls_json);
|
||||
} catch (const std::exception &) {
|
||||
return "";
|
||||
}
|
||||
if (!arr.is_array() || arr.empty()) return "";
|
||||
|
||||
std::ostringstream os;
|
||||
os << "\n\n<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "tool_calls>\n";
|
||||
for (const auto &call : arr) {
|
||||
// OpenAI shape: { id, type, function: { name, arguments (JSON string) } }
|
||||
// Anthropic shape comes through normalized by LocalAI.
|
||||
std::string name;
|
||||
std::string args_str;
|
||||
if (call.contains("function")) {
|
||||
const auto &fn = call["function"];
|
||||
if (fn.contains("name") && fn["name"].is_string())
|
||||
name = fn["name"].get<std::string>();
|
||||
if (fn.contains("arguments") && fn["arguments"].is_string())
|
||||
args_str = fn["arguments"].get<std::string>();
|
||||
}
|
||||
os << "<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "invoke name=\"" << name << "\">\n";
|
||||
if (!args_str.empty()) {
|
||||
json args;
|
||||
try {
|
||||
args = json::parse(args_str);
|
||||
} catch (...) {
|
||||
args = json{};
|
||||
}
|
||||
if (args.is_object()) {
|
||||
for (auto it = args.begin(); it != args.end(); ++it) {
|
||||
render_param(os, it.key(), it.value());
|
||||
}
|
||||
}
|
||||
}
|
||||
os << "</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "invoke>\n";
|
||||
}
|
||||
os << "</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "tool_calls>";
|
||||
return os.str();
|
||||
}
|
||||
|
||||
std::string RenderToolResult(const std::string &tool_call_id, const std::string &content) {
|
||||
std::ostringstream os;
|
||||
// ds4_server.c wraps tool results in a "tool_result" DSML tag carrying
|
||||
// the tool_call_id. Match that shape.
|
||||
os << "<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "tool_result id=\"" << tool_call_id << "\">"
|
||||
<< content
|
||||
<< "</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "tool_result>";
|
||||
return os.str();
|
||||
}
|
||||
|
||||
std::string RenderToolsManifest(const std::string &tools_json) {
|
||||
if (tools_json.empty()) return "";
|
||||
json arr;
|
||||
try {
|
||||
arr = json::parse(tools_json);
|
||||
} catch (const std::exception &) {
|
||||
return "";
|
||||
}
|
||||
if (!arr.is_array() || arr.empty()) return "";
|
||||
|
||||
// Extract each OpenAI tool's `function` object, dump as compact JSON, one
|
||||
// per line. Mirrors openai_function_schema_from_tool() in ds4_server.c.
|
||||
std::ostringstream schemas;
|
||||
for (const auto &tool : arr) {
|
||||
if (tool.contains("function") && tool["function"].is_object()) {
|
||||
schemas << tool["function"].dump() << "\n";
|
||||
} else if (tool.is_object()) {
|
||||
// Anthropic / direct-schema form: pass through.
|
||||
schemas << tool.dump() << "\n";
|
||||
}
|
||||
}
|
||||
if (schemas.tellp() == std::streampos(0)) return "";
|
||||
|
||||
// Verbatim text from ds4_server.c append_tools_prompt_text. Do NOT
|
||||
// paraphrase - the model was trained on these exact bytes.
|
||||
std::ostringstream os;
|
||||
os << "## Tools\n\n"
|
||||
"You have access to a set of tools to help answer the user question. "
|
||||
"You can invoke tools by writing a \"<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "tool_calls>\" block like the following:\n\n"
|
||||
"<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "tool_calls>\n"
|
||||
"<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "invoke name=\"$TOOL_NAME\">\n"
|
||||
"<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "parameter name=\"$PARAMETER_NAME\" string=\"true|false\">$PARAMETER_VALUE</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "parameter>\n"
|
||||
"...\n"
|
||||
"</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "invoke>\n"
|
||||
"<\xef\xbd\x9c" "DSML\xef\xbd\x9c" "invoke name=\"$TOOL_NAME2\">\n"
|
||||
"...\n"
|
||||
"</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "invoke>\n"
|
||||
"</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "tool_calls>\n\n"
|
||||
"String parameters should be specified as raw text and set `string=\"true\"`. "
|
||||
"Preserve characters such as `>`, `&`, and `&&` exactly; never replace normal string characters with XML or HTML entity escapes. "
|
||||
"Only if a string value itself contains the exact closing parameter tag `</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "parameter>`, write that tag as `</\xef\xbd\x9c" "DSML\xef\xbd\x9c" "parameter>` inside the value. "
|
||||
"For all other types (numbers, booleans, arrays, objects), pass the value in JSON format and set `string=\"false\"`.\n\n"
|
||||
"If thinking_mode is enabled (triggered by <think>), you MUST output your complete reasoning inside <think>...</think> BEFORE any tool calls or final response.\n\n"
|
||||
"Otherwise, output directly after </think> with tool calls or final response.\n\n"
|
||||
"### Available Tool Schemas\n\n"
|
||||
<< schemas.str()
|
||||
<< "\nYou MUST strictly follow the above defined tool name and parameter schemas to invoke tool calls. "
|
||||
"Use the exact parameter names from the schemas.";
|
||||
return os.str();
|
||||
}
|
||||
|
||||
} // namespace ds4cpp
|
||||
@@ -1,27 +0,0 @@
|
||||
#pragma once
|
||||
#include <string>
|
||||
|
||||
namespace ds4cpp {
|
||||
|
||||
// Render an assistant message's tool_calls JSON array into the DSML block
|
||||
// that ds4 expects in its prompt. `tool_calls_json` is the value of
|
||||
// proto.Message.tool_calls (OpenAI shape: array of {id, type, function:{name, arguments}}).
|
||||
// Returns the DSML text to append after the assistant's content.
|
||||
std::string RenderAssistantToolCalls(const std::string &tool_calls_json);
|
||||
|
||||
// Render a role="tool" message into the DSML "tool result" block. ds4's
|
||||
// prompt template expects tool results inside a specific tag; we wrap the
|
||||
// `content` with that tag and include the `tool_call_id` so the model can
|
||||
// correlate.
|
||||
std::string RenderToolResult(const std::string &tool_call_id, const std::string &content);
|
||||
|
||||
// Render the "## Tools" manifest that ds4 expects in the SYSTEM prompt when
|
||||
// tools are available. Without this preamble the model has no idea tools
|
||||
// exist and will not emit DSML tool calls. Mirrors append_tools_prompt_text()
|
||||
// in ds4_server.c (~line 1646): a fixed preamble + "### Available Tool
|
||||
// Schemas" section + one JSON schema per line (extracted from each OpenAI
|
||||
// tool's .function object) + a fixed closing instruction. Returns empty
|
||||
// when tools_json is empty / unparseable.
|
||||
std::string RenderToolsManifest(const std::string &tools_json);
|
||||
|
||||
} // namespace ds4cpp
|
||||
@@ -1,696 +0,0 @@
|
||||
// ds4 LocalAI gRPC backend.
|
||||
//
|
||||
// Wraps antirez/ds4's `ds4_engine_*` / `ds4_session_*` public API
|
||||
// (see ds4/ds4.h) over LocalAI's backend.proto. Tool calls, thinking
|
||||
// mode, and disk KV cache are wired in follow-up commits; this commit
|
||||
// is just the bind/listen/Health/Free skeleton.
|
||||
|
||||
#include "backend.pb.h"
|
||||
#include "backend.grpc.pb.h"
|
||||
|
||||
#include "dsml_parser.h" // populated in Task 12
|
||||
#include "dsml_renderer.h" // populated in Task 16
|
||||
#include "kv_cache.h" // populated in Task 17
|
||||
|
||||
extern "C" {
|
||||
#include "ds4.h"
|
||||
}
|
||||
|
||||
#include <grpcpp/grpcpp.h>
|
||||
#include <grpcpp/server.h>
|
||||
#include <grpcpp/server_builder.h>
|
||||
#include <grpcpp/ext/proto_server_reflection_plugin.h>
|
||||
|
||||
#include <atomic>
|
||||
#include <chrono>
|
||||
#include <csignal>
|
||||
#include <cstring>
|
||||
#include <iostream>
|
||||
#include <memory>
|
||||
#include <mutex>
|
||||
#include <string>
|
||||
#include <thread>
|
||||
#include <vector>
|
||||
|
||||
using grpc::Server;
|
||||
using grpc::ServerBuilder;
|
||||
using grpc::ServerContext;
|
||||
using grpc::ServerWriter;
|
||||
// NOTE: do NOT alias `grpc::Status` as `Status` - the Status RPC method below
|
||||
// would shadow the type, breaking the other RPC method declarations that use
|
||||
// it as a return type. Use GStatus instead.
|
||||
using GStatus = ::grpc::Status;
|
||||
using grpc::StatusCode;
|
||||
|
||||
namespace {
|
||||
|
||||
// Global state - ds4 is single-engine-per-process by design.
|
||||
std::mutex g_engine_mu;
|
||||
ds4_engine *g_engine = nullptr;
|
||||
ds4_session *g_session = nullptr;
|
||||
int g_ctx_size = 32768;
|
||||
std::string g_kv_cache_dir; // empty disables disk cache
|
||||
|
||||
std::atomic<Server *> g_server{nullptr};
|
||||
|
||||
// Parse a "key:value" option string. Returns empty when no colon.
|
||||
static std::pair<std::string, std::string> split_option(const std::string &opt) {
|
||||
auto colon = opt.find(':');
|
||||
if (colon == std::string::npos) return {opt, ""};
|
||||
return {opt.substr(0, colon), opt.substr(colon + 1)};
|
||||
}
|
||||
|
||||
static void append_token_text(ds4_engine *engine, int token, std::string &out) {
|
||||
size_t len = 0;
|
||||
const char *text = ds4_token_text(engine, token, &len);
|
||||
if (text && len > 0) out.append(text, len);
|
||||
}
|
||||
|
||||
struct CollectCtx {
|
||||
ds4_engine *engine;
|
||||
std::string raw_buf; // exact raw bytes for Reply.message
|
||||
ds4cpp::DsmlParser parser;
|
||||
backend::Reply *reply;
|
||||
int tokens;
|
||||
|
||||
// Per-tool aggregation: accumulate ChatDelta tool_calls so we emit one
|
||||
// delta with all calls, mirroring how vllm's non-streaming path returns.
|
||||
struct Pending {
|
||||
std::string id;
|
||||
std::string name;
|
||||
std::string args;
|
||||
};
|
||||
std::vector<Pending> pending;
|
||||
|
||||
std::string content_buf;
|
||||
std::string reasoning_buf;
|
||||
};
|
||||
|
||||
static void apply_events(CollectCtx *c, const std::vector<ds4cpp::ParserEvent> &events) {
|
||||
for (const auto &e : events) {
|
||||
switch (e.type) {
|
||||
case ds4cpp::ParserEvent::CONTENT:
|
||||
c->content_buf += e.text;
|
||||
break;
|
||||
case ds4cpp::ParserEvent::REASONING:
|
||||
c->reasoning_buf += e.text;
|
||||
break;
|
||||
case ds4cpp::ParserEvent::TOOL_START:
|
||||
if ((int)c->pending.size() <= e.index)
|
||||
c->pending.resize(e.index + 1);
|
||||
c->pending[e.index].id = e.tool_id;
|
||||
c->pending[e.index].name = e.tool_name;
|
||||
break;
|
||||
case ds4cpp::ParserEvent::TOOL_ARGS:
|
||||
if ((int)c->pending.size() > e.index)
|
||||
c->pending[e.index].args += e.text;
|
||||
break;
|
||||
case ds4cpp::ParserEvent::TOOL_END:
|
||||
// No-op for non-streaming: the final delta is emitted at the end.
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static void collect_emit(void *ud, int token) {
|
||||
auto *c = static_cast<CollectCtx *>(ud);
|
||||
if (token == ds4_token_eos(c->engine)) return;
|
||||
size_t len = 0;
|
||||
const char *text = ds4_token_text(c->engine, token, &len);
|
||||
if (!text || len == 0) return;
|
||||
std::string chunk(text, len);
|
||||
c->raw_buf += chunk;
|
||||
std::vector<ds4cpp::ParserEvent> events;
|
||||
c->parser.Feed(chunk, events);
|
||||
apply_events(c, events);
|
||||
c->tokens++;
|
||||
}
|
||||
static void collect_done(void *) {}
|
||||
|
||||
struct StreamCtx {
|
||||
ds4_engine *engine;
|
||||
ServerWriter<backend::Reply> *writer;
|
||||
ds4cpp::DsmlParser parser;
|
||||
int tokens;
|
||||
bool aborted;
|
||||
// Track which tool indices we've seen TOOL_START for, so subsequent
|
||||
// ARGS deltas can elide the redundant id/name fields.
|
||||
std::vector<bool> tool_started;
|
||||
};
|
||||
|
||||
static void stream_emit(void *ud, int token) {
|
||||
auto *s = static_cast<StreamCtx *>(ud);
|
||||
if (s->aborted) return;
|
||||
if (token == ds4_token_eos(s->engine)) return;
|
||||
size_t len = 0;
|
||||
const char *text = ds4_token_text(s->engine, token, &len);
|
||||
if (!text || len == 0) return;
|
||||
std::string chunk(text, len);
|
||||
std::vector<ds4cpp::ParserEvent> events;
|
||||
s->parser.Feed(chunk, events);
|
||||
if (events.empty()) { s->tokens++; return; }
|
||||
|
||||
backend::Reply reply;
|
||||
auto *delta = reply.add_chat_deltas();
|
||||
bool any_field = false;
|
||||
for (const auto &e : events) {
|
||||
switch (e.type) {
|
||||
case ds4cpp::ParserEvent::CONTENT:
|
||||
delta->set_content(delta->content() + e.text);
|
||||
any_field = true;
|
||||
break;
|
||||
case ds4cpp::ParserEvent::REASONING:
|
||||
delta->set_reasoning_content(delta->reasoning_content() + e.text);
|
||||
any_field = true;
|
||||
break;
|
||||
case ds4cpp::ParserEvent::TOOL_START: {
|
||||
if ((int)s->tool_started.size() <= e.index)
|
||||
s->tool_started.resize(e.index + 1, false);
|
||||
s->tool_started[e.index] = true;
|
||||
auto *tc = delta->add_tool_calls();
|
||||
tc->set_index(e.index);
|
||||
tc->set_id(e.tool_id);
|
||||
tc->set_name(e.tool_name);
|
||||
any_field = true;
|
||||
break;
|
||||
}
|
||||
case ds4cpp::ParserEvent::TOOL_ARGS: {
|
||||
auto *tc = delta->add_tool_calls();
|
||||
tc->set_index(e.index);
|
||||
tc->set_arguments(e.text);
|
||||
any_field = true;
|
||||
break;
|
||||
}
|
||||
case ds4cpp::ParserEvent::TOOL_END:
|
||||
// No marker delta needed - the Go side closes the tool call on
|
||||
// the final aggregator pass.
|
||||
break;
|
||||
}
|
||||
}
|
||||
reply.set_message(chunk);
|
||||
reply.set_tokens(1);
|
||||
if (any_field) {
|
||||
if (!s->writer->Write(reply)) s->aborted = true;
|
||||
}
|
||||
s->tokens++;
|
||||
}
|
||||
static void stream_done(void *) {}
|
||||
|
||||
// Per-thread RNG seed for ds4_session_sample. Initialized lazily from
|
||||
// system_clock; ds4 owns the random walk after that.
|
||||
static uint64_t *get_rng() {
|
||||
static thread_local uint64_t seed = 0;
|
||||
if (seed == 0) {
|
||||
seed = static_cast<uint64_t>(
|
||||
std::chrono::system_clock::now().time_since_epoch().count());
|
||||
if (seed == 0) seed = 1;
|
||||
}
|
||||
return &seed;
|
||||
}
|
||||
|
||||
struct SampleParams {
|
||||
float temperature;
|
||||
int top_k;
|
||||
float top_p;
|
||||
float min_p;
|
||||
};
|
||||
|
||||
// Compute the effective sampling parameters for the next token, mirroring
|
||||
// ds4_server.c:7102-7115:
|
||||
// - thinking mode enabled -> override (T=1, top_k=0, top_p=1, min_p=0)
|
||||
// - inside DSML structural position (tool-call markers) -> force T=0
|
||||
// - otherwise -> the request's user-supplied sampling settings
|
||||
// The parser argument carries state from tokens emitted so far; its
|
||||
// IsInDsmlStructural() predicts the next token's classification.
|
||||
static SampleParams compute_sample_params(const backend::PredictOptions *request,
|
||||
const ds4cpp::DsmlParser &parser,
|
||||
bool think_enabled);
|
||||
|
||||
static ds4_think_mode parse_think_mode(const backend::PredictOptions *request) {
|
||||
// Per the vllm backend convention, "enable_thinking" gates thinking on/off,
|
||||
// and "reasoning_effort" picks the strength when on.
|
||||
const auto &md = request->metadata();
|
||||
auto et = md.find("enable_thinking");
|
||||
bool enabled = true; // default ON per ds4-server
|
||||
if (et != md.end()) enabled = (et->second == "true" || et->second == "1");
|
||||
if (!enabled) return DS4_THINK_NONE;
|
||||
auto re = md.find("reasoning_effort");
|
||||
if (re != md.end() && (re->second == "max" || re->second == "xhigh"))
|
||||
return DS4_THINK_MAX;
|
||||
return DS4_THINK_HIGH;
|
||||
}
|
||||
|
||||
static SampleParams compute_sample_params(const backend::PredictOptions *request,
|
||||
const ds4cpp::DsmlParser &parser,
|
||||
bool think_enabled) {
|
||||
SampleParams p = {
|
||||
request->temperature(),
|
||||
request->topk(),
|
||||
request->topp(),
|
||||
request->minp(),
|
||||
};
|
||||
if (think_enabled) {
|
||||
// Match ds4-server: thinking mode wants creativity in the reasoning
|
||||
// pass and the trailing content, so the entire generation overrides
|
||||
// sampling unless DSML structural bytes take over below.
|
||||
p.temperature = 1.0f;
|
||||
p.top_k = 0;
|
||||
p.top_p = 1.0f;
|
||||
p.min_p = 0.0f;
|
||||
}
|
||||
if (parser.IsInDsmlStructural()) {
|
||||
// Tool-call structural bytes (tags, markers, headers) must parse
|
||||
// cleanly. Force greedy regardless of user/thinking settings.
|
||||
p.temperature = 0.0f;
|
||||
}
|
||||
return p;
|
||||
}
|
||||
|
||||
// Build the rendered text for cache keying. We feed the same text the model
|
||||
// will see; that lets the cache survive small client-side reformatting of
|
||||
// chat history (the cache is keyed on bytes, not tokens).
|
||||
static std::string render_prompt_text(const backend::PredictOptions *request) {
|
||||
// Two-mode: either the raw prompt or the chat-template path. We mirror
|
||||
// build_prompt's branching but accumulate text (not tokens) so we can
|
||||
// SHA1 it for the cache key. ds4_session caches a tokens-indexed
|
||||
// checkpoint, but the disk format keys on bytes per ds4-server's design.
|
||||
if (!request->usetokenizertemplate() || request->messages_size() == 0) {
|
||||
return request->prompt();
|
||||
}
|
||||
std::string out;
|
||||
const std::string sys_role = "system";
|
||||
for (const auto &m : request->messages()) {
|
||||
if (m.role() == sys_role) { out += "[sys] " + m.content() + "\n"; break; }
|
||||
}
|
||||
for (const auto &m : request->messages()) {
|
||||
if (m.role() == sys_role) continue;
|
||||
out += "[" + m.role() + "] " + m.content() + "\n";
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
ds4cpp::KvCache g_kv_cache;
|
||||
|
||||
// Try to recover prefill state for `rendered`. Returns the matched prefix length.
|
||||
static size_t maybe_load_cache(const std::string &rendered) {
|
||||
if (!g_kv_cache.enabled() || !g_session) return 0;
|
||||
return g_kv_cache.LoadLongestPrefix(g_session, rendered, g_ctx_size);
|
||||
}
|
||||
|
||||
static void maybe_save_cache(const std::string &rendered) {
|
||||
if (g_kv_cache.enabled() && g_session) {
|
||||
g_kv_cache.Save(g_session, rendered, g_ctx_size);
|
||||
}
|
||||
}
|
||||
|
||||
static void build_prompt(ds4_engine *engine, const backend::PredictOptions *request,
|
||||
ds4_tokens *out) {
|
||||
if (!request->usetokenizertemplate() || request->messages_size() == 0) {
|
||||
ds4_tokenize_text(engine, request->prompt().c_str(), out);
|
||||
return;
|
||||
}
|
||||
// Chat-template path: render via ds4's helpers.
|
||||
ds4_chat_begin(engine, out);
|
||||
|
||||
ds4_think_mode think = parse_think_mode(request);
|
||||
|
||||
// ds4_encode_chat_prompt is convenient when there is exactly one
|
||||
// system+user pair, but for arbitrary turn lists we use the granular
|
||||
// append helpers. Pull the first system message (if any), then append
|
||||
// every other message in order.
|
||||
const std::string sys_role = "system";
|
||||
std::string system_text;
|
||||
for (const auto &m : request->messages()) {
|
||||
if (m.role() == sys_role) { system_text = m.content(); break; }
|
||||
}
|
||||
// Inject the tools manifest into the system prompt when tools are present.
|
||||
// ds4 was trained to emit DSML tool calls ONLY when this preamble is in
|
||||
// the system message - without it, the model has no idea tools exist and
|
||||
// the e2e tool-call test will fail. The renderer lives in dsml_renderer
|
||||
// and is a verbatim port of ds4_server.c's append_tools_prompt_text.
|
||||
std::string tools_manifest;
|
||||
if (!request->tools().empty()) {
|
||||
tools_manifest = ds4cpp::RenderToolsManifest(request->tools());
|
||||
}
|
||||
if (!system_text.empty() || !tools_manifest.empty()) {
|
||||
std::string combined = system_text;
|
||||
if (!tools_manifest.empty()) {
|
||||
if (!combined.empty()) combined += "\n\n";
|
||||
combined += tools_manifest;
|
||||
}
|
||||
ds4_chat_append_message(engine, out, "system", combined.c_str());
|
||||
}
|
||||
for (const auto &m : request->messages()) {
|
||||
if (m.role() == sys_role) continue;
|
||||
if (m.role() == "assistant" && !m.tool_calls().empty()) {
|
||||
std::string combined = m.content();
|
||||
combined += ds4cpp::RenderAssistantToolCalls(m.tool_calls());
|
||||
ds4_chat_append_message(engine, out, "assistant", combined.c_str());
|
||||
} else if (m.role() == "tool") {
|
||||
std::string body = ds4cpp::RenderToolResult(m.tool_call_id(), m.content());
|
||||
ds4_chat_append_message(engine, out, "user", body.c_str());
|
||||
} else {
|
||||
ds4_chat_append_message(engine, out, m.role().c_str(), m.content().c_str());
|
||||
}
|
||||
}
|
||||
ds4_chat_append_assistant_prefix(engine, out, think);
|
||||
}
|
||||
|
||||
class DS4Backend final : public backend::Backend::Service {
|
||||
public:
|
||||
GStatus Health(ServerContext *, const backend::HealthMessage *,
|
||||
backend::Reply *reply) override {
|
||||
reply->set_message(std::string("OK"));
|
||||
return GStatus::OK;
|
||||
}
|
||||
|
||||
GStatus Free(ServerContext *, const backend::HealthMessage *,
|
||||
backend::Result *result) override {
|
||||
std::lock_guard<std::mutex> lock(g_engine_mu);
|
||||
if (g_session) { ds4_session_free(g_session); g_session = nullptr; }
|
||||
if (g_engine) { ds4_engine_close(g_engine); g_engine = nullptr; }
|
||||
result->set_success(true);
|
||||
return GStatus::OK;
|
||||
}
|
||||
|
||||
GStatus LoadModel(ServerContext *, const backend::ModelOptions *request,
|
||||
backend::Result *result) override {
|
||||
std::lock_guard<std::mutex> lock(g_engine_mu);
|
||||
|
||||
if (g_engine) {
|
||||
if (g_session) { ds4_session_free(g_session); g_session = nullptr; }
|
||||
ds4_engine_close(g_engine);
|
||||
g_engine = nullptr;
|
||||
}
|
||||
|
||||
std::string model_path = request->modelfile();
|
||||
if (model_path.empty()) model_path = request->model();
|
||||
if (model_path.empty()) {
|
||||
result->set_success(false);
|
||||
result->set_message("ds4: ModelOptions.Model or .ModelFile must be set");
|
||||
return GStatus::OK;
|
||||
}
|
||||
|
||||
std::string mtp_path;
|
||||
int mtp_draft = 0;
|
||||
float mtp_margin = 3.0f;
|
||||
for (const auto &opt : request->options()) {
|
||||
auto [k, v] = split_option(opt);
|
||||
if (k == "mtp_path") mtp_path = v;
|
||||
else if (k == "mtp_draft") mtp_draft = std::stoi(v);
|
||||
else if (k == "mtp_margin") mtp_margin = std::stof(v);
|
||||
else if (k == "kv_cache_dir") g_kv_cache_dir = v;
|
||||
}
|
||||
|
||||
g_kv_cache.SetDir(g_kv_cache_dir);
|
||||
|
||||
ds4_engine_options opt = {};
|
||||
opt.model_path = model_path.c_str();
|
||||
opt.mtp_path = mtp_path.empty() ? nullptr : mtp_path.c_str();
|
||||
opt.n_threads = request->threads() > 0 ? request->threads() : 0;
|
||||
opt.mtp_draft_tokens = mtp_draft;
|
||||
opt.mtp_margin = mtp_margin;
|
||||
opt.directional_steering_file = nullptr;
|
||||
opt.warm_weights = false;
|
||||
opt.quality = false;
|
||||
|
||||
#if defined(DS4_NO_GPU)
|
||||
opt.backend = DS4_BACKEND_CPU;
|
||||
#elif defined(__APPLE__)
|
||||
opt.backend = DS4_BACKEND_METAL;
|
||||
#else
|
||||
opt.backend = DS4_BACKEND_CUDA;
|
||||
#endif
|
||||
|
||||
int rc = ds4_engine_open(&g_engine, &opt);
|
||||
if (rc != 0 || !g_engine) {
|
||||
result->set_success(false);
|
||||
result->set_message("ds4_engine_open failed (rc=" + std::to_string(rc) + ")");
|
||||
return GStatus::OK;
|
||||
}
|
||||
|
||||
g_ctx_size = request->contextsize() > 0 ? request->contextsize() : 32768;
|
||||
rc = ds4_session_create(&g_session, g_engine, g_ctx_size);
|
||||
if (rc != 0 || !g_session) {
|
||||
ds4_engine_close(g_engine);
|
||||
g_engine = nullptr;
|
||||
result->set_success(false);
|
||||
result->set_message("ds4_session_create failed (rc=" + std::to_string(rc) + ")");
|
||||
return GStatus::OK;
|
||||
}
|
||||
|
||||
result->set_success(true);
|
||||
result->set_message("loaded " + model_path);
|
||||
return GStatus::OK;
|
||||
}
|
||||
|
||||
GStatus TokenizeString(ServerContext *, const backend::PredictOptions *request,
|
||||
backend::TokenizationResponse *response) override {
|
||||
std::lock_guard<std::mutex> lock(g_engine_mu);
|
||||
if (!g_engine) return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
|
||||
ds4_tokens out = {};
|
||||
ds4_tokenize_text(g_engine, request->prompt().c_str(), &out);
|
||||
for (int i = 0; i < out.len; ++i) response->add_tokens(out.v[i]);
|
||||
response->set_length(out.len);
|
||||
ds4_tokens_free(&out);
|
||||
return GStatus::OK;
|
||||
}
|
||||
|
||||
GStatus Predict(ServerContext *, const backend::PredictOptions *request,
|
||||
backend::Reply *reply) override {
|
||||
std::lock_guard<std::mutex> lock(g_engine_mu);
|
||||
if (!g_engine || !g_session) {
|
||||
return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
|
||||
}
|
||||
ds4_tokens prompt = {};
|
||||
build_prompt(g_engine, request, &prompt);
|
||||
int n_predict = request->tokens() > 0 ? request->tokens() : 256;
|
||||
|
||||
CollectCtx collect = {g_engine, "", {}, reply, 0, {}, "", ""};
|
||||
std::string cache_key = render_prompt_text(request);
|
||||
size_t cache_hit = maybe_load_cache(cache_key);
|
||||
(void)cache_hit; // future: skip prompt prefix if hit covers full prompt
|
||||
|
||||
// Manual generation loop on g_session. When MTP speculative weights
|
||||
// were loaded (LoadModel option 'mtp_path:'), we use the
|
||||
// ds4_session_eval_speculative_argmax path which may accept N>1
|
||||
// tokens per outer iteration. Otherwise per-token argmax + eval.
|
||||
// Either way g_session advances so the disk KV cache picks up a
|
||||
// real checkpoint after the call (see maybe_save_cache below).
|
||||
char err[256] = {0};
|
||||
int rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
|
||||
int prompt_len = prompt.len;
|
||||
ds4_tokens_free(&prompt);
|
||||
if (rc == 0) {
|
||||
const int eos = ds4_token_eos(g_engine);
|
||||
const int draft_max = ds4_engine_mtp_draft_tokens(g_engine);
|
||||
const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
|
||||
int produced = 0;
|
||||
while (produced < n_predict) {
|
||||
SampleParams sp = compute_sample_params(request, collect.parser, think_enabled);
|
||||
int first;
|
||||
if (sp.temperature <= 0.0f) {
|
||||
first = ds4_session_argmax(g_session);
|
||||
} else {
|
||||
first = ds4_session_sample(g_session,
|
||||
sp.temperature, sp.top_k,
|
||||
sp.top_p, sp.min_p, get_rng());
|
||||
}
|
||||
if (first == eos) break;
|
||||
// MTP only when sampling is greedy (ds4-server gate).
|
||||
if (draft_max > 0 && sp.temperature <= 0.0f) {
|
||||
constexpr int kAcceptedMax = 8;
|
||||
int accepted[kAcceptedMax];
|
||||
int cap = std::min(kAcceptedMax, draft_max + 1);
|
||||
int n = ds4_session_eval_speculative_argmax(
|
||||
g_session, first, draft_max, eos,
|
||||
accepted, cap, err, sizeof(err));
|
||||
if (n < 0) { rc = -1; break; }
|
||||
bool stop = false;
|
||||
for (int j = 0; j < n; ++j) {
|
||||
if (accepted[j] == eos) { stop = true; break; }
|
||||
collect_emit(&collect, accepted[j]);
|
||||
if (++produced >= n_predict) { stop = true; break; }
|
||||
}
|
||||
if (stop) break;
|
||||
} else {
|
||||
collect_emit(&collect, first);
|
||||
if (++produced >= n_predict) break;
|
||||
rc = ds4_session_eval(g_session, first, err, sizeof(err));
|
||||
if (rc != 0) break;
|
||||
}
|
||||
}
|
||||
collect_done(&collect);
|
||||
}
|
||||
maybe_save_cache(cache_key);
|
||||
|
||||
// Flush any buffered parser state.
|
||||
std::vector<ds4cpp::ParserEvent> events;
|
||||
collect.parser.Flush(events);
|
||||
apply_events(&collect, events);
|
||||
|
||||
if (rc != 0) {
|
||||
return GStatus(StatusCode::INTERNAL,
|
||||
std::string("ds4 generation failed: ") + err);
|
||||
}
|
||||
|
||||
// Emit one ChatDelta with content/reasoning/tool_calls.
|
||||
auto *delta = reply->add_chat_deltas();
|
||||
delta->set_content(collect.content_buf);
|
||||
delta->set_reasoning_content(collect.reasoning_buf);
|
||||
for (size_t i = 0; i < collect.pending.size(); ++i) {
|
||||
auto *tc = delta->add_tool_calls();
|
||||
tc->set_index(static_cast<int32_t>(i));
|
||||
tc->set_id(collect.pending[i].id);
|
||||
tc->set_name(collect.pending[i].name);
|
||||
tc->set_arguments(collect.pending[i].args);
|
||||
}
|
||||
|
||||
reply->set_message(collect.raw_buf);
|
||||
reply->set_tokens(collect.tokens);
|
||||
reply->set_prompt_tokens(prompt_len);
|
||||
return GStatus::OK;
|
||||
}
|
||||
|
||||
GStatus PredictStream(ServerContext *, const backend::PredictOptions *request,
|
||||
ServerWriter<backend::Reply> *writer) override {
|
||||
std::lock_guard<std::mutex> lock(g_engine_mu);
|
||||
if (!g_engine || !g_session) {
|
||||
return GStatus(StatusCode::FAILED_PRECONDITION, "ds4: model not loaded");
|
||||
}
|
||||
ds4_tokens prompt = {};
|
||||
build_prompt(g_engine, request, &prompt);
|
||||
int n_predict = request->tokens() > 0 ? request->tokens() : 256;
|
||||
|
||||
StreamCtx s = {g_engine, writer, {}, 0, false, {}};
|
||||
std::string cache_key = render_prompt_text(request);
|
||||
size_t cache_hit = maybe_load_cache(cache_key);
|
||||
(void)cache_hit;
|
||||
|
||||
// Manual loop on g_session - see Predict() above for the rationale.
|
||||
// MTP speculative path used when ds4_engine_mtp_draft_tokens > 0.
|
||||
char err[256] = {0};
|
||||
int rc = ds4_session_sync(g_session, &prompt, err, sizeof(err));
|
||||
ds4_tokens_free(&prompt);
|
||||
if (rc == 0) {
|
||||
const int eos = ds4_token_eos(g_engine);
|
||||
const int draft_max = ds4_engine_mtp_draft_tokens(g_engine);
|
||||
const bool think_enabled = ds4_think_mode_enabled(parse_think_mode(request));
|
||||
int produced = 0;
|
||||
while (produced < n_predict && !s.aborted) {
|
||||
SampleParams sp = compute_sample_params(request, s.parser, think_enabled);
|
||||
int first;
|
||||
if (sp.temperature <= 0.0f) {
|
||||
first = ds4_session_argmax(g_session);
|
||||
} else {
|
||||
first = ds4_session_sample(g_session,
|
||||
sp.temperature, sp.top_k,
|
||||
sp.top_p, sp.min_p, get_rng());
|
||||
}
|
||||
if (first == eos) break;
|
||||
if (draft_max > 0 && sp.temperature <= 0.0f) {
|
||||
constexpr int kAcceptedMax = 8;
|
||||
int accepted[kAcceptedMax];
|
||||
int cap = std::min(kAcceptedMax, draft_max + 1);
|
||||
int n = ds4_session_eval_speculative_argmax(
|
||||
g_session, first, draft_max, eos,
|
||||
accepted, cap, err, sizeof(err));
|
||||
if (n < 0) { rc = -1; break; }
|
||||
bool stop = false;
|
||||
for (int j = 0; j < n; ++j) {
|
||||
if (accepted[j] == eos) { stop = true; break; }
|
||||
stream_emit(&s, accepted[j]);
|
||||
if (s.aborted) { stop = true; break; }
|
||||
if (++produced >= n_predict) { stop = true; break; }
|
||||
}
|
||||
if (stop) break;
|
||||
} else {
|
||||
stream_emit(&s, first);
|
||||
if (s.aborted || ++produced >= n_predict) break;
|
||||
rc = ds4_session_eval(g_session, first, err, sizeof(err));
|
||||
if (rc != 0) break;
|
||||
}
|
||||
}
|
||||
stream_done(&s);
|
||||
}
|
||||
maybe_save_cache(cache_key);
|
||||
|
||||
// Flush parser state.
|
||||
std::vector<ds4cpp::ParserEvent> events;
|
||||
s.parser.Flush(events);
|
||||
if (!events.empty() && !s.aborted) {
|
||||
backend::Reply reply;
|
||||
auto *delta = reply.add_chat_deltas();
|
||||
for (const auto &e : events) {
|
||||
if (e.type == ds4cpp::ParserEvent::CONTENT) {
|
||||
delta->set_content(delta->content() + e.text);
|
||||
} else if (e.type == ds4cpp::ParserEvent::REASONING) {
|
||||
delta->set_reasoning_content(delta->reasoning_content() + e.text);
|
||||
}
|
||||
}
|
||||
s.writer->Write(reply);
|
||||
}
|
||||
|
||||
if (rc != 0 && !s.aborted) {
|
||||
return GStatus(StatusCode::INTERNAL,
|
||||
std::string("ds4 generation failed: ") + err);
|
||||
}
|
||||
return GStatus::OK;
|
||||
}
|
||||
|
||||
GStatus Status(ServerContext *, const backend::HealthMessage *,
|
||||
backend::StatusResponse *response) override {
|
||||
std::lock_guard<std::mutex> lock(g_engine_mu);
|
||||
response->set_state(g_engine ? backend::StatusResponse::READY
|
||||
: backend::StatusResponse::UNINITIALIZED);
|
||||
return GStatus::OK;
|
||||
}
|
||||
};
|
||||
|
||||
void RunServer(const std::string &addr) {
|
||||
DS4Backend service;
|
||||
grpc::EnableDefaultHealthCheckService(true);
|
||||
grpc::reflection::InitProtoReflectionServerBuilderPlugin();
|
||||
|
||||
ServerBuilder builder;
|
||||
builder.AddListeningPort(addr, grpc::InsecureServerCredentials());
|
||||
builder.RegisterService(&service);
|
||||
builder.SetMaxReceiveMessageSize(64 * 1024 * 1024);
|
||||
builder.SetMaxSendMessageSize(64 * 1024 * 1024);
|
||||
|
||||
std::unique_ptr<Server> server(builder.BuildAndStart());
|
||||
if (!server) {
|
||||
std::cerr << "ds4 grpc-server: failed to bind " << addr << "\n";
|
||||
std::exit(1);
|
||||
}
|
||||
g_server = server.get();
|
||||
std::cerr << "ds4 grpc-server listening on " << addr << "\n";
|
||||
server->Wait();
|
||||
}
|
||||
|
||||
void signal_handler(int) {
|
||||
if (auto *srv = g_server.load()) {
|
||||
srv->Shutdown(std::chrono::system_clock::now() +
|
||||
std::chrono::seconds(3));
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
int main(int argc, char *argv[]) {
|
||||
std::string addr = "127.0.0.1:50051";
|
||||
for (int i = 1; i < argc; ++i) {
|
||||
std::string a = argv[i];
|
||||
const std::string addr_flag = "--addr=";
|
||||
if (a.rfind(addr_flag, 0) == 0) addr = a.substr(addr_flag.size());
|
||||
else if (a == "--addr" && i + 1 < argc) addr = argv[++i];
|
||||
else if (a == "--help" || a == "-h") {
|
||||
std::cout << "Usage: grpc-server --addr=HOST:PORT\n";
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
std::signal(SIGINT, signal_handler);
|
||||
std::signal(SIGTERM, signal_handler);
|
||||
RunServer(addr);
|
||||
return 0;
|
||||
}
|
||||
@@ -1,205 +0,0 @@
|
||||
#include "kv_cache.h"
|
||||
|
||||
#include <cerrno>
|
||||
#include <cstdio>
|
||||
#include <cstring>
|
||||
#include <dirent.h>
|
||||
#include <fstream>
|
||||
#include <sys/stat.h>
|
||||
#include <vector>
|
||||
|
||||
namespace ds4cpp {
|
||||
|
||||
namespace {
|
||||
|
||||
// Minimal SHA1 (public domain reference). 30 lines; used only here.
|
||||
struct Sha1 {
|
||||
uint32_t h[5];
|
||||
uint64_t bits;
|
||||
uint8_t block[64];
|
||||
size_t used;
|
||||
Sha1() { h[0]=0x67452301; h[1]=0xEFCDAB89; h[2]=0x98BADCFE; h[3]=0x10325476; h[4]=0xC3D2E1F0; bits=0; used=0; }
|
||||
static uint32_t rol(uint32_t x, int n){ return (x<<n)|(x>>(32-n)); }
|
||||
void transform(const uint8_t *b) {
|
||||
uint32_t w[80];
|
||||
for (int i=0;i<16;i++) w[i] = (uint32_t)b[i*4]<<24 | (uint32_t)b[i*4+1]<<16 | (uint32_t)b[i*4+2]<<8 | b[i*4+3];
|
||||
for (int i=16;i<80;i++) w[i] = rol(w[i-3]^w[i-8]^w[i-14]^w[i-16], 1);
|
||||
uint32_t a=h[0],bb=h[1],c=h[2],d=h[3],e=h[4];
|
||||
for (int i=0;i<80;i++) {
|
||||
uint32_t f,k;
|
||||
if (i<20) { f=(bb&c)|((~bb)&d); k=0x5A827999; }
|
||||
else if (i<40) { f=bb^c^d; k=0x6ED9EBA1; }
|
||||
else if (i<60) { f=(bb&c)|(bb&d)|(c&d); k=0x8F1BBCDC; }
|
||||
else { f=bb^c^d; k=0xCA62C1D6; }
|
||||
uint32_t t = rol(a,5)+f+e+k+w[i];
|
||||
e=d; d=c; c=rol(bb,30); bb=a; a=t;
|
||||
}
|
||||
h[0]+=a; h[1]+=bb; h[2]+=c; h[3]+=d; h[4]+=e;
|
||||
}
|
||||
void update(const void *p, size_t n) {
|
||||
const uint8_t *bp = (const uint8_t*)p;
|
||||
bits += (uint64_t)n*8;
|
||||
while (n) {
|
||||
size_t take = 64-used;
|
||||
if (take>n) take=n;
|
||||
std::memcpy(block+used, bp, take);
|
||||
used += take; bp += take; n -= take;
|
||||
if (used == 64) { transform(block); used = 0; }
|
||||
}
|
||||
}
|
||||
void final(uint8_t out[20]) {
|
||||
uint8_t pad[64] = {0x80};
|
||||
size_t padlen = (used < 56) ? (56-used) : (120-used);
|
||||
uint64_t lb = bits;
|
||||
uint8_t len[8];
|
||||
for (int i=0;i<8;i++) len[7-i] = (uint8_t)(lb >> (i*8));
|
||||
update(pad, padlen);
|
||||
update(len, 8);
|
||||
for (int i=0;i<5;i++) {
|
||||
out[i*4] = h[i]>>24;
|
||||
out[i*4+1] = h[i]>>16;
|
||||
out[i*4+2] = h[i]>>8;
|
||||
out[i*4+3] = h[i];
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
std::string mkdir_p(const std::string &d) {
|
||||
if (d.empty()) return d;
|
||||
struct stat st{};
|
||||
if (stat(d.c_str(), &st) == 0) return d;
|
||||
mkdir(d.c_str(), 0755);
|
||||
return d;
|
||||
}
|
||||
|
||||
bool file_exists(const std::string &p) {
|
||||
struct stat st{};
|
||||
return stat(p.c_str(), &st) == 0;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
std::string Sha1Hex(const void *data, size_t len) {
|
||||
Sha1 s;
|
||||
s.update(data, len);
|
||||
uint8_t out[20];
|
||||
s.final(out);
|
||||
char hex[41];
|
||||
for (int i = 0; i < 20; ++i) std::snprintf(hex + i*2, 3, "%02x", out[i]);
|
||||
hex[40] = 0;
|
||||
return std::string(hex);
|
||||
}
|
||||
|
||||
KvCache::KvCache() = default;
|
||||
|
||||
void KvCache::SetDir(const std::string &dir) {
|
||||
dir_ = dir;
|
||||
if (!dir_.empty()) {
|
||||
mkdir_p(dir_);
|
||||
std::fprintf(stderr, "ds4 KvCache: enabled at %s\n", dir_.c_str());
|
||||
} else {
|
||||
std::fprintf(stderr, "ds4 KvCache: disabled (no dir set)\n");
|
||||
}
|
||||
}
|
||||
|
||||
std::string KvCache::Path(const std::string &rendered_text) const {
|
||||
if (dir_.empty()) return "";
|
||||
return dir_ + "/" + Sha1Hex(rendered_text.data(), rendered_text.size()) + ".kv";
|
||||
}
|
||||
|
||||
size_t KvCache::LoadLongestPrefix(ds4_session *session,
|
||||
const std::string &rendered_text,
|
||||
int ctx_size) {
|
||||
if (dir_.empty() || !session) return 0;
|
||||
// Strategy: enumerate all .kv files in dir, read their stored prefix
|
||||
// header, pick the longest one that is also a prefix of rendered_text.
|
||||
DIR *d = opendir(dir_.c_str());
|
||||
if (!d) return 0;
|
||||
struct dirent *de;
|
||||
size_t best_len = 0;
|
||||
std::string best_path;
|
||||
while ((de = readdir(d)) != nullptr) {
|
||||
std::string name = de->d_name;
|
||||
if (name.size() < 4 || name.substr(name.size()-3) != ".kv") continue;
|
||||
std::string path = dir_ + "/" + name;
|
||||
std::ifstream f(path, std::ios::binary);
|
||||
if (!f) continue;
|
||||
char magic[4]; f.read(magic, 4);
|
||||
if (f.gcount() != 4 || std::memcmp(magic, "DS4G", 4) != 0) continue;
|
||||
uint32_t version=0, file_ctx=0, prefix_len=0;
|
||||
f.read((char*)&version, 4); f.read((char*)&file_ctx, 4); f.read((char*)&prefix_len, 4);
|
||||
if (version != 1) continue;
|
||||
if ((int)file_ctx != ctx_size) continue;
|
||||
if (prefix_len > rendered_text.size()) continue;
|
||||
std::vector<char> prefix(prefix_len);
|
||||
f.read(prefix.data(), prefix_len);
|
||||
if (std::memcmp(prefix.data(), rendered_text.data(), prefix_len) != 0) continue;
|
||||
if (prefix_len > best_len) {
|
||||
best_len = prefix_len;
|
||||
best_path = path;
|
||||
}
|
||||
}
|
||||
closedir(d);
|
||||
if (best_len == 0) return 0;
|
||||
|
||||
// Load best_path's payload into session.
|
||||
std::ifstream f(best_path, std::ios::binary);
|
||||
char magic[4]; f.read(magic, 4);
|
||||
uint32_t version, file_ctx, prefix_len;
|
||||
f.read((char*)&version, 4); f.read((char*)&file_ctx, 4); f.read((char*)&prefix_len, 4);
|
||||
f.seekg(prefix_len, std::ios::cur);
|
||||
uint64_t payload_bytes = 0;
|
||||
f.read((char*)&payload_bytes, 8);
|
||||
// ds4_session_load_payload reads from a FILE*; reopen via fopen.
|
||||
FILE *fp = std::fopen(best_path.c_str(), "rb");
|
||||
if (!fp) return 0;
|
||||
// Seek past header + prefix + payload_bytes field.
|
||||
std::fseek(fp, 4 + 4 + 4 + 4 + prefix_len + 8, SEEK_SET);
|
||||
char errbuf[256] = {0};
|
||||
int rc = ds4_session_load_payload(session, fp, payload_bytes, errbuf, sizeof(errbuf));
|
||||
std::fclose(fp);
|
||||
if (rc != 0) return 0;
|
||||
return best_len;
|
||||
}
|
||||
|
||||
void KvCache::Save(ds4_session *session, const std::string &rendered_text, int ctx_size) {
|
||||
if (dir_.empty()) {
|
||||
std::fprintf(stderr, "ds4 KvCache::Save: skipped (dir empty)\n");
|
||||
return;
|
||||
}
|
||||
if (!session) {
|
||||
std::fprintf(stderr, "ds4 KvCache::Save: skipped (session null)\n");
|
||||
return;
|
||||
}
|
||||
std::string path = Path(rendered_text);
|
||||
uint64_t payload_bytes = ds4_session_payload_bytes(session);
|
||||
std::fprintf(stderr, "ds4 KvCache::Save: path=%s payload_bytes=%llu prefix_len=%zu\n",
|
||||
path.c_str(), (unsigned long long)payload_bytes, rendered_text.size());
|
||||
FILE *fp = std::fopen(path.c_str(), "wb");
|
||||
if (!fp) {
|
||||
std::fprintf(stderr, "ds4 KvCache::Save: fopen failed: %s\n", std::strerror(errno));
|
||||
return;
|
||||
}
|
||||
char magic[4] = {'D','S','4','G'};
|
||||
uint32_t version = 1;
|
||||
uint32_t ctx = static_cast<uint32_t>(ctx_size);
|
||||
uint32_t prefix_len = static_cast<uint32_t>(rendered_text.size());
|
||||
std::fwrite(magic, 4, 1, fp);
|
||||
std::fwrite(&version, 4, 1, fp);
|
||||
std::fwrite(&ctx, 4, 1, fp);
|
||||
std::fwrite(&prefix_len, 4, 1, fp);
|
||||
std::fwrite(rendered_text.data(), prefix_len, 1, fp);
|
||||
std::fwrite(&payload_bytes, 8, 1, fp);
|
||||
char errbuf[256] = {0};
|
||||
int rc = ds4_session_save_payload(session, fp, errbuf, sizeof(errbuf));
|
||||
std::fclose(fp);
|
||||
if (rc != 0) {
|
||||
std::fprintf(stderr, "ds4 KvCache::Save: ds4_session_save_payload rc=%d err=%s; removing %s\n",
|
||||
rc, errbuf, path.c_str());
|
||||
std::remove(path.c_str());
|
||||
} else {
|
||||
std::fprintf(stderr, "ds4 KvCache::Save: wrote %s ok\n", path.c_str());
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace ds4cpp
|
||||
@@ -1,44 +0,0 @@
|
||||
#pragma once
|
||||
#include <string>
|
||||
extern "C" {
|
||||
#include "ds4.h"
|
||||
}
|
||||
|
||||
namespace ds4cpp {
|
||||
|
||||
// Disk-backed KV cache for ds4 sessions. Keyed by SHA1(rendered prompt prefix).
|
||||
// Format (our own, NOT bit-compatible with ds4-server's KVC files - interop
|
||||
// is a follow-up plan):
|
||||
//
|
||||
// "DS4G" (4 bytes magic) + u32 version=1 + u32 ctx_size +
|
||||
// u32 prefix_text_len + prefix_text + u64 payload_bytes + payload
|
||||
class KvCache {
|
||||
public:
|
||||
KvCache(); // disabled (dir empty)
|
||||
|
||||
// Set the cache directory. Empty disables.
|
||||
void SetDir(const std::string &dir);
|
||||
|
||||
// Returns the cache file path for a given rendered text prefix.
|
||||
std::string Path(const std::string &rendered_text) const;
|
||||
|
||||
// Look up the longest cached prefix that is also a prefix of
|
||||
// `rendered_text`. Loads it into `session` if found. Returns the
|
||||
// matched prefix length in bytes (0 if no hit).
|
||||
size_t LoadLongestPrefix(ds4_session *session,
|
||||
const std::string &rendered_text,
|
||||
int ctx_size);
|
||||
|
||||
// Save the current session, associated with this rendered text prefix.
|
||||
void Save(ds4_session *session, const std::string &rendered_text, int ctx_size);
|
||||
|
||||
bool enabled() const { return !dir_.empty(); }
|
||||
|
||||
private:
|
||||
std::string dir_;
|
||||
};
|
||||
|
||||
// Compute SHA1 of arbitrary bytes; returns 40-char hex.
|
||||
std::string Sha1Hex(const void *data, size_t len);
|
||||
|
||||
} // namespace ds4cpp
|
||||
@@ -1,39 +0,0 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
CURDIR=$(dirname "$(realpath "$0")")
|
||||
REPO_ROOT="${CURDIR}/../../.."
|
||||
|
||||
mkdir -p "$CURDIR/package/lib"
|
||||
cp -avf "$CURDIR/grpc-server" "$CURDIR/package/"
|
||||
cp -rfv "$CURDIR/run.sh" "$CURDIR/package/"
|
||||
|
||||
UNAME_S=$(uname -s)
|
||||
if [ "$UNAME_S" = "Darwin" ]; then
|
||||
# Darwin: bundle dylibs via otool -L (handled by scripts/build/ds4-darwin.sh).
|
||||
echo "package.sh: Darwin handled by ds4-darwin.sh"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
|
||||
cp -arfLv /lib64/ld-linux-x86-64.so.2 "$CURDIR/package/lib/ld.so"
|
||||
LIBDIR=/lib/x86_64-linux-gnu
|
||||
elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
|
||||
cp -arfLv /lib/ld-linux-aarch64.so.1 "$CURDIR/package/lib/ld.so"
|
||||
LIBDIR=/lib/aarch64-linux-gnu
|
||||
else
|
||||
echo "package.sh: unknown architecture" >&2; exit 1
|
||||
fi
|
||||
|
||||
for lib in libc.so.6 libgcc_s.so.1 libstdc++.so.6 libm.so.6 libgomp.so.1 \
|
||||
libdl.so.2 librt.so.1 libpthread.so.0; do
|
||||
cp -arfLv "$LIBDIR/$lib" "$CURDIR/package/lib/$lib"
|
||||
done
|
||||
|
||||
GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
|
||||
if [ -f "$GPU_LIB_SCRIPT" ]; then
|
||||
source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
|
||||
package_gpu_libs
|
||||
fi
|
||||
|
||||
echo "ds4 package contents:"
|
||||
ls -lah "$CURDIR/package/" "$CURDIR/package/lib/"
|
||||
@@ -1,9 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Entry point for the ds4 backend image / BACKEND_BINARY mode.
|
||||
set -e
|
||||
CURDIR=$(dirname "$(realpath "$0")")
|
||||
export LD_LIBRARY_PATH="$CURDIR/lib:$LD_LIBRARY_PATH"
|
||||
if [ -f "$CURDIR/lib/ld.so" ]; then
|
||||
exec "$CURDIR/lib/ld.so" "$CURDIR/grpc-server" "$@"
|
||||
fi
|
||||
exec "$CURDIR/grpc-server" "$@"
|
||||
@@ -1,5 +1,5 @@
|
||||
|
||||
IK_LLAMA_VERSION?=eb570eb96689c235933b813693ca28ab9d3d26de
|
||||
IK_LLAMA_VERSION?=8b56d813a9ed04fa7b7fe2588fddd845cf64eccb
|
||||
LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp
|
||||
|
||||
CMAKE_ARGS?=
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
|
||||
LLAMA_VERSION?=389ff61d77b5c71cec0cf92fe4e5d01ace80b797
|
||||
LLAMA_VERSION?=bbeb89d76c41bc250f16e4a6fefcc9b530d6e3f3
|
||||
LLAMA_REPO?=https://github.com/ggerganov/llama.cpp
|
||||
|
||||
CMAKE_ARGS?=
|
||||
|
||||
@@ -2,7 +2,6 @@ package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/binary"
|
||||
"fmt"
|
||||
"os"
|
||||
@@ -999,7 +998,7 @@ func (s *SherpaBackend) loadOnlineASR(opts *pb.ModelOptions) error {
|
||||
// Transcription
|
||||
// =============================================================
|
||||
|
||||
func (s *SherpaBackend) AudioTranscription(_ context.Context, req *pb.TranscriptRequest) (pb.TranscriptResult, error) {
|
||||
func (s *SherpaBackend) AudioTranscription(req *pb.TranscriptRequest) (pb.TranscriptResult, error) {
|
||||
if s.onlineRecognizer != 0 {
|
||||
return s.runOnlineASR(req, nil)
|
||||
}
|
||||
@@ -1057,7 +1056,6 @@ func (s *SherpaBackend) AudioTranscription(_ context.Context, req *pb.Transcript
|
||||
// Closes `results` before returning so the server wrapper's reader
|
||||
// goroutine can exit.
|
||||
func (s *SherpaBackend) AudioTranscriptionStream(
|
||||
_ context.Context,
|
||||
req *pb.TranscriptRequest,
|
||||
results chan *pb.TranscriptStreamResponse,
|
||||
) error {
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
@@ -80,7 +79,7 @@ var _ = Describe("Sherpa-ONNX", func() {
|
||||
})
|
||||
|
||||
It("rejects AudioTranscription", func() {
|
||||
_, err := (&SherpaBackend{}).AudioTranscription(context.Background(), &pb.TranscriptRequest{
|
||||
_, err := (&SherpaBackend{}).AudioTranscription(&pb.TranscriptRequest{
|
||||
Dst: "/tmp/nonexistent.wav",
|
||||
})
|
||||
Expect(err).To(HaveOccurred())
|
||||
|
||||
@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
|
||||
|
||||
# stablediffusion.cpp (ggml)
|
||||
STABLEDIFFUSION_GGML_REPO?=https://github.com/leejet/stable-diffusion.cpp
|
||||
STABLEDIFFUSION_GGML_VERSION?=90e87bc846f17059771efb8aaa31e9ef0cab6f78
|
||||
STABLEDIFFUSION_GGML_VERSION?=3d6064b37ef4607917f8acf2ca8c8906d5087413
|
||||
|
||||
CMAKE_ARGS+=-DGGML_MAX_NAME=128
|
||||
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
@@ -481,7 +480,7 @@ func (w *byteWriter) Write(p []byte) (int, error) {
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
func (v *VibevoiceCpp) AudioTranscription(_ context.Context, req *pb.TranscriptRequest) (pb.TranscriptResult, error) {
|
||||
func (v *VibevoiceCpp) AudioTranscription(req *pb.TranscriptRequest) (pb.TranscriptResult, error) {
|
||||
if v.asrModel == "" {
|
||||
return pb.TranscriptResult{}, fmt.Errorf("vibevoice-cpp: AudioTranscription requested but no ASR model was loaded")
|
||||
}
|
||||
@@ -624,9 +623,9 @@ func (v *VibevoiceCpp) Diarize(req *pb.DiarizeRequest) (pb.DiarizeResponse, erro
|
||||
// transcription, emit each segment's content as a delta, then close
|
||||
// with a final_result whose Text equals the concatenated deltas (the
|
||||
// e2e harness asserts those match).
|
||||
func (v *VibevoiceCpp) AudioTranscriptionStream(ctx context.Context, req *pb.TranscriptRequest, results chan *pb.TranscriptStreamResponse) error {
|
||||
func (v *VibevoiceCpp) AudioTranscriptionStream(req *pb.TranscriptRequest, results chan *pb.TranscriptStreamResponse) error {
|
||||
defer close(results)
|
||||
res, err := v.AudioTranscription(ctx, req)
|
||||
res, err := v.AudioTranscription(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -107,7 +107,7 @@ var _ = Describe("VibeVoice-cpp", func() {
|
||||
})
|
||||
|
||||
It("rejects AudioTranscription without a loaded ASR model", func() {
|
||||
_, err := (&VibevoiceCpp{}).AudioTranscription(context.Background(), &pb.TranscriptRequest{
|
||||
_, err := (&VibevoiceCpp{}).AudioTranscription(&pb.TranscriptRequest{
|
||||
Dst: "/tmp/some.wav",
|
||||
})
|
||||
Expect(err).To(HaveOccurred())
|
||||
@@ -255,7 +255,7 @@ var _ = Describe("VibeVoice-cpp", func() {
|
||||
|
||||
It("closes the channel and errors on AudioTranscriptionStream without a loaded model", func() {
|
||||
ch := make(chan *pb.TranscriptStreamResponse, 4)
|
||||
err := (&VibevoiceCpp{}).AudioTranscriptionStream(context.Background(), &pb.TranscriptRequest{
|
||||
err := (&VibevoiceCpp{}).AudioTranscriptionStream(&pb.TranscriptRequest{
|
||||
Dst: "/tmp/some.wav",
|
||||
}, ch)
|
||||
Expect(err).To(HaveOccurred())
|
||||
|
||||
@@ -1,7 +1,6 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
@@ -28,7 +27,7 @@ func (v *Voxtral) Load(opts *pb.ModelOptions) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func (v *Voxtral) AudioTranscription(_ context.Context, opts *pb.TranscriptRequest) (pb.TranscriptResult, error) {
|
||||
func (v *Voxtral) AudioTranscription(opts *pb.TranscriptRequest) (pb.TranscriptResult, error) {
|
||||
dir, err := os.MkdirTemp("", "voxtral")
|
||||
if err != nil {
|
||||
return pb.TranscriptResult{}, err
|
||||
|
||||
@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
|
||||
|
||||
# whisper.cpp version
|
||||
WHISPER_REPO?=https://github.com/ggml-org/whisper.cpp
|
||||
WHISPER_CPP_VERSION?=c33c5618b72bb345df029b730b36bc0e369845a3
|
||||
WHISPER_CPP_VERSION?=4bf733672b2871d4153158af4f621a6dd9104f4a
|
||||
SO_TARGET?=libgowhisper.so
|
||||
|
||||
CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF
|
||||
|
||||
@@ -1,47 +1,12 @@
|
||||
#include "gowhisper.h"
|
||||
#include "ggml-backend.h"
|
||||
#include "whisper.h"
|
||||
#include <atomic>
|
||||
#include <vector>
|
||||
|
||||
static struct whisper_vad_context *vctx;
|
||||
static struct whisper_context *ctx;
|
||||
static std::vector<float> flat_segs;
|
||||
|
||||
static std::atomic<int> g_abort{0};
|
||||
|
||||
static std::atomic<uintptr_t> g_go_new_segment_cb{0};
|
||||
static std::atomic<uintptr_t> g_go_new_segment_user_data{0};
|
||||
|
||||
static bool abort_cb(void * /*user_data*/) {
|
||||
return g_abort.load(std::memory_order_relaxed) != 0;
|
||||
}
|
||||
|
||||
static void new_segment_cb(struct whisper_context *cb_ctx,
|
||||
struct whisper_state * /*state*/, int n_new,
|
||||
void * /*user_data*/) {
|
||||
uintptr_t go_cb = g_go_new_segment_cb.load(std::memory_order_relaxed);
|
||||
if (go_cb == 0) {
|
||||
return;
|
||||
}
|
||||
int total = whisper_full_n_segments(cb_ctx);
|
||||
int idx_first = total - n_new;
|
||||
if (idx_first < 0) {
|
||||
idx_first = 0;
|
||||
}
|
||||
uintptr_t ud = g_go_new_segment_user_data.load(std::memory_order_relaxed);
|
||||
reinterpret_cast<go_new_segment_cb>(go_cb)(idx_first, n_new, ud);
|
||||
}
|
||||
|
||||
extern "C" void set_abort(int v) {
|
||||
g_abort.store(v, std::memory_order_relaxed);
|
||||
}
|
||||
|
||||
extern "C" void set_new_segment_callback(uintptr_t cb_ptr, uintptr_t user_data) {
|
||||
g_go_new_segment_cb.store(cb_ptr, std::memory_order_relaxed);
|
||||
g_go_new_segment_user_data.store(user_data, std::memory_order_relaxed);
|
||||
}
|
||||
|
||||
static void ggml_log_cb(enum ggml_log_level level, const char *log,
|
||||
void *data) {
|
||||
const char *level_str;
|
||||
@@ -159,28 +124,10 @@ int transcribe(uint32_t threads, char *lang, bool translate, bool tdrz,
|
||||
wparams.tdrz_enable = tdrz;
|
||||
wparams.initial_prompt = prompt;
|
||||
|
||||
// Reset stale abort flag from any prior cancelled call, then install the
|
||||
// ggml abort hook so a subsequent set_abort(1) from Go aborts the next
|
||||
// compute graph step.
|
||||
g_abort.store(0, std::memory_order_relaxed);
|
||||
// Only install the new-segment callback when streaming is requested
|
||||
// (Go side calls set_new_segment_callback before transcribe()). Leaving
|
||||
// it always-on is harmless but adds a function-pointer dispatch per
|
||||
// segment for the offline path.
|
||||
if (g_go_new_segment_cb.load(std::memory_order_relaxed) != 0) {
|
||||
wparams.new_segment_callback = new_segment_cb;
|
||||
wparams.new_segment_callback_user_data = nullptr;
|
||||
}
|
||||
wparams.abort_callback = abort_cb;
|
||||
wparams.abort_callback_user_data = nullptr;
|
||||
|
||||
fprintf(stderr, "info: Enable tdrz: %d\n", tdrz);
|
||||
fprintf(stderr, "info: Initial prompt: \"%s\"\n", prompt);
|
||||
|
||||
if (whisper_full(ctx, wparams, pcmf32, pcmf32_len)) {
|
||||
if (g_abort.load(std::memory_order_relaxed)) {
|
||||
return 2; // aborted by client
|
||||
}
|
||||
fprintf(stderr, "error: transcription failed\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
@@ -15,16 +15,4 @@ int64_t get_segment_t1(int i);
|
||||
int n_tokens(int i);
|
||||
int32_t get_token_id(int i, int j);
|
||||
bool get_segment_speaker_turn_next(int i);
|
||||
void set_abort(int v);
|
||||
|
||||
// Function pointer from Go (returned by purego.NewCallback). Invoked once
|
||||
// per new-segment event during whisper_full(). The callback runs on the
|
||||
// decode thread - if Go blocks (slow gRPC consumer), the decode blocks
|
||||
// too. That is the intended backpressure path.
|
||||
typedef void (*go_new_segment_cb)(int idx_first, int n_new, uintptr_t user_data);
|
||||
|
||||
// Install the callback used by the next transcribe() call. Pass cb=0 to
|
||||
// clear. user_data is opaque to C; the Go side uses it to look up
|
||||
// per-call state.
|
||||
void set_new_segment_callback(uintptr_t cb_ptr, uintptr_t user_data);
|
||||
}
|
||||
|
||||
@@ -1,21 +1,16 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"unsafe"
|
||||
|
||||
"github.com/go-audio/wav"
|
||||
"github.com/mudler/LocalAI/pkg/grpc/base"
|
||||
pb "github.com/mudler/LocalAI/pkg/grpc/proto"
|
||||
"github.com/mudler/LocalAI/pkg/utils"
|
||||
"google.golang.org/grpc/codes"
|
||||
"google.golang.org/grpc/status"
|
||||
)
|
||||
|
||||
var (
|
||||
@@ -29,84 +24,8 @@ var (
|
||||
CppNTokens func(i int) int
|
||||
CppGetTokenID func(i int, j int) int
|
||||
CppGetSegmentSpeakerTurnNext func(i int) bool
|
||||
CppSetAbort func(v int)
|
||||
// Set by main.go via purego.RegisterLibFunc. Installs (or clears with cb=0)
|
||||
// the C-side trampoline that whisper.cpp invokes per new segment.
|
||||
CppSetNewSegmentCallback func(cbPtr uintptr, userData uintptr)
|
||||
)
|
||||
|
||||
// streamCallStates maps per-AudioTranscriptionStream call IDs to the
|
||||
// state the Go callback needs to emit deltas. Only one entry is ever
|
||||
// live today (base.SingleThread), but the map shape mirrors
|
||||
// sherpa-onnx's TTS callback registry and survives a future SingleThread
|
||||
// removal without a contract change.
|
||||
var (
|
||||
streamCallStates sync.Map // uint64 -> *streamCallState
|
||||
streamCallSeq atomic.Uint64
|
||||
goNewSegmentCb uintptr // purego.NewCallback(onNewSegment) result; set in main.go at boot
|
||||
)
|
||||
|
||||
type streamCallState struct {
|
||||
results chan *pb.TranscriptStreamResponse
|
||||
diarize bool
|
||||
// nextIdx tracks how many segments we've already emitted. The C
|
||||
// trampoline passes idx_first = total - n_new, but we walk from
|
||||
// nextIdx to (idx_first + n_new) defensively in case whisper.cpp ever
|
||||
// coalesces multiple commits into a single callback invocation.
|
||||
nextIdx int
|
||||
// assembled mirrors the literal concat of every Delta sent on results.
|
||||
// We reuse it as the final TranscriptResult.Text so the e2e
|
||||
// invariant `final.Text == concat(deltas)` holds exactly. Written from
|
||||
// the cgo decode thread inside onNewSegment and read by the streaming
|
||||
// method after CppTranscribe returns; the cgo boundary provides the
|
||||
// happens-before edge.
|
||||
assembled strings.Builder
|
||||
}
|
||||
|
||||
// onNewSegment is the Go side of the C trampoline declared in
|
||||
// gowhisper.cpp:new_segment_cb. Whisper.cpp invokes it once per
|
||||
// new-segment event during whisper_full(). Reads segment text via the
|
||||
// existing CppGetSegment* getters (safe to call against the singleton
|
||||
// ctx; whisper.cpp is the only writer and it has already published the
|
||||
// segments by the time this fires).
|
||||
//
|
||||
// Sends deltas synchronously: if the channel is full, this blocks the
|
||||
// whisper decode thread. That's the intended backpressure path -
|
||||
// dropping deltas would break the concat(deltas) == final.Text invariant
|
||||
// the e2e suite asserts.
|
||||
func onNewSegment(idxFirst int32, nNew int32, userData uintptr) {
|
||||
v, ok := streamCallStates.Load(uint64(userData))
|
||||
if !ok {
|
||||
return // call already torn down (race with cancel + cb fire)
|
||||
}
|
||||
state := v.(*streamCallState)
|
||||
end := int(idxFirst) + int(nNew)
|
||||
for i := state.nextIdx; i < end; i++ {
|
||||
txt := strings.ToValidUTF8(strings.Clone(CppGetSegmentText(i)), "<22>")
|
||||
txt = strings.TrimSpace(txt)
|
||||
if state.diarize && CppGetSegmentSpeakerTurnNext(i) {
|
||||
txt += " [SPEAKER_TURN]"
|
||||
}
|
||||
if txt == "" {
|
||||
state.nextIdx = i + 1
|
||||
continue
|
||||
}
|
||||
// Prefix subsequent deltas with a single space so the assembled
|
||||
// stream reads as one space-joined transcript. The first delta has
|
||||
// no leading space, otherwise concat(deltas) would not match
|
||||
// final.Text and the e2e invariant would break.
|
||||
var delta string
|
||||
if state.assembled.Len() == 0 {
|
||||
delta = txt
|
||||
} else {
|
||||
delta = " " + txt
|
||||
}
|
||||
state.results <- &pb.TranscriptStreamResponse{Delta: delta}
|
||||
state.assembled.WriteString(delta)
|
||||
state.nextIdx = i + 1
|
||||
}
|
||||
}
|
||||
|
||||
type Whisper struct {
|
||||
base.SingleThread
|
||||
}
|
||||
@@ -173,11 +92,7 @@ func (w *Whisper) VAD(req *pb.VADRequest) (pb.VADResponse, error) {
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (w *Whisper) AudioTranscription(ctx context.Context, opts *pb.TranscriptRequest) (pb.TranscriptResult, error) {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return pb.TranscriptResult{}, status.Error(codes.Canceled, "transcription cancelled")
|
||||
}
|
||||
|
||||
func (w *Whisper) AudioTranscription(opts *pb.TranscriptRequest) (pb.TranscriptResult, error) {
|
||||
dir, err := os.MkdirTemp("", "whisper")
|
||||
if err != nil {
|
||||
return pb.TranscriptResult{}, err
|
||||
@@ -190,12 +105,14 @@ func (w *Whisper) AudioTranscription(ctx context.Context, opts *pb.TranscriptReq
|
||||
return pb.TranscriptResult{}, err
|
||||
}
|
||||
|
||||
// Open samples
|
||||
fh, err := os.Open(convertedPath)
|
||||
if err != nil {
|
||||
return pb.TranscriptResult{}, err
|
||||
}
|
||||
defer fh.Close()
|
||||
|
||||
// Read samples
|
||||
d := wav.NewDecoder(fh)
|
||||
buf, err := d.FullPCMBuffer()
|
||||
if err != nil {
|
||||
@@ -203,6 +120,8 @@ func (w *Whisper) AudioTranscription(ctx context.Context, opts *pb.TranscriptReq
|
||||
}
|
||||
|
||||
data := buf.AsFloat32Buffer().Data
|
||||
// whisper.cpp resamples to 16 kHz internally; this matches buf.Format.SampleRate
|
||||
// for the converted file produced by AudioToWav above.
|
||||
var duration float32
|
||||
if buf.Format != nil && buf.Format.SampleRate > 0 {
|
||||
duration = float32(len(data)) / float32(buf.Format.SampleRate)
|
||||
@@ -210,31 +129,7 @@ func (w *Whisper) AudioTranscription(ctx context.Context, opts *pb.TranscriptReq
|
||||
segsLen := uintptr(0xdeadbeef)
|
||||
segsLenPtr := unsafe.Pointer(&segsLen)
|
||||
|
||||
// Watcher: flips the C-side abort flag when ctx is cancelled. The
|
||||
// goroutine is joined synchronously (close(done) signals it to exit,
|
||||
// wg.Wait() blocks until it has) so a late CppSetAbort(1) cannot fire
|
||||
// after the function returns and corrupt the next transcription call.
|
||||
done := make(chan struct{})
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
CppSetAbort(1)
|
||||
case <-done:
|
||||
}
|
||||
}()
|
||||
defer func() {
|
||||
close(done)
|
||||
wg.Wait()
|
||||
}()
|
||||
|
||||
ret := CppTranscribe(opts.Threads, opts.Language, opts.Translate, opts.Diarize, data, uintptr(len(data)), segsLenPtr, opts.Prompt)
|
||||
if ret == 2 {
|
||||
return pb.TranscriptResult{}, status.Error(codes.Canceled, "transcription cancelled")
|
||||
}
|
||||
if ret != 0 {
|
||||
if ret := CppTranscribe(opts.Threads, opts.Language, opts.Translate, opts.Diarize, data, uintptr(len(data)), segsLenPtr, opts.Prompt); ret != 0 {
|
||||
return pb.TranscriptResult{}, fmt.Errorf("Failed Transcribe")
|
||||
}
|
||||
|
||||
@@ -276,120 +171,3 @@ func (w *Whisper) AudioTranscription(ctx context.Context, opts *pb.TranscriptReq
|
||||
Duration: duration,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// AudioTranscriptionStream runs whisper_full() and emits deltas via
|
||||
// whisper.cpp's new_segment_callback as segments are decoded, then a
|
||||
// final TranscriptResult. The offline AudioTranscription is unchanged;
|
||||
// both paths share whisper's single-instance ctx and the SingleThread
|
||||
// concurrency model.
|
||||
func (w *Whisper) AudioTranscriptionStream(ctx context.Context, opts *pb.TranscriptRequest, results chan *pb.TranscriptStreamResponse) error {
|
||||
defer close(results)
|
||||
|
||||
if err := ctx.Err(); err != nil {
|
||||
return status.Error(codes.Canceled, "transcription cancelled")
|
||||
}
|
||||
|
||||
dir, err := os.MkdirTemp("", "whisper")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() { _ = os.RemoveAll(dir) }()
|
||||
|
||||
convertedPath := filepath.Join(dir, "converted.wav")
|
||||
if err := utils.AudioToWav(opts.Dst, convertedPath); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
fh, err := os.Open(convertedPath)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() { _ = fh.Close() }()
|
||||
|
||||
d := wav.NewDecoder(fh)
|
||||
buf, err := d.FullPCMBuffer()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
data := buf.AsFloat32Buffer().Data
|
||||
var duration float32
|
||||
if buf.Format != nil && buf.Format.SampleRate > 0 {
|
||||
duration = float32(len(data)) / float32(buf.Format.SampleRate)
|
||||
}
|
||||
|
||||
// Register per-call state and install the C-side callback. defer
|
||||
// teardown so even a panic clears the C pointer (otherwise a stale
|
||||
// callback fires on the next AudioTranscription call).
|
||||
callID := streamCallSeq.Add(1)
|
||||
state := &streamCallState{
|
||||
results: results,
|
||||
diarize: opts.Diarize,
|
||||
}
|
||||
streamCallStates.Store(callID, state)
|
||||
CppSetNewSegmentCallback(goNewSegmentCb, uintptr(callID))
|
||||
defer func() {
|
||||
CppSetNewSegmentCallback(0, 0)
|
||||
streamCallStates.Delete(callID)
|
||||
}()
|
||||
|
||||
// Same abort-watcher pattern as AudioTranscription. Joined synchronously
|
||||
// so a late CppSetAbort(1) cannot fire after this function returns.
|
||||
done := make(chan struct{})
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
CppSetAbort(1)
|
||||
case <-done:
|
||||
}
|
||||
}()
|
||||
defer func() {
|
||||
close(done)
|
||||
wg.Wait()
|
||||
}()
|
||||
|
||||
segsLen := uintptr(0xdeadbeef)
|
||||
segsLenPtr := unsafe.Pointer(&segsLen)
|
||||
ret := CppTranscribe(opts.Threads, opts.Language, opts.Translate, opts.Diarize, data, uintptr(len(data)), segsLenPtr, opts.Prompt)
|
||||
if ret == 2 {
|
||||
return status.Error(codes.Canceled, "transcription cancelled")
|
||||
}
|
||||
if ret != 0 {
|
||||
return fmt.Errorf("Failed Transcribe")
|
||||
}
|
||||
|
||||
// Build the final TranscriptResult. Segments[] mirrors the offline
|
||||
// path so the SSE done event carries the same per-segment shape.
|
||||
// final.Text reuses the assembled stream so concat(deltas) == final.Text
|
||||
// holds exactly, matching the e2e contract.
|
||||
segments := []*pb.TranscriptSegment{}
|
||||
for i := range int(segsLen) {
|
||||
s := CppGetSegmentStart(i) * 10000000
|
||||
t := CppGetSegmentEnd(i) * 10000000
|
||||
txt := strings.ToValidUTF8(strings.Clone(CppGetSegmentText(i)), "<22>")
|
||||
tokens := make([]int32, CppNTokens(i))
|
||||
if opts.Diarize && CppGetSegmentSpeakerTurnNext(i) {
|
||||
txt += " [SPEAKER_TURN]"
|
||||
}
|
||||
for j := range tokens {
|
||||
tokens[j] = int32(CppGetTokenID(i, j))
|
||||
}
|
||||
segments = append(segments, &pb.TranscriptSegment{
|
||||
Id: int32(i),
|
||||
Text: txt,
|
||||
Start: s, End: t,
|
||||
Tokens: tokens,
|
||||
})
|
||||
}
|
||||
|
||||
final := &pb.TranscriptResult{
|
||||
Segments: segments,
|
||||
Text: state.assembled.String(),
|
||||
Language: opts.Language,
|
||||
Duration: duration,
|
||||
}
|
||||
results <- &pb.TranscriptStreamResponse{FinalResult: final}
|
||||
return nil
|
||||
}
|
||||
|
||||
@@ -1,174 +0,0 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/ebitengine/purego"
|
||||
pb "github.com/mudler/LocalAI/pkg/grpc/proto"
|
||||
. "github.com/onsi/ginkgo/v2"
|
||||
. "github.com/onsi/gomega"
|
||||
"google.golang.org/grpc/codes"
|
||||
"google.golang.org/grpc/status"
|
||||
)
|
||||
|
||||
func TestWhisper(t *testing.T) {
|
||||
RegisterFailHandler(Fail)
|
||||
RunSpecs(t, "Whisper Backend Suite")
|
||||
}
|
||||
|
||||
var (
|
||||
libLoadOnce sync.Once
|
||||
libLoadErr error
|
||||
)
|
||||
|
||||
// ensureLibLoaded mirrors main.go's bootstrap so a Go test can drive the
|
||||
// bridge without spinning up the gRPC server. Skips the current spec when the
|
||||
// shared library isn't present (e.g. running before `make backends/whisper`).
|
||||
func ensureLibLoaded() {
|
||||
libLoadOnce.Do(func() {
|
||||
libName := os.Getenv("WHISPER_LIBRARY")
|
||||
if libName == "" {
|
||||
libName = "./libgowhisper-fallback.so"
|
||||
}
|
||||
if _, err := os.Stat(libName); err != nil {
|
||||
libLoadErr = err
|
||||
return
|
||||
}
|
||||
gosd, err := purego.Dlopen(libName, purego.RTLD_NOW|purego.RTLD_GLOBAL)
|
||||
if err != nil {
|
||||
libLoadErr = err
|
||||
return
|
||||
}
|
||||
purego.RegisterLibFunc(&CppLoadModel, gosd, "load_model")
|
||||
purego.RegisterLibFunc(&CppTranscribe, gosd, "transcribe")
|
||||
purego.RegisterLibFunc(&CppGetSegmentText, gosd, "get_segment_text")
|
||||
purego.RegisterLibFunc(&CppGetSegmentStart, gosd, "get_segment_t0")
|
||||
purego.RegisterLibFunc(&CppGetSegmentEnd, gosd, "get_segment_t1")
|
||||
purego.RegisterLibFunc(&CppNTokens, gosd, "n_tokens")
|
||||
purego.RegisterLibFunc(&CppGetTokenID, gosd, "get_token_id")
|
||||
purego.RegisterLibFunc(&CppGetSegmentSpeakerTurnNext, gosd, "get_segment_speaker_turn_next")
|
||||
purego.RegisterLibFunc(&CppSetAbort, gosd, "set_abort")
|
||||
purego.RegisterLibFunc(&CppSetNewSegmentCallback, gosd, "set_new_segment_callback")
|
||||
})
|
||||
if libLoadErr != nil {
|
||||
Skip("whisper library not loadable: " + libLoadErr.Error())
|
||||
}
|
||||
}
|
||||
|
||||
// fixturesOrSkip returns the model + audio paths or skips the spec if either
|
||||
// env var is unset. The test never runs in default CI — it requires a real
|
||||
// whisper model and a long audio file (~3 minutes) on disk.
|
||||
func fixturesOrSkip() (string, string) {
|
||||
modelPath := os.Getenv("WHISPER_MODEL_PATH")
|
||||
audioPath := os.Getenv("WHISPER_AUDIO_PATH")
|
||||
if modelPath == "" || audioPath == "" {
|
||||
Skip("set WHISPER_MODEL_PATH and WHISPER_AUDIO_PATH to run this spec")
|
||||
}
|
||||
return modelPath, audioPath
|
||||
}
|
||||
|
||||
var _ = Describe("Whisper", func() {
|
||||
Context("AudioTranscription cancellation", func() {
|
||||
It("returns codes.Canceled and resets the abort flag for the next call", func() {
|
||||
modelPath, audioPath := fixturesOrSkip()
|
||||
ensureLibLoaded()
|
||||
|
||||
w := &Whisper{}
|
||||
Expect(w.Load(&pb.ModelOptions{ModelFile: modelPath})).To(Succeed())
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
go func() {
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
cancel()
|
||||
}()
|
||||
|
||||
start := time.Now()
|
||||
_, err := w.AudioTranscription(ctx, &pb.TranscriptRequest{
|
||||
Dst: audioPath,
|
||||
Threads: 4,
|
||||
Language: "en",
|
||||
})
|
||||
elapsed := time.Since(start)
|
||||
|
||||
Expect(err).To(HaveOccurred(), "transcription completed in %s without cancel — try a longer audio file", elapsed)
|
||||
st, ok := status.FromError(err)
|
||||
Expect(ok).To(BeTrue(), "expected gRPC status error, got %v", err)
|
||||
Expect(st.Code()).To(Equal(codes.Canceled), "expected codes.Canceled, got %v", err)
|
||||
Expect(elapsed).To(BeNumerically("<", 5*time.Second), "cancellation took %s, expected <5s", elapsed)
|
||||
|
||||
// Subsequent transcription must succeed — proves g_abort reset.
|
||||
res, err := w.AudioTranscription(context.Background(), &pb.TranscriptRequest{
|
||||
Dst: audioPath,
|
||||
Threads: 4,
|
||||
Language: "en",
|
||||
})
|
||||
Expect(err).ToNot(HaveOccurred(), "post-cancel transcription failed")
|
||||
Expect(res.Text).ToNot(BeEmpty(), "post-cancel transcription returned empty text")
|
||||
})
|
||||
})
|
||||
|
||||
Context("AudioTranscriptionStream", func() {
|
||||
It("emits multiple deltas progressively for a multi-segment clip", func() {
|
||||
modelPath, audioPath := fixturesOrSkip()
|
||||
ensureLibLoaded()
|
||||
|
||||
// The streaming method dispatches through the package-level
|
||||
// goNewSegmentCb. main.go normally builds it; in this test
|
||||
// process main() is never called, so build it here lazily.
|
||||
// purego.NewCallback returns a stable pointer; calling it once
|
||||
// per process is correct.
|
||||
if goNewSegmentCb == 0 {
|
||||
goNewSegmentCb = purego.NewCallback(onNewSegment)
|
||||
}
|
||||
|
||||
w := &Whisper{}
|
||||
Expect(w.Load(&pb.ModelOptions{ModelFile: modelPath})).To(Succeed())
|
||||
|
||||
results := make(chan *pb.TranscriptStreamResponse, 64)
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
done <- w.AudioTranscriptionStream(context.Background(), &pb.TranscriptRequest{
|
||||
Dst: audioPath,
|
||||
Threads: 4,
|
||||
Language: "en",
|
||||
Stream: true,
|
||||
}, results)
|
||||
}()
|
||||
|
||||
var deltas []string
|
||||
var assembled strings.Builder
|
||||
var finalText string
|
||||
var finalSegmentCount int
|
||||
for chunk := range results {
|
||||
if d := chunk.GetDelta(); d != "" {
|
||||
deltas = append(deltas, d)
|
||||
assembled.WriteString(d)
|
||||
}
|
||||
if final := chunk.GetFinalResult(); final != nil {
|
||||
finalText = final.GetText()
|
||||
finalSegmentCount = len(final.GetSegments())
|
||||
}
|
||||
}
|
||||
Expect(<-done).ToNot(HaveOccurred())
|
||||
|
||||
// The whisper-specific bar: real streaming via new_segment_callback
|
||||
// fires once per decoded segment, so a multi-segment clip MUST
|
||||
// produce >=2 delta events. A faked-streaming impl (run
|
||||
// whisper_full to completion, then walk the segment list) would
|
||||
// also pass len(deltas) >= 1, which is why the generic e2e spec
|
||||
// is not strict enough.
|
||||
Expect(len(deltas)).To(BeNumerically(">=", 2),
|
||||
"expected multiple deltas from a multi-segment clip, got %d (assembled=%q)",
|
||||
len(deltas), assembled.String())
|
||||
Expect(finalSegmentCount).To(BeNumerically(">=", 2),
|
||||
"expected final to carry multiple segments")
|
||||
Expect(assembled.String()).To(Equal(finalText),
|
||||
"concat(deltas) must equal final.Text")
|
||||
})
|
||||
})
|
||||
})
|
||||
@@ -41,19 +41,12 @@ func main() {
|
||||
{&CppNTokens, "n_tokens"},
|
||||
{&CppGetTokenID, "get_token_id"},
|
||||
{&CppGetSegmentSpeakerTurnNext, "get_segment_speaker_turn_next"},
|
||||
{&CppSetAbort, "set_abort"},
|
||||
{&CppSetNewSegmentCallback, "set_new_segment_callback"},
|
||||
}
|
||||
|
||||
for _, lf := range libFuncs {
|
||||
purego.RegisterLibFunc(lf.FuncPtr, gosd, lf.Name)
|
||||
}
|
||||
|
||||
// Build a stable C-callable function pointer from the Go callback. The
|
||||
// pointer lives for the lifetime of the process; per-call dispatch is
|
||||
// keyed by user_data through streamCallStates.
|
||||
goNewSegmentCb = purego.NewCallback(onNewSegment)
|
||||
|
||||
flag.Parse()
|
||||
|
||||
if err := grpc.StartServer(*addr, &Whisper{}); err != nil {
|
||||
|
||||
@@ -72,29 +72,6 @@
|
||||
nvidia-cuda-12: "cuda12-turboquant"
|
||||
nvidia-l4t-cuda-12: "nvidia-l4t-arm64-turboquant"
|
||||
nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-turboquant"
|
||||
- &ds4
|
||||
name: "ds4"
|
||||
alias: "ds4"
|
||||
license: mit
|
||||
description: |
|
||||
antirez/ds4 - DeepSeek V4 Flash inference engine. Single-model,
|
||||
optimized for Metal (Darwin) and CUDA (Linux). Requires the GGUFs
|
||||
published at huggingface.co/antirez/deepseek-v4-gguf.
|
||||
urls:
|
||||
- https://github.com/antirez/ds4
|
||||
tags:
|
||||
- text-to-text
|
||||
- LLM
|
||||
- CPU
|
||||
- CUDA
|
||||
- Metal
|
||||
capabilities:
|
||||
default: "cpu-ds4"
|
||||
nvidia: "cuda13-ds4"
|
||||
nvidia-cuda-13: "cuda13-ds4"
|
||||
nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-ds4"
|
||||
metal: "metal-ds4"
|
||||
metal-darwin-arm64: "metal-ds4"
|
||||
- &whispercpp
|
||||
name: "whisper"
|
||||
alias: "whisper"
|
||||
@@ -310,7 +287,6 @@
|
||||
amd: "rocm-sglang"
|
||||
intel: "intel-sglang"
|
||||
nvidia-cuda-12: "cuda12-sglang"
|
||||
nvidia-cuda-13: "cuda13-sglang"
|
||||
nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-sglang"
|
||||
cpu: "cpu-sglang"
|
||||
- &vllm-omni
|
||||
@@ -1150,15 +1126,6 @@
|
||||
nvidia-cuda-12: "cuda12-turboquant-development"
|
||||
nvidia-l4t-cuda-12: "nvidia-l4t-arm64-turboquant-development"
|
||||
nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-turboquant-development"
|
||||
- !!merge <<: *ds4
|
||||
name: "ds4-development"
|
||||
capabilities:
|
||||
default: "cpu-ds4-development"
|
||||
nvidia: "cuda13-ds4-development"
|
||||
nvidia-cuda-13: "cuda13-ds4-development"
|
||||
nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-ds4-development"
|
||||
metal: "metal-ds4-development"
|
||||
metal-darwin-arm64: "metal-ds4-development"
|
||||
- !!merge <<: *stablediffusionggml
|
||||
name: "stablediffusion-ggml-development"
|
||||
capabilities:
|
||||
@@ -1705,47 +1672,6 @@
|
||||
uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-turboquant"
|
||||
mirrors:
|
||||
- localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-turboquant
|
||||
## ds4
|
||||
- !!merge <<: *ds4
|
||||
name: "cpu-ds4"
|
||||
uri: "quay.io/go-skynet/local-ai-backends:latest-cpu-ds4"
|
||||
mirrors:
|
||||
- localai/localai-backends:latest-cpu-ds4
|
||||
- !!merge <<: *ds4
|
||||
name: "cpu-ds4-development"
|
||||
uri: "quay.io/go-skynet/local-ai-backends:master-cpu-ds4"
|
||||
mirrors:
|
||||
- localai/localai-backends:master-cpu-ds4
|
||||
- !!merge <<: *ds4
|
||||
name: "cuda13-ds4"
|
||||
uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-ds4"
|
||||
mirrors:
|
||||
- localai/localai-backends:latest-gpu-nvidia-cuda-13-ds4
|
||||
- !!merge <<: *ds4
|
||||
name: "cuda13-ds4-development"
|
||||
uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-ds4"
|
||||
mirrors:
|
||||
- localai/localai-backends:master-gpu-nvidia-cuda-13-ds4
|
||||
- !!merge <<: *ds4
|
||||
name: "cuda13-nvidia-l4t-arm64-ds4"
|
||||
uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-ds4"
|
||||
mirrors:
|
||||
- localai/localai-backends:latest-nvidia-l4t-cuda-13-arm64-ds4
|
||||
- !!merge <<: *ds4
|
||||
name: "cuda13-nvidia-l4t-arm64-ds4-development"
|
||||
uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-ds4"
|
||||
mirrors:
|
||||
- localai/localai-backends:master-nvidia-l4t-cuda-13-arm64-ds4
|
||||
- !!merge <<: *ds4
|
||||
name: "metal-ds4"
|
||||
uri: "quay.io/go-skynet/local-ai-backends:latest-metal-darwin-arm64-ds4"
|
||||
mirrors:
|
||||
- localai/localai-backends:latest-metal-darwin-arm64-ds4
|
||||
- !!merge <<: *ds4
|
||||
name: "metal-ds4-development"
|
||||
uri: "quay.io/go-skynet/local-ai-backends:master-metal-darwin-arm64-ds4"
|
||||
mirrors:
|
||||
- localai/localai-backends:master-metal-darwin-arm64-ds4
|
||||
## whisper
|
||||
- !!merge <<: *whispercpp
|
||||
name: "whisper-development"
|
||||
@@ -2039,7 +1965,6 @@
|
||||
amd: "rocm-sglang-development"
|
||||
intel: "intel-sglang-development"
|
||||
nvidia-cuda-12: "cuda12-sglang-development"
|
||||
nvidia-cuda-13: "cuda13-sglang-development"
|
||||
nvidia-l4t-cuda-13: "cuda13-nvidia-l4t-arm64-sglang-development"
|
||||
cpu: "cpu-sglang-development"
|
||||
- !!merge <<: *sglang
|
||||
@@ -2047,11 +1972,6 @@
|
||||
uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-sglang"
|
||||
mirrors:
|
||||
- localai/localai-backends:latest-gpu-nvidia-cuda-12-sglang
|
||||
- !!merge <<: *sglang
|
||||
name: "cuda13-sglang"
|
||||
uri: "quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-13-sglang"
|
||||
mirrors:
|
||||
- localai/localai-backends:latest-gpu-nvidia-cuda-13-sglang
|
||||
- !!merge <<: *sglang
|
||||
name: "cuda13-nvidia-l4t-arm64-sglang"
|
||||
uri: "quay.io/go-skynet/local-ai-backends:latest-nvidia-l4t-cuda-13-arm64-sglang"
|
||||
@@ -2077,11 +1997,6 @@
|
||||
uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-12-sglang"
|
||||
mirrors:
|
||||
- localai/localai-backends:master-gpu-nvidia-cuda-12-sglang
|
||||
- !!merge <<: *sglang
|
||||
name: "cuda13-sglang-development"
|
||||
uri: "quay.io/go-skynet/local-ai-backends:master-gpu-nvidia-cuda-13-sglang"
|
||||
mirrors:
|
||||
- localai/localai-backends:master-gpu-nvidia-cuda-13-sglang
|
||||
- !!merge <<: *sglang
|
||||
name: "cuda13-nvidia-l4t-arm64-sglang-development"
|
||||
uri: "quay.io/go-skynet/local-ai-backends:master-nvidia-l4t-cuda-13-arm64-sglang"
|
||||
|
||||
@@ -23,15 +23,3 @@ fi
|
||||
|
||||
|
||||
installRequirements
|
||||
|
||||
# chatterbox-tts upstream pulls `russian-text-stresser` (unpinned git URL) which
|
||||
# transitively pins spacy==3.6.* and other ancient packages. That cascade forces
|
||||
# pip to backtrack through Jinja2/MarkupSafe/omegaconf/ruamel.yaml into Python-2-era
|
||||
# sdists that no longer build. We install chatterbox-tts itself with --no-deps and
|
||||
# list its real runtime deps in requirements-*.txt instead.
|
||||
echo "Installing chatterbox-tts with --no-deps"
|
||||
if [ "x${USE_PIP}" == "xtrue" ]; then
|
||||
pip install ${EXTRA_PIP_INSTALL_FLAGS:-} --no-deps "chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster"
|
||||
else
|
||||
uv pip install ${EXTRA_PIP_INSTALL_FLAGS:-} --no-deps "chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster"
|
||||
fi
|
||||
|
||||
@@ -4,16 +4,6 @@ torch
|
||||
torchaudio
|
||||
numpy>=1.24.0,<1.26.0
|
||||
transformers
|
||||
# chatterbox-tts itself is installed with --no-deps in install.sh.
|
||||
# These are its real runtime deps, mirroring upstream's pyproject.toml
|
||||
# minus russian-text-stresser (whose ancient pins break the resolver).
|
||||
omegaconf==2.3.0
|
||||
resampy==0.4.3
|
||||
librosa
|
||||
s3tokenizer
|
||||
diffusers
|
||||
resemble-perth==1.0.1
|
||||
conformer
|
||||
safetensors
|
||||
spacy-pkuseg
|
||||
pykakasi==2.3.0
|
||||
# https://github.com/mudler/LocalAI/pull/6240#issuecomment-3329518289
|
||||
chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster
|
||||
#chatterbox-tts==0.1.4
|
||||
@@ -2,17 +2,6 @@ torch
|
||||
torchaudio
|
||||
transformers
|
||||
numpy>=1.24.0,<1.26.0
|
||||
# chatterbox-tts itself is installed with --no-deps in install.sh.
|
||||
# These are its real runtime deps, mirroring upstream's pyproject.toml
|
||||
# minus russian-text-stresser (whose ancient pins break the resolver).
|
||||
omegaconf==2.3.0
|
||||
resampy==0.4.3
|
||||
librosa
|
||||
s3tokenizer
|
||||
diffusers
|
||||
resemble-perth==1.0.1
|
||||
conformer
|
||||
safetensors
|
||||
spacy-pkuseg
|
||||
pykakasi==2.3.0
|
||||
# https://github.com/mudler/LocalAI/pull/6240#issuecomment-3329518289
|
||||
chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster
|
||||
accelerate
|
||||
|
||||
@@ -3,17 +3,6 @@ torch
|
||||
torchaudio
|
||||
transformers
|
||||
numpy>=1.24.0,<1.26.0
|
||||
# chatterbox-tts itself is installed with --no-deps in install.sh.
|
||||
# These are its real runtime deps, mirroring upstream's pyproject.toml
|
||||
# minus russian-text-stresser (whose ancient pins break the resolver).
|
||||
omegaconf==2.3.0
|
||||
resampy==0.4.3
|
||||
librosa
|
||||
s3tokenizer
|
||||
diffusers
|
||||
resemble-perth==1.0.1
|
||||
conformer
|
||||
safetensors
|
||||
spacy-pkuseg
|
||||
pykakasi==2.3.0
|
||||
# https://github.com/mudler/LocalAI/pull/6240#issuecomment-3329518289
|
||||
chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster
|
||||
accelerate
|
||||
|
||||
@@ -3,17 +3,6 @@ torch==2.10.0+rocm7.0
|
||||
torchaudio==2.10.0+rocm7.0
|
||||
transformers
|
||||
numpy>=1.24.0,<1.26.0
|
||||
# chatterbox-tts itself is installed with --no-deps in install.sh.
|
||||
# These are its real runtime deps, mirroring upstream's pyproject.toml
|
||||
# minus russian-text-stresser (whose ancient pins break the resolver).
|
||||
omegaconf==2.3.0
|
||||
resampy==0.4.3
|
||||
librosa
|
||||
s3tokenizer
|
||||
diffusers
|
||||
resemble-perth==1.0.1
|
||||
conformer
|
||||
safetensors
|
||||
spacy-pkuseg
|
||||
pykakasi==2.3.0
|
||||
# https://github.com/mudler/LocalAI/pull/6240#issuecomment-3329518289
|
||||
chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster
|
||||
accelerate
|
||||
|
||||
@@ -3,19 +3,8 @@ torch
|
||||
torchaudio
|
||||
transformers
|
||||
numpy>=1.24.0,<1.26.0
|
||||
# chatterbox-tts itself is installed with --no-deps in install.sh.
|
||||
# These are its real runtime deps, mirroring upstream's pyproject.toml
|
||||
# minus russian-text-stresser (whose ancient pins break the resolver).
|
||||
omegaconf==2.3.0
|
||||
resampy==0.4.3
|
||||
librosa
|
||||
s3tokenizer
|
||||
diffusers
|
||||
resemble-perth==1.0.1
|
||||
conformer
|
||||
safetensors
|
||||
spacy-pkuseg
|
||||
pykakasi==2.3.0
|
||||
# https://github.com/mudler/LocalAI/pull/6240#issuecomment-3329518289
|
||||
chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster
|
||||
accelerate
|
||||
oneccl_bind_pt==2.3.100+xpu
|
||||
optimum[openvino]
|
||||
|
||||
@@ -3,17 +3,5 @@ torch
|
||||
torchaudio
|
||||
transformers
|
||||
numpy>=1.24.0,<1.26.0
|
||||
# chatterbox-tts itself is installed with --no-deps in install.sh.
|
||||
# These are its real runtime deps, mirroring upstream's pyproject.toml
|
||||
# minus russian-text-stresser (whose ancient pins break the resolver).
|
||||
omegaconf==2.3.0
|
||||
resampy==0.4.3
|
||||
librosa
|
||||
s3tokenizer
|
||||
diffusers
|
||||
resemble-perth==1.0.1
|
||||
conformer
|
||||
safetensors
|
||||
spacy-pkuseg
|
||||
pykakasi==2.3.0
|
||||
chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster
|
||||
accelerate
|
||||
|
||||
@@ -3,17 +3,5 @@ torch
|
||||
torchaudio
|
||||
transformers
|
||||
numpy>=1.24.0,<1.26.0
|
||||
# chatterbox-tts itself is installed with --no-deps in install.sh.
|
||||
# These are its real runtime deps, mirroring upstream's pyproject.toml
|
||||
# minus russian-text-stresser (whose ancient pins break the resolver).
|
||||
omegaconf==2.3.0
|
||||
resampy==0.4.3
|
||||
librosa
|
||||
s3tokenizer
|
||||
diffusers
|
||||
resemble-perth==1.0.1
|
||||
conformer
|
||||
safetensors
|
||||
spacy-pkuseg
|
||||
pykakasi==2.3.0
|
||||
chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster
|
||||
accelerate
|
||||
|
||||
@@ -3,16 +3,5 @@ torchaudio
|
||||
accelerate
|
||||
numpy>=1.24.0,<1.26.0
|
||||
transformers
|
||||
# chatterbox-tts itself is installed with --no-deps in install.sh.
|
||||
# These are its real runtime deps, mirroring upstream's pyproject.toml
|
||||
# minus russian-text-stresser (whose ancient pins break the resolver).
|
||||
omegaconf==2.3.0
|
||||
resampy==0.4.3
|
||||
librosa
|
||||
s3tokenizer
|
||||
diffusers
|
||||
resemble-perth==1.0.1
|
||||
conformer
|
||||
safetensors
|
||||
spacy-pkuseg
|
||||
pykakasi==2.3.0
|
||||
# https://github.com/mudler/LocalAI/pull/6240#issuecomment-3329518289
|
||||
chatterbox-tts@git+https://git@github.com/mudler/chatterbox.git@faster
|
||||
@@ -10,5 +10,4 @@ protogen-clean:
|
||||
|
||||
.PHONY: clean
|
||||
clean: protogen-clean
|
||||
rm -rf venv __pycache__
|
||||
# trigger per-arch+merge rebuild for faster-whisper pilot
|
||||
rm -rf venv __pycache__
|
||||
@@ -8,12 +8,6 @@ run: sglang
|
||||
bash run.sh
|
||||
@echo "sglang run."
|
||||
|
||||
.PHONY: test
|
||||
test: sglang
|
||||
@echo "Testing sglang..."
|
||||
bash test.sh
|
||||
@echo "sglang tested."
|
||||
|
||||
.PHONY: protogen-clean
|
||||
protogen-clean:
|
||||
$(RM) backend_pb2_grpc.py backend_pb2.py
|
||||
|
||||
@@ -9,18 +9,10 @@ The streaming path applies sglang's per-request FunctionCallParser and
|
||||
ReasoningParser so tool_calls and reasoning_content are emitted
|
||||
incrementally inside ChatDelta, which is a capability sglang exposes
|
||||
natively and vLLM does not.
|
||||
|
||||
Like the vLLM backend, this one accepts an arbitrary ``engine_args:``
|
||||
map in the model YAML; keys are validated against ``ServerArgs`` fields
|
||||
and forwarded to ``Engine(**kwargs)``. That covers speculative decoding
|
||||
(EAGLE/EAGLE3/DFLASH/NGRAM/STANDALONE plus MTP via NEXTN), attention
|
||||
backend selection, MoE knobs, hierarchical cache, and so on.
|
||||
"""
|
||||
import asyncio
|
||||
from concurrent import futures
|
||||
import argparse
|
||||
import dataclasses
|
||||
import difflib
|
||||
import signal
|
||||
import sys
|
||||
import os
|
||||
@@ -46,7 +38,6 @@ from grpc_auth import get_auth_interceptors
|
||||
# are wrapped in try/except so older / leaner installs that omit them
|
||||
# still load the backend for plain text generation.
|
||||
from sglang.srt.entrypoints.engine import Engine
|
||||
from sglang.srt.server_args import ServerArgs
|
||||
|
||||
try:
|
||||
from sglang.srt.function_call.function_call_parser import FunctionCallParser
|
||||
@@ -75,19 +66,6 @@ except Exception:
|
||||
HAS_TRANSFORMERS = False
|
||||
|
||||
|
||||
# sglang 0.5.11 renamed SamplingParams.seed -> sampling_seed (PR #21952).
|
||||
# Earlier 0.5.x releases (e.g. 0.5.1.post2 — the wheel still pinned by the
|
||||
# pypi.jetson-ai-lab.io sbsa/cu130 mirror used by the l4t13 build profile)
|
||||
# accept only `seed`. Detect the supported keyword once at import time so
|
||||
# both versions work without a hard pin floor.
|
||||
try:
|
||||
import inspect as _inspect
|
||||
from sglang.srt.sampling.sampling_params import SamplingParams as _SamplingParams
|
||||
_SEED_KEY = "sampling_seed" if "sampling_seed" in _inspect.signature(_SamplingParams).parameters else "seed"
|
||||
except Exception:
|
||||
_SEED_KEY = "sampling_seed"
|
||||
|
||||
|
||||
_ONE_DAY_IN_SECONDS = 60 * 60 * 24
|
||||
MAX_WORKERS = int(os.environ.get('PYTHON_GRPC_MAX_WORKERS', '1'))
|
||||
|
||||
@@ -104,37 +82,6 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
|
||||
opts[key.strip()] = value.strip()
|
||||
return opts
|
||||
|
||||
def _apply_engine_args(self, engine_kwargs: dict, engine_args_json: str) -> dict:
|
||||
"""Merge user-supplied engine_args (JSON object) into the kwargs dict
|
||||
that will be forwarded to ``sglang.Engine`` (which constructs a
|
||||
``ServerArgs`` from them).
|
||||
|
||||
Mirrors ``backend/python/vllm/backend.py::_apply_engine_args`` but
|
||||
operates on the kwargs dict because sglang's ``Engine.__init__``
|
||||
accepts ``**kwargs`` directly rather than a pre-built dataclass.
|
||||
Validation happens against ``ServerArgs`` fields so a typo fails
|
||||
early with a close-match suggestion instead of producing a confusing
|
||||
``TypeError`` deep inside engine startup.
|
||||
"""
|
||||
if not engine_args_json:
|
||||
return engine_kwargs
|
||||
try:
|
||||
extra = json.loads(engine_args_json)
|
||||
except json.JSONDecodeError as e:
|
||||
raise ValueError(f"engine_args is not valid JSON: {e}") from e
|
||||
if not isinstance(extra, dict):
|
||||
raise ValueError(
|
||||
f"engine_args must be a JSON object, got {type(extra).__name__}"
|
||||
)
|
||||
valid = {f.name for f in dataclasses.fields(ServerArgs)}
|
||||
for key in extra:
|
||||
if key not in valid:
|
||||
suggestion = difflib.get_close_matches(key, valid, n=1)
|
||||
hint = f" did you mean {suggestion[0]!r}?" if suggestion else ""
|
||||
raise ValueError(f"unknown engine_args key {key!r}.{hint}")
|
||||
engine_kwargs.update(extra)
|
||||
return engine_kwargs
|
||||
|
||||
def _messages_to_dicts(self, messages) -> List[dict]:
|
||||
result: List[dict] = []
|
||||
for msg in messages:
|
||||
@@ -190,16 +137,6 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
|
||||
if self.reasoning_parser_name:
|
||||
engine_kwargs["reasoning_parser"] = self.reasoning_parser_name
|
||||
|
||||
# engine_args from YAML overrides typed fields above so operators can
|
||||
# tune anything ServerArgs exposes (speculative decoding, attention
|
||||
# backend, MoE, hierarchical cache, …) without waiting on protobuf
|
||||
# changes.
|
||||
try:
|
||||
engine_kwargs = self._apply_engine_args(engine_kwargs, request.EngineArgs)
|
||||
except ValueError as err:
|
||||
print(f"engine_args error: {err}", file=sys.stderr)
|
||||
return backend_pb2.Result(success=False, message=str(err))
|
||||
|
||||
try:
|
||||
self.llm = Engine(**engine_kwargs)
|
||||
except Exception as err:
|
||||
@@ -284,7 +221,7 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
|
||||
"TopP": "top_p",
|
||||
"TopK": "top_k",
|
||||
"MinP": "min_p",
|
||||
"Seed": _SEED_KEY,
|
||||
"Seed": "seed",
|
||||
"StopPrompts": "stop",
|
||||
"StopTokenIds": "stop_token_ids",
|
||||
"IgnoreEOS": "ignore_eos",
|
||||
|
||||
@@ -23,32 +23,17 @@ if [ "x${BUILD_PROFILE}" == "xcpu" ]; then
|
||||
EXTRA_PIP_INSTALL_FLAGS+=" --index-strategy=unsafe-best-match"
|
||||
fi
|
||||
|
||||
# cublas12 needs a cu128 torch index (see requirements-cublas12.txt) — without
|
||||
# unsafe-best-match uv falls through to default PyPI's cu130 torch wheel and
|
||||
# the resulting sgl-kernel can't load on our cu12 host libs.
|
||||
if [ "x${BUILD_PROFILE}" == "xcublas12" ]; then
|
||||
EXTRA_PIP_INSTALL_FLAGS+=" --index-strategy=unsafe-best-match"
|
||||
fi
|
||||
|
||||
# sglang 0.5.11 (Gemma 4 support) declares flash-attn-4 as a hard dep, but
|
||||
# upstream only publishes pre-release wheels (4.0.0b*). uv rejects
|
||||
# pre-releases by default — opt in for sglang specifically. Drop this once
|
||||
# flash-attn-4 4.0 stable lands.
|
||||
EXTRA_PIP_INSTALL_FLAGS+=" --prerelease=allow"
|
||||
|
||||
# JetPack 7 / L4T arm64 wheels are built for cp312 and shipped via
|
||||
# pypi.jetson-ai-lab.io. Bump the venv Python so the prebuilt sglang
|
||||
# wheel resolves cleanly. The actual install on l4t13 goes through
|
||||
# pyproject.toml (see the elif branch below) so [tool.uv.sources] can
|
||||
# pin only torch/torchvision/torchaudio/sglang to the jetson-ai-lab
|
||||
# index — leaving PyPI as the path for transitive deps like
|
||||
# markdown-it-py / anthropic / propcache that the L4T mirror's proxy
|
||||
# 503s on. No --index-strategy flag here: the explicit index keeps the
|
||||
# scoping clean.
|
||||
# wheel resolves cleanly. unsafe-best-match is required because the
|
||||
# jetson-ai-lab index lists transitive deps (e.g. decord) at older
|
||||
# versions only — without it uv refuses to fall through to PyPI for a
|
||||
# compatible wheel and resolution fails.
|
||||
if [ "x${BUILD_PROFILE}" == "xl4t13" ]; then
|
||||
PYTHON_VERSION="3.12"
|
||||
PYTHON_PATCH="12"
|
||||
PY_STANDALONE_TAG="20251120"
|
||||
EXTRA_PIP_INSTALL_FLAGS+=" --index-strategy=unsafe-best-match"
|
||||
fi
|
||||
|
||||
# sglang's CPU path has no prebuilt wheel on PyPI — upstream publishes
|
||||
@@ -110,27 +95,6 @@ if [ "x${BUILD_TYPE}" == "x" ] || [ "x${FROM_SOURCE:-}" == "xtrue" ]; then
|
||||
fi
|
||||
uv pip install ${EXTRA_PIP_INSTALL_FLAGS:-} .
|
||||
popd
|
||||
# L4T arm64 (JetPack 7): drive the install through pyproject.toml so that
|
||||
# [tool.uv.sources] can pin torch/torchvision/torchaudio/sglang to the
|
||||
# jetson-ai-lab index, while everything else (transitive deps and
|
||||
# PyPI-resolvable packages like transformers / accelerate) comes from
|
||||
# PyPI. Bypasses installRequirements because uv pip install -r
|
||||
# requirements.txt does not honor sources — see
|
||||
# backend/python/sglang/pyproject.toml for the rationale. Mirrors the
|
||||
# equivalent path in backend/python/vllm/install.sh.
|
||||
elif [ "x${BUILD_PROFILE}" == "xl4t13" ]; then
|
||||
ensureVenv
|
||||
if [ "x${PORTABLE_PYTHON}" == "xtrue" ]; then
|
||||
export C_INCLUDE_PATH="${C_INCLUDE_PATH:-}:$(_portable_dir)/include/python${PYTHON_VERSION}"
|
||||
fi
|
||||
pushd "${backend_dir}"
|
||||
# Build deps first (matches installRequirements' requirements-install.txt
|
||||
# pass — sglang/sgl-kernel sdists need packaging/setuptools-scm in the
|
||||
# venv before they can build under --no-build-isolation).
|
||||
uv pip install ${EXTRA_PIP_INSTALL_FLAGS:-} -r requirements-install.txt
|
||||
uv pip install ${EXTRA_PIP_INSTALL_FLAGS:-} --requirement pyproject.toml
|
||||
popd
|
||||
runProtogen
|
||||
else
|
||||
installRequirements
|
||||
fi
|
||||
|
||||
@@ -1,68 +0,0 @@
|
||||
# L4T arm64 (JetPack 7 / sbsa cu130) install spec for the sglang backend.
|
||||
#
|
||||
# Why this file exists, and why only the l4t13 BUILD_PROFILE consumes it:
|
||||
#
|
||||
# pypi.jetson-ai-lab.io hosts the L4T-specific torch / sglang / sgl-kernel
|
||||
# wheels we need on aarch64 + cuda13, but it ALSO transparently proxies the
|
||||
# rest of PyPI through `/+f/<sha>/<filename>` URLs that 503 frequently.
|
||||
# With `--extra-index-url` + `--index-strategy=unsafe-best-match` (the
|
||||
# historical fix in install.sh) uv would pick those proxy URLs for ordinary
|
||||
# PyPI packages — markdown-it-py, anthropic, propcache, etc. — and trip on
|
||||
# the 503s. See e.g. CI run 25439791228 (markdown-it-py-4.0.0).
|
||||
#
|
||||
# `explicit = true` on the index makes uv consult the L4T mirror ONLY for
|
||||
# packages mapped under [tool.uv.sources]. Everything else goes to PyPI.
|
||||
# This breaks the historical 503 path without losing access to the L4T
|
||||
# wheels we actually need from there. Mirrors the equivalent fix already
|
||||
# in backend/python/vllm/pyproject.toml.
|
||||
#
|
||||
# `uv pip install -r requirements.txt` does NOT honor [tool.uv.sources]
|
||||
# (sources are project-mode only, not pip-compat mode), so install.sh's
|
||||
# l4t13 branch invokes `uv pip install --requirement pyproject.toml`
|
||||
# directly. Other BUILD_PROFILEs continue to use the requirements-*.txt
|
||||
# pipeline through libbackend.sh's installRequirements and never read
|
||||
# this file.
|
||||
[project]
|
||||
name = "localai-sglang-l4t13"
|
||||
version = "0.0.0"
|
||||
requires-python = ">=3.12,<3.13"
|
||||
dependencies = [
|
||||
# Mirror of requirements.txt — kept in sync manually for now since the
|
||||
# l4t13 path bypasses installRequirements (see install.sh).
|
||||
"grpcio==1.80.0",
|
||||
"protobuf",
|
||||
"certifi",
|
||||
"setuptools",
|
||||
"pillow",
|
||||
# L4T-specific accelerator stack (sourced from jetson-ai-lab below).
|
||||
"torch",
|
||||
"torchvision",
|
||||
"torchaudio",
|
||||
# sglang on jetson — the [all] extra is deliberately omitted because it
|
||||
# pulls outlines/decord, and decord has no aarch64 cp312 wheel anywhere
|
||||
# (PyPI nor the jetson-ai-lab index ships only legacy cp35-cp37). With
|
||||
# [all] uv backtracks through versions trying to satisfy decord and
|
||||
# lands on sglang==0.1.16. The 0.5.0 floor matches the only major
|
||||
# series the jetson-ai-lab sbsa/cu130 mirror currently publishes
|
||||
# (sglang==0.5.1.post2 as of 2026-05-06). Bumping to >=0.5.11 here
|
||||
# would make the build unsatisfiable until the mirror catches up.
|
||||
# Gemma 4 / MTP recipes are therefore not supported on l4t13 — those
|
||||
# features land on cublas12/cublas13 hosts that pull the newer wheel
|
||||
# from PyPI. backend.py keeps backward compat with the 0.5.x SamplingParams
|
||||
# field rename via runtime detection.
|
||||
"sglang>=0.5.0",
|
||||
# PyPI-resolvable packages that complete the runtime.
|
||||
"accelerate",
|
||||
"transformers",
|
||||
]
|
||||
|
||||
[[tool.uv.index]]
|
||||
name = "jetson-ai-lab"
|
||||
url = "https://pypi.jetson-ai-lab.io/sbsa/cu130"
|
||||
explicit = true
|
||||
|
||||
[tool.uv.sources]
|
||||
torch = { index = "jetson-ai-lab" }
|
||||
torchvision = { index = "jetson-ai-lab" }
|
||||
torchaudio = { index = "jetson-ai-lab" }
|
||||
sglang = { index = "jetson-ai-lab" }
|
||||
@@ -1,4 +1,3 @@
|
||||
# Bump this pin deliberately — sglang releases weekly and API surfaces
|
||||
# (FunctionCallParser, ReasoningParser) move between releases.
|
||||
# 0.5.11 is the floor for Gemma 4 support (PR sgl-project/sglang#21952).
|
||||
sglang[all]>=0.5.11
|
||||
sglang[all]>=0.4.0
|
||||
|
||||
@@ -1,12 +1,5 @@
|
||||
# sglang 0.5.11 hard-pins torch==2.9.1. PyPI's default torch 2.9.1 wheel is
|
||||
# now the cu130 build, which drags in cu130-flavoured sgl-kernel/sglang-kernel
|
||||
# binaries that need libnvrtc.so.13 — incompatible with our cu12 host libs.
|
||||
# Pin the cu128 PyTorch index so uv pulls cu12-flavoured torch (and the
|
||||
# matching sgl-kernel cu12 wheels). install.sh adds --index-strategy=unsafe-best-match
|
||||
# for cublas12 so uv consults this index alongside PyPI.
|
||||
--extra-index-url https://download.pytorch.org/whl/cu128
|
||||
accelerate
|
||||
torch==2.9.1
|
||||
torch==2.7.1
|
||||
torchvision
|
||||
torchaudio
|
||||
torchaudio==2.7.1
|
||||
transformers
|
||||
|
||||
@@ -1,4 +0,0 @@
|
||||
# Bump this pin deliberately — sglang releases weekly and API surfaces
|
||||
# (FunctionCallParser, ReasoningParser) move between releases.
|
||||
# 0.5.11 is the floor for Gemma 4 support (PR sgl-project/sglang#21952).
|
||||
sglang[all]>=0.5.11
|
||||
@@ -1,6 +0,0 @@
|
||||
--extra-index-url https://download.pytorch.org/whl/cu130
|
||||
accelerate
|
||||
torch
|
||||
torchvision
|
||||
torchaudio
|
||||
transformers
|
||||
12
backend/python/sglang/requirements-l4t13.txt
Normal file
12
backend/python/sglang/requirements-l4t13.txt
Normal file
@@ -0,0 +1,12 @@
|
||||
--extra-index-url https://pypi.jetson-ai-lab.io/sbsa/cu130
|
||||
accelerate
|
||||
torch
|
||||
torchvision
|
||||
torchaudio
|
||||
transformers
|
||||
# Drop the [all] extra: it pulls outlines/decord, and decord has no
|
||||
# aarch64 cp312 wheel anywhere (PyPI nor the jetson-ai-lab index ships
|
||||
# only legacy cp35-cp37). With [all] uv backtracks through versions
|
||||
# trying to satisfy decord and lands on sglang==0.1.16. Floor at 0.5.0
|
||||
# so uv can't silently downgrade if a future resolution misfires.
|
||||
sglang>=0.5.0
|
||||
@@ -1,101 +0,0 @@
|
||||
"""Unit tests for the sglang backend.
|
||||
|
||||
Helper-level tests run without launching the gRPC server or loading model
|
||||
weights — they only exercise the pure-Python helpers on
|
||||
``BackendServicer``. They do still require ``sglang`` to be importable
|
||||
because ``_apply_engine_args`` validates keys against
|
||||
``ServerArgs``'s dataclass fields.
|
||||
"""
|
||||
import unittest
|
||||
|
||||
|
||||
class TestSglangHelpers(unittest.TestCase):
|
||||
"""Tests for the pure helpers on BackendServicer (no gRPC, no engine)."""
|
||||
|
||||
def _servicer(self):
|
||||
import sys
|
||||
import os
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
from backend import BackendServicer # noqa: E402
|
||||
return BackendServicer()
|
||||
|
||||
def test_parse_options(self):
|
||||
servicer = self._servicer()
|
||||
opts = servicer._parse_options([
|
||||
"tool_parser:hermes",
|
||||
"reasoning_parser:deepseek_r1",
|
||||
"invalid_no_colon",
|
||||
"key_with_colons:a:b:c",
|
||||
])
|
||||
self.assertEqual(opts["tool_parser"], "hermes")
|
||||
self.assertEqual(opts["reasoning_parser"], "deepseek_r1")
|
||||
self.assertEqual(opts["key_with_colons"], "a:b:c")
|
||||
self.assertNotIn("invalid_no_colon", opts)
|
||||
|
||||
def test_apply_engine_args_known_keys(self):
|
||||
"""User-supplied JSON merges into the kwargs dict; pre-set typed
|
||||
fields stay put when not overridden."""
|
||||
import json as _json
|
||||
servicer = self._servicer()
|
||||
base = {
|
||||
"model_path": "facebook/opt-125m",
|
||||
"mem_fraction_static": 0.7,
|
||||
}
|
||||
extras = _json.dumps({
|
||||
"trust_remote_code": True,
|
||||
"speculative_algorithm": "EAGLE",
|
||||
"speculative_num_steps": 1,
|
||||
})
|
||||
out = servicer._apply_engine_args(base, extras)
|
||||
self.assertIs(out, base) # in-place merge — same dict back
|
||||
self.assertTrue(out["trust_remote_code"])
|
||||
self.assertEqual(out["speculative_algorithm"], "EAGLE")
|
||||
self.assertEqual(out["speculative_num_steps"], 1)
|
||||
self.assertEqual(out["model_path"], "facebook/opt-125m")
|
||||
self.assertEqual(out["mem_fraction_static"], 0.7)
|
||||
|
||||
def test_apply_engine_args_engine_args_overrides_typed_fields(self):
|
||||
"""engine_args wins over previously-set typed kwargs (vLLM precedence)."""
|
||||
import json as _json
|
||||
servicer = self._servicer()
|
||||
base = {"model_path": "facebook/opt-125m", "mem_fraction_static": 0.7}
|
||||
out = servicer._apply_engine_args(
|
||||
base, _json.dumps({"mem_fraction_static": 0.5}),
|
||||
)
|
||||
self.assertEqual(out["mem_fraction_static"], 0.5)
|
||||
|
||||
def test_apply_engine_args_unknown_key_raises(self):
|
||||
"""Typo'd key raises ValueError with a close-match suggestion."""
|
||||
import json as _json
|
||||
servicer = self._servicer()
|
||||
base = {"model_path": "facebook/opt-125m"}
|
||||
with self.assertRaises(ValueError) as ctx:
|
||||
servicer._apply_engine_args(
|
||||
base, _json.dumps({"trust_remotecode": True}),
|
||||
)
|
||||
msg = str(ctx.exception)
|
||||
self.assertIn("trust_remotecode", msg)
|
||||
self.assertIn("trust_remote_code", msg)
|
||||
|
||||
def test_apply_engine_args_empty_passthrough(self):
|
||||
"""Empty / None engine_args returns the kwargs dict untouched."""
|
||||
servicer = self._servicer()
|
||||
base = {"model_path": "facebook/opt-125m"}
|
||||
self.assertIs(servicer._apply_engine_args(base, ""), base)
|
||||
self.assertIs(servicer._apply_engine_args(base, None), base)
|
||||
|
||||
def test_apply_engine_args_invalid_json_raises(self):
|
||||
servicer = self._servicer()
|
||||
with self.assertRaises(ValueError) as ctx:
|
||||
servicer._apply_engine_args({}, "not-json")
|
||||
self.assertIn("not valid JSON", str(ctx.exception))
|
||||
|
||||
def test_apply_engine_args_non_object_raises(self):
|
||||
servicer = self._servicer()
|
||||
with self.assertRaises(ValueError) as ctx:
|
||||
servicer._apply_engine_args({}, "[1,2,3]")
|
||||
self.assertIn("must be a JSON object", str(ctx.exception))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,12 +0,0 @@
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
backend_dir=$(dirname $0)
|
||||
|
||||
if [ -d $backend_dir/common ]; then
|
||||
source $backend_dir/common/libbackend.sh
|
||||
else
|
||||
source $backend_dir/../common/libbackend.sh
|
||||
fi
|
||||
|
||||
runUnittests
|
||||
@@ -3,5 +3,5 @@
|
||||
# on a cu130 host. Pull the cu130-flavoured wheel from vLLM's per-tag index
|
||||
# instead — the cublas13 case in install.sh adds --index-strategy=unsafe-best-match
|
||||
# so uv consults this index alongside PyPI.
|
||||
--extra-index-url https://wheels.vllm.ai/0.20.2/cu130
|
||||
vllm==0.20.2
|
||||
--extra-index-url https://wheels.vllm.ai/0.20.1/cu130
|
||||
vllm==0.20.1
|
||||
|
||||
9
backend/rust/kokoros/Cargo.lock
generated
9
backend/rust/kokoros/Cargo.lock
generated
@@ -1392,14 +1392,15 @@ checksum = "384b8ab6d37215f3c5301a95a4accb5d64aa607f1fcb26a11b5303878451b4fe"
|
||||
|
||||
[[package]]
|
||||
name = "openssl"
|
||||
version = "0.10.79"
|
||||
version = "0.10.76"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "bf0b434746ee2832f4f0baf10137e1cabb18cbe6912c69e2e33263c45250f542"
|
||||
checksum = "951c002c75e16ea2c65b8c7e4d3d51d5530d8dfa7d060b4776828c88cfb18ecf"
|
||||
dependencies = [
|
||||
"bitflags",
|
||||
"cfg-if",
|
||||
"foreign-types",
|
||||
"libc",
|
||||
"once_cell",
|
||||
"openssl-macros",
|
||||
"openssl-sys",
|
||||
]
|
||||
@@ -1423,9 +1424,9 @@ checksum = "7c87def4c32ab89d880effc9e097653c8da5d6ef28e6b539d313baaacfbafcbe"
|
||||
|
||||
[[package]]
|
||||
name = "openssl-sys"
|
||||
version = "0.9.115"
|
||||
version = "0.9.112"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "158fe5b292746440aa6e7a7e690e55aeb72d41505e2804c23c6973ad0e9c9781"
|
||||
checksum = "57d55af3b3e226502be1526dfdba67ab0e9c96fc293004e79576b2b9edb0dbdb"
|
||||
dependencies = [
|
||||
"cc",
|
||||
"libc",
|
||||
|
||||
@@ -40,7 +40,6 @@ type AudioTransformOutputs struct {
|
||||
// required; `referencePath` is optional (empty => backend zero-fills the
|
||||
// reference channel).
|
||||
func ModelAudioTransform(
|
||||
ctx context.Context,
|
||||
audioPath, referencePath string,
|
||||
opts AudioTransformOptions,
|
||||
loader *model.ModelLoader,
|
||||
@@ -82,7 +81,7 @@ func ModelAudioTransform(
|
||||
startTime = time.Now()
|
||||
}
|
||||
|
||||
res, err := transformModel.AudioTransform(ctx, &proto.AudioTransformRequest{
|
||||
res, err := transformModel.AudioTransform(context.Background(), &proto.AudioTransformRequest{
|
||||
AudioPath: audioPath,
|
||||
ReferencePath: referencePath,
|
||||
Dst: dst,
|
||||
|
||||
@@ -12,7 +12,6 @@ import (
|
||||
)
|
||||
|
||||
func Detection(
|
||||
ctx context.Context,
|
||||
sourceFile string,
|
||||
prompt string,
|
||||
points []float32,
|
||||
@@ -39,7 +38,7 @@ func Detection(
|
||||
startTime = time.Now()
|
||||
}
|
||||
|
||||
res, err := detectionModel.Detect(ctx, &proto.DetectOptions{
|
||||
res, err := detectionModel.Detect(context.Background(), &proto.DetectOptions{
|
||||
Src: sourceFile,
|
||||
Prompt: prompt,
|
||||
Points: points,
|
||||
|
||||
@@ -63,7 +63,7 @@ func loadDiarizationModel(ml *model.ModelLoader, modelConfig config.ModelConfig,
|
||||
|
||||
// ModelDiarization runs the Diarize RPC against the configured backend
|
||||
// and returns a normalized schema.DiarizationResult.
|
||||
func ModelDiarization(ctx context.Context, req DiarizationRequest, ml *model.ModelLoader, modelConfig config.ModelConfig, appConfig *config.ApplicationConfig) (*schema.DiarizationResult, error) {
|
||||
func ModelDiarization(req DiarizationRequest, ml *model.ModelLoader, modelConfig config.ModelConfig, appConfig *config.ApplicationConfig) (*schema.DiarizationResult, error) {
|
||||
m, err := loadDiarizationModel(ml, modelConfig, appConfig)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -74,7 +74,7 @@ func ModelDiarization(ctx context.Context, req DiarizationRequest, ml *model.Mod
|
||||
threads = uint32(*modelConfig.Threads)
|
||||
}
|
||||
|
||||
r, err := m.Diarize(ctx, req.toProto(threads))
|
||||
r, err := m.Diarize(context.Background(), req.toProto(threads))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
@@ -12,7 +12,6 @@ import (
|
||||
)
|
||||
|
||||
func FaceAnalyze(
|
||||
ctx context.Context,
|
||||
img string,
|
||||
actions []string,
|
||||
antiSpoofing bool,
|
||||
@@ -36,7 +35,7 @@ func FaceAnalyze(
|
||||
startTime = time.Now()
|
||||
}
|
||||
|
||||
res, err := faceModel.FaceAnalyze(ctx, &proto.FaceAnalyzeRequest{
|
||||
res, err := faceModel.FaceAnalyze(context.Background(), &proto.FaceAnalyzeRequest{
|
||||
Img: img,
|
||||
Actions: actions,
|
||||
AntiSpoofing: antiSpoofing,
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user