fix(tests): remove duplicate scheduling stubs

Assisted-by: Codex:gpt-5 [golangci-lint]

.agents/ai-coding-assistants.md

@@ -35,19 +35,33 @@ All contributions must comply with LocalAI's licensing requirements:
 
 ## Signed-off-by and Developer Certificate of Origin
 
-**AI agents MUST NOT add `Signed-off-by` tags.** Only humans can legally
-certify the Developer Certificate of Origin (DCO). The human submitter
-is responsible for:
+Only humans can certify the Developer Certificate of Origin (DCO). AI
+agents MUST NOT invent or guess a human identity for `Signed-off-by` —
+doing so forges the DCO certification.
 
-- Reviewing all AI-generated code
+However, when a human operator explicitly directs the AI to commit on
+their behalf, the AI is acting as a typing tool — no different from an
+editor macro or `git commit -s`. In that case the AI SHOULD add
+`Signed-off-by:` using the **configured `user.name` / `user.email`** of
+the current git repository (i.e. the operator's own identity). The
+resulting trailer is the operator's signature; they take responsibility
+for it by reviewing and pushing the commit. The AI MUST NOT use any
+other identity and MUST NOT add its own name to the sign-off.
+
+When running `git commit`, prefer `git commit --signoff` (or `-s`) so
+the trailer is emitted by git itself from the configured identity,
+rather than hand-writing it in a heredoc — this guarantees the sign-off
+matches whatever identity the operator is currently using.
+
+The human submitter remains responsible for:
+
+- Reviewing all AI-generated code before it's pushed or merged
 - Ensuring compliance with licensing requirements
-- Adding their own `Signed-off-by` tag (when the project requires DCO)
-  to certify the contribution
 - Taking full responsibility for the contribution
 
-AI agents MUST NOT add `Co-Authored-By` trailers for themselves either.
-A human reviewer owns the contribution; the AI's involvement is recorded
-via `Assisted-by` (see below).
+AI agents MUST NOT add `Co-Authored-By` trailers for themselves. A human
+reviewer owns the contribution; the AI's involvement is recorded via
+`Assisted-by` (see below).
 
 ## Attribution
 
@@ -84,6 +98,12 @@ Assisted-by: Claude:claude-opus-4-7 golangci-lint
 Signed-off-by: Jane Developer <jane@example.com>
 ```
 
+The `Signed-off-by` line uses Jane's own identity because Jane is the
+submitter operating the AI. If Jane asks Claude to create the commit via
+`git commit -s`, git emits that exact trailer from Jane's configured
+identity — no separate human step is needed beyond Jane reviewing the
+diff before pushing.
+
 ## Scope and Responsibility
 
 Using an AI assistant does not reduce the contributor's responsibility.

.agents/ci-caching.md

@@ -125,7 +125,7 @@ The per-backend prefix match only sees files under a backend's own directory, so
 | `backend/backend.proto` | nothing if the edit is additive-only, otherwise everything (see below) |
 | `backend/Dockerfile.<x>` | the Linux entries whose `dockerfile:` names it |
 | `backend/python/common/` | Python, Linux + Darwin |
-| `scripts/build/package-gpu-libs.sh` | Python, Linux only |
+| `scripts/build/package-gpu-libs.sh` | every Linux entry (Python, Go and C++ all run it) |
 | `scripts/build/<lang>-darwin.sh` | the Darwin entries that build target routes to |
 | `.github/workflows/backend_build[_darwin].yml` | everything on that OS |
 | anything else under `scripts/build/` (except `*_test.sh`) | everything — conservative default for unclassified packaging inputs |
@@ -359,6 +359,26 @@ GitHub Actions caches are limited to 10 GB per repo. Steady-state worst case: ~8
 
 One residual self-hosted reference remains in `test-extra.yml` (`tests-vibevoice-cpp-grpc-transcription` uses `bigger-runner` for the 30s JFK-decode timeout headroom). That's a separate concern.
 
+### Small always-on jobs routed to `arc-runner-set`
+
+The hosted pool is shared across the whole *account*, not per repo, so a burst in one repo starves the others. On 2026-07-31 it went to **zero scheduled jobs for 35 consecutive minutes** with 39 jobs queued, while `arc-runner-set` completed 12 jobs without interruption over the same window. Actions was healthy globally at the time (other public repos were scheduling normally), so this is an account-level throttle, not an outage.
+
+`gh-pages.yml` (`build` + `deploy`) is therefore routed to `arc-runner-set` when `github.repository == 'mudler/LocalAI'`. It needs no fork-safety clause because it only triggers on push-to-master and `workflow_dispatch`, so it never executes pull-request code. The repository guard keeps forks (which have no such runner label) from queueing forever. It fetches its own toolchains via `setup-go` / `actions-hugo` and uses no `sudo`/`apt`.
+
+#### What the `arc-runner-set` image actually contains
+
+Measured 2026-07-31 on run `30637392862` by a preflight step, not assumed:
+
+| present | **absent** |
+|---|---|
+| `git`, `curl`, `unzip`, `tar`, `ldd`, `python3` | **`make`**, **`gcc`** |
+
+That is why `lint.yml` is **not** on the self-hosted pool. Both of its jobs were routed there and both failed in one second: `golangci-lint` needs `make` (for `make protogen-go`, itself needing `curl`+`unzip` to fetch protoc, and for `make lint`), and `build-scripts` additionally needs a C toolchain because the packaging-script tests compile a throwaway binary and inspect it with `ldd`. Both jobs are back on `ubuntu-latest`.
+
+The preflight steps were deliberately left in place. They cost about a second on the hosted pool and mean that whenever the runner image gains `make` + `gcc`, re-routing is one `runs-on:` line per job and any remaining gap reports itself by name rather than as an opaque mid-build failure.
+
+Note for any future re-route: `lint.yml` also triggers on `pull_request`, and a fork PR runs untrusted contributor code. That must never reach a persistent self-hosted runner, so any re-route has to stay push-only, e.g. `${{ (github.event_name == 'push' && github.repository == 'mudler/LocalAI') && 'arc-runner-set' || 'ubuntu-latest' }}`.
+
 ## Touching the cache pipeline
 
 When changing `image_build.yml`, `backend_build.yml`, any of the `backend/Dockerfile.*` files, `Dockerfile.base-grpc-builder`, `.docker/install-base-deps.sh`, `.docker/<backend>-compile.sh`, or `scripts/changed-backends.js`:

.agents/preparing-a-release.md

@@ -0,0 +1,26 @@
+# Preparing a Release
+
+A release is not finished when the tag is pushed. The GitHub release, the blog post and the demo clips ship together, because the changelog says what moved and the post and the clips are what make anyone care.
+
+## What a release must include
+
+1. **Labels on the merged PRs.** GitHub generates the raw notes from PR labels, so label first, generate second. Wrong labels mean a miscategorised changelog that has to be edited by hand.
+2. **`RELEASE_NOTES_vX.Y.Z.md`** at the repository root, in the house style: what changed, why it matters, PR numbers so people can read the diffs.
+3. **A blog post under `website/content/blog/`.** One post per release, front matter with `title`, `date`, `author`, `category: "Release"`, `tags`, `summary` and `extracss: ["blog.css"]`. Cover the two or three changes that alter what a user does day to day, not the whole changelog, and link the PR numbers. See `website/content/blog/what-landed-in-localai-4-8.md` for the shape.
+4. **Demo clips for the notable features.** Anything visible (a new backend, a UI change, a new endpoint, a measured speedup) gets a short screen recording. Put the file in `website/static/media/`, reference it from the blog post, and reuse it on the marketing pages where it fits.
+
+A release without a post and without clips is incomplete, in the same way a user-facing code change without a docs update is incomplete.
+
+## Clip conventions
+
+- MP4, H.264, no audio track unless the feature is about audio. Keep them short (10 to 30 seconds) and loopable.
+- Record the real thing. A clip from the engine's own benchmark suite or a real session, never a mockup.
+- Where the change is a speedup, record both sides on the same machine on the same input, so the comparison is honest.
+- Name the file after the feature, not the release (`vllm-race.mp4`, not `v4-8-demo.mp4`), so it stays reusable once the release is old.
+- The marketing site plays clips with `muted loop playsinline preload="none"` and a `data-lazy` attribute, which the site's IntersectionObserver uses to play and pause them on scroll. Follow that pattern for anything you add.
+
+## Order of work
+
+Label the PRs, generate and edit the release notes, cut the draft release, record the clips while the branch is still fresh in your head, then write the post against the notes and the clips. Publishing the release and merging the post should happen on the same day.
+
+The `creating-localai-releases` skill drives steps 1 to 3 and captures the React UI screenshots that go into the notes.

.docker/install-base-deps.sh

@@ -113,6 +113,54 @@ if [ "${BUILD_TYPE:-}" = "vulkan" ] && [ "${SKIP_DRIVERS:-false}" = "false" ]; t
     rm -rf /var/lib/apt/lists/*
 fi
 
+# --- 2b. Intel graphics driver (BUILD_TYPE=sycl*) ---
+# The Intel oneAPI base image brings the compilers and the oneAPI libraries, but
+# not the driver that talks to the graphics card. The packaging step copies that
+# driver into the backend, so that the backend works on a machine which has no
+# Intel graphics packages of its own, for the same reason the Vulkan section
+# above installs the Mesa drivers. Install it here so there is something to copy.
+#
+# Only the sycl builds are covered, because those are the ones whose packaging
+# copies the driver. See package_intel_libs in scripts/build/package-gpu-libs.sh.
+#
+# The driver comes from Intel's own package repository, not from the Ubuntu
+# archive. The archive has 23.43 from late 2023, which does not know any card
+# released since, so a machine with a recent Intel GPU would end up carrying a
+# driver that cannot drive it. Intel's repository has 25.18 for the same Ubuntu
+# release.
+#
+# Anything that goes wrong here fails the build, on purpose. An unreachable
+# repository is a passing problem that a retry fixes, whereas carrying a
+# different driver than intended, or none, is a difference nobody would notice
+# until a user reports an idle GPU.
+if case "${BUILD_TYPE:-}" in sycl*) true;; *) false;; esac \
+    && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
+    # Ubuntu release name, which is what the repository is indexed by.
+    ubuntu_codename=$(. /etc/os-release && echo "${VERSION_CODENAME:-}")
+    if [ -z "$ubuntu_codename" ]; then
+        echo "ERROR: cannot tell which Ubuntu release this image is, so cannot pick the Intel driver repository" >&2
+        exit 1
+    fi
+
+    # The key is armored text, which apt reads directly from a .asc file, so
+    # there is no need for gnupg here. "unified" is the component Intel ships
+    # its current driver in.
+    mkdir -p /usr/share/keyrings
+    curl -fsSL https://repositories.intel.com/gpu/intel-graphics.key \
+        -o /usr/share/keyrings/intel-graphics.asc
+    echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.asc] https://repositories.intel.com/gpu/ubuntu ${ubuntu_codename} unified" \
+        > /etc/apt/sources.list.d/intel-graphics.list
+    apt-get update
+    # The first package holds the driver OpenCL talks to, the second the driver
+    # Level Zero talks to. Between them they pull in the compiler and the memory
+    # manager that both need.
+    apt-get install -y --no-install-recommends \
+        intel-opencl-icd \
+        libze-intel-gpu1
+    apt-get clean
+    rm -rf /var/lib/apt/lists/*
+fi
+
 # --- 3. CUDA toolkit (BUILD_TYPE=cublas|l4t) ---
 if { [ "${BUILD_TYPE:-}" = "cublas" ] || [ "${BUILD_TYPE:-}" = "l4t" ]; } && [ "${SKIP_DRIVERS:-false}" = "false" ]; then
     apt-get update

.docker/llama-cpp-build-target.sh

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+arch=${1:?target architecture is required}
+build_type=${2-}
+
+# SYCL compiles the whole tree with icpx -fsycl, and icpx never finishes
+# ggml-cpu/arch/x86/repack.cpp at -march=sapphirerapids: the job sits on that one
+# translation unit until GitHub kills it at 6h. gcc builds the same file in
+# seconds, so only the SYCL images have to give up the CPU variant matrix.
+case "$build_type" in
+  sycl*)
+    echo llama-cpp-fallback
+    exit 0
+    ;;
+esac
+
+# GPU arm64 base images do not consistently provide the gcc-14 toolchain needed
+# to compile ggml's armv9.2 CPU variants. Keep their portable fallback until the
+# builder images can supply that compiler.
+if [ "$arch" = "arm64" ] && [ -n "$build_type" ]; then
+  echo llama-cpp-fallback
+else
+  echo llama-cpp-cpu-all
+fi

.docker/llama-cpp-compile.sh

@@ -18,10 +18,12 @@ if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
 fi
 
 cd /LocalAI/backend/cpp/llama-cpp
-if [ -z "${BUILD_TYPE:-}" ]; then
-  # Pure CPU image (BUILD_TYPE empty): one build with ggml CPU_ALL_VARIANTS replaces the
-  # per-microarch binaries (x86: avx/avx2/avx512/fallback; arm64: armv8.x/armv9.x). ggml
-  # dlopens the best libggml-cpu-*.so at runtime by probing host CPU features.
+BUILD_TARGET=$(/LocalAI/.docker/llama-cpp-build-target.sh "${TARGETARCH}" "${BUILD_TYPE:-}")
+if [ "$BUILD_TARGET" = "llama-cpp-cpu-all" ]; then
+  # One build with ggml CPU_ALL_VARIANTS replaces the per-microarch binaries (x86:
+  # avx/avx2/avx512/fallback; arm64: armv8.x/armv9.x). BUILD_TYPE remains in the
+  # environment, so GPU builds retain their accelerator backend while ggml dlopens the
+  # best CPU library when work is offloaded to the host.
   #
   # arm64: the CPU_ALL_VARIANTS table includes armv9.2 SME variants whose -march=...+sme is
   # rejected by the Ubuntu 24.04 default gcc-13. gcc-14 accepts it, so build the arm64
@@ -35,14 +37,8 @@ if [ -z "${BUILD_TYPE:-}" ]; then
     apt-get update -qq && apt-get install -y -qq gcc-14 g++-14
     export CC=gcc-14 CXX=g++-14
   fi
-  make llama-cpp-cpu-all
-else
-  # GPU build (cublas/hipblas/sycl/vulkan/...): the accelerator does the compute, so a
-  # single fallback CPU build is enough - no per-microarch CPU variants needed. (This also
-  # keeps the heavy GPU backend compile from also building the whole CPU variant matrix,
-  # and avoids the gcc-14 apt step on GPU base images such as nvidia l4t.)
-  make llama-cpp-fallback
 fi
+make "$BUILD_TARGET"
 make llama-cpp-grpc
 make llama-cpp-rpc-server
 

.docker/turboquant-build-target.sh

@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+arch=${1:?target architecture is required}
+build_type=${2-}
+
+# SYCL compiles the whole tree with icpx -fsycl, and icpx never finishes
+# ggml-cpu/arch/x86/repack.cpp at -march=sapphirerapids: the job sits on that one
+# translation unit until GitHub kills it at 6h. gcc builds the same file in
+# seconds, so only the SYCL images have to give up the CPU variant matrix.
+case "$build_type" in
+  sycl*)
+    echo turboquant-fallback
+    exit 0
+    ;;
+esac
+
+# GPU arm64 base images do not consistently provide the gcc-14 toolchain needed
+# to compile ggml's armv9.2 CPU variants. Keep their portable fallback until the
+# builder images can supply that compiler.
+if [ "$arch" = "arm64" ] && [ -n "$build_type" ]; then
+  echo turboquant-fallback
+else
+  echo turboquant-cpu-all
+fi

.docker/turboquant-compile.sh

@@ -19,20 +19,18 @@ fi
 
 cd /LocalAI/backend/cpp/turboquant
 
-if [ -z "${BUILD_TYPE:-}" ]; then
-  # Pure CPU image: one ggml CPU_ALL_VARIANTS build replaces the per-microarch binaries.
+BUILD_TARGET=$(/LocalAI/.docker/turboquant-build-target.sh "${TARGETARCH}" "${BUILD_TYPE:-}")
+if [ "$BUILD_TARGET" = "turboquant-cpu-all" ]; then
+  # BUILD_TYPE remains in the environment, so GPU builds retain their accelerator while
+  # ggml selects the best CPU library when model work is offloaded to the host.
   # arm64: the armv9.2 SME variants need gcc-14 (gcc-13 rejects +sme).
   if [ "${TARGETARCH}" = "arm64" ]; then
+    sh /LocalAI/.docker/apt-mirror.sh || true
     apt-get update -qq && apt-get install -y -qq gcc-14 g++-14
     export CC=gcc-14 CXX=g++-14
   fi
-  make turboquant-cpu-all
-else
-  # GPU build (cublas/hipblas/sycl/vulkan/...): single fallback CPU build, the accelerator
-  # does the compute. Keeps the GPU compile from also building the CPU variant matrix and
-  # avoids the gcc-14 apt step on GPU base images such as nvidia l4t.
-  make turboquant-fallback
 fi
+make "$BUILD_TARGET"
 make turboquant-grpc
 make turboquant-rpc-server
 

.github/backend-matrix.yml

@@ -480,6 +480,22 @@ include:
     dockerfile: "./backend/Dockerfile.turboquant"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "12"
+    cuda-minor-version: "8"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-12-buun-llama-cpp'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-cuda-12-amd64'
+    # bigger-runner: same rationale as -gpu-nvidia-cuda-12-llama-cpp above
+    # (observed 6h5m wall-clock on v4.2.1, just past the 6h job timeout).
+    runs-on: 'bigger-runner'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "buun-llama-cpp"
+    dockerfile: "./backend/Dockerfile.buun-llama-cpp"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "8"
@@ -1165,6 +1181,21 @@ include:
     dockerfile: "./backend/Dockerfile.turboquant"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-nvidia-cuda-13-buun-llama-cpp'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-cuda-13-amd64'
+    # bigger-runner: observed 6h5m wall-clock on v4.2.1 — at the GHA timeout.
+    runs-on: 'bigger-runner'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "buun-llama-cpp"
+    dockerfile: "./backend/Dockerfile.buun-llama-cpp"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -1208,6 +1239,20 @@ include:
     backend: "turboquant"
     dockerfile: "./backend/Dockerfile.turboquant"
     context: "./"
+  - build-type: 'cublas'
+    cuda-major-version: "13"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    skip-drivers: 'false'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-cuda-13-arm64-buun-llama-cpp'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-cuda-13-arm64'
+    base-image: "ubuntu:24.04"
+    runs-on: 'ubuntu-24.04-arm'
+    ubuntu-version: '2404'
+    backend: "buun-llama-cpp"
+    dockerfile: "./backend/Dockerfile.buun-llama-cpp"
+    context: "./"
   - build-type: 'cublas'
     cuda-major-version: "13"
     cuda-minor-version: "0"
@@ -2477,6 +2522,20 @@ include:
     dockerfile: "./backend/Dockerfile.turboquant"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'sycl_f32'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-intel-sycl-f32-buun-llama-cpp'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-intel-amd64'
+    runs-on: 'ubuntu-latest'
+    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+    skip-drivers: 'false'
+    backend: "buun-llama-cpp"
+    dockerfile: "./backend/Dockerfile.buun-llama-cpp"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'sycl_f32'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -2519,6 +2578,20 @@ include:
     dockerfile: "./backend/Dockerfile.turboquant"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: 'sycl_f16'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-intel-sycl-f16-buun-llama-cpp'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-intel-amd64'
+    runs-on: 'ubuntu-latest'
+    base-image: "intel/oneapi-basekit:2025.3.0-0-devel-ubuntu24.04"
+    skip-drivers: 'false'
+    backend: "buun-llama-cpp"
+    dockerfile: "./backend/Dockerfile.buun-llama-cpp"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: 'sycl_f16'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -2985,6 +3058,21 @@ include:
     dockerfile: "./backend/Dockerfile.turboquant"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-buun-llama-cpp'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-amd64'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "buun-llama-cpp"
+    dockerfile: "./backend/Dockerfile.buun-llama-cpp"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -3015,6 +3103,21 @@ include:
     dockerfile: "./backend/Dockerfile.turboquant"
     context: "./"
     ubuntu-version: '2404'
+  - build-type: ''
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-cpu-buun-llama-cpp'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-arm64'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "buun-llama-cpp"
+    dockerfile: "./backend/Dockerfile.buun-llama-cpp"
+    context: "./"
+    ubuntu-version: '2404'
   - build-type: ''
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -3276,6 +3379,20 @@ include:
     dockerfile: "./backend/Dockerfile.turboquant"
     context: "./"
     ubuntu-version: '2204'
+  - build-type: 'cublas'
+    cuda-major-version: "12"
+    cuda-minor-version: "0"
+    platforms: 'linux/arm64'
+    skip-drivers: 'false'
+    tag-latest: 'auto'
+    tag-suffix: '-nvidia-l4t-arm64-buun-llama-cpp'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-l4t-cuda-12-arm64'
+    base-image: "nvcr.io/nvidia/l4t-jetpack:r36.4.0"
+    runs-on: 'ubuntu-24.04-arm'
+    backend: "buun-llama-cpp"
+    dockerfile: "./backend/Dockerfile.buun-llama-cpp"
+    context: "./"
+    ubuntu-version: '2204'
   - build-type: 'cublas'
     cuda-major-version: "12"
     cuda-minor-version: "0"
@@ -3336,6 +3453,22 @@ include:
     context: "./"
     ubuntu-version: '2404'
   # Stablediffusion-ggml
+  - build-type: 'vulkan'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/amd64'
+    platform-tag: 'amd64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-vulkan-buun-llama-cpp'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-vulkan-amd64'
+    runs-on: 'ubuntu-latest'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "buun-llama-cpp"
+    dockerfile: "./backend/Dockerfile.buun-llama-cpp"
+    context: "./"
+    ubuntu-version: '2404'
+  # Stablediffusion-ggml
   - build-type: 'vulkan'
     cuda-major-version: ""
     cuda-minor-version: ""
@@ -3368,6 +3501,22 @@ include:
     context: "./"
     ubuntu-version: '2404'
   # Stablediffusion-ggml
+  - build-type: 'vulkan'
+    cuda-major-version: ""
+    cuda-minor-version: ""
+    platforms: 'linux/arm64'
+    platform-tag: 'arm64'
+    tag-latest: 'auto'
+    tag-suffix: '-gpu-vulkan-buun-llama-cpp'
+    builder-base-image: 'quay.io/go-skynet/ci-cache:base-grpc-vulkan-arm64'
+    runs-on: 'ubuntu-24.04-arm'
+    base-image: "ubuntu:24.04"
+    skip-drivers: 'false'
+    backend: "buun-llama-cpp"
+    dockerfile: "./backend/Dockerfile.buun-llama-cpp"
+    context: "./"
+    ubuntu-version: '2404'
+  # Stablediffusion-ggml
   - build-type: 'vulkan'
     cuda-major-version: ""
     cuda-minor-version: ""

.github/ci/gen-redirects.sh

@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+#
+# Generate client-side redirects for the documentation URLs that used to live at
+# the site root.
+#
+# Until this site existed, the Hugo docs site WAS localai.io, so pages
+# were published at /features/..., /getting-started/..., /faq/ and so on. The
+# docs now build under /docs/, and GitHub Pages serves static files only: there
+# is no server-side rewrite, no .htaccess, no _redirects. The only way to keep
+# every published, bookmarked and search-indexed URL alive is to leave a real
+# HTML file at the old address that sends the browser to the new one.
+#
+# Anything the main site already publishes wins: it owns /, /engines/,
+# /blog/ and friends, so an existing file is never replaced.
+#
+# Usage: gen-redirects.sh <public-dir> [base-url]
+#   public-dir  merged output directory (main site with docs/ inside it)
+#   base-url    absolute or root-relative prefix the deployment is served from,
+#               trailing slash optional (default "/")
+
+set -euo pipefail
+
+PUBLIC_DIR=${1:?usage: gen-redirects.sh <public-dir> [base-url]}
+BASE_URL=${2:-/}
+
+# Normalise to exactly one trailing slash so concatenation below is predictable.
+BASE_URL="${BASE_URL%/}/"
+
+DOCS_DIR="${PUBLIC_DIR}/docs"
+
+if [ ! -d "$DOCS_DIR" ]; then
+  echo "gen-redirects: no docs output at ${DOCS_DIR}" >&2
+  exit 1
+fi
+
+created=0
+skipped=0
+
+# Every .html file is a reachable old URL, not just directory indexes: the
+# generated model gallery ships as a bare gallery.html and used to sit at the
+# root too.
+while IFS= read -r src; do
+  rel=${src#"$DOCS_DIR"/}
+  dst="${PUBLIC_DIR}/${rel}"
+
+  if [ -e "$dst" ]; then
+    skipped=$((skipped + 1))
+    continue
+  fi
+
+  # Link to the directory, not to its index.html, so the redirect target is the
+  # canonical URL the docs site itself advertises.
+  target="${BASE_URL}docs/${rel%index.html}"
+
+  mkdir -p "$(dirname "$dst")"
+  printf '%s' '<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>Moved</title>
+<link rel="canonical" href="'"$target"'">
+<meta name="robots" content="noindex">
+<meta http-equiv="refresh" content="0; url='"$target"'">
+</head>
+<body>
+<p>This page moved to <a href="'"$target"'">'"$target"'</a>.</p>
+</body>
+</html>
+' > "$dst"
+
+  created=$((created + 1))
+done <<EOF
+$(find "$DOCS_DIR" -type f -name '*.html' | sort)
+EOF
+
+echo "gen-redirects: ${created} redirect(s) written, ${skipped} path(s) left to the main site"

.github/ci/refresh-site-counters.sh

@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+# Refreshes the counters shown on the landing page from the GitHub API.
+#
+# The numbers used to be typed into the templates by hand, which meant they
+# only moved when somebody remembered, and a stale star count on the front
+# page is worse than no star count. Everything the API can answer for lives
+# in website/data/stats.yaml and is rewritten wholesale by this script.
+#
+# Anything the API cannot answer for (the Discord member count) is read back
+# out of the existing file and carried through untouched.
+set -euo pipefail
+
+REPO="${REPO:-mudler/LocalAI}"
+OUT="${OUT:-website/data/stats.yaml}"
+
+# The contributors and releases endpoints are paginated and never report a
+# total. Asking for one item per page makes the last page number equal to the
+# item count, which the Link header hands over.
+count_via_link_header() {
+  local path="$1" link last
+  link=$(gh api -i "${path}?per_page=1" 2>/dev/null | tr -d '\r' | grep -i '^link:' || true)
+  if [ -z "$link" ]; then
+    # No Link header means a single page, so count that page directly.
+    gh api "${path}?per_page=100" --jq 'length'
+    return
+  fi
+  last=$(sed -n 's/.*[?&]page=\([0-9]*\)>; rel="last".*/\1/p' <<<"$link")
+  [ -n "$last" ] || { gh api "${path}?per_page=100" --jq 'length'; return; }
+  printf '%s\n' "$last"
+}
+
+read -r stars forks < <(gh api "repos/${REPO}" --jq '"\(.stargazers_count) \(.forks_count)"')
+contributors=$(count_via_link_header "repos/${REPO}/contributors")
+releases=$(count_via_link_header "repos/${REPO}/releases")
+
+# Not derivable from the GitHub API, so keep whatever is already on disk.
+discord=$(sed -n 's/^discord: *\([0-9]*\).*/\1/p' "$OUT" 2>/dev/null | head -1)
+discord="${discord:-0}"
+
+for n in stars forks contributors releases; do
+  v="${!n}"
+  [[ "$v" =~ ^[0-9]+$ ]] && [ "$v" -gt 0 ] || {
+    echo "refusing to write: ${n} came back as '${v}'" >&2
+    exit 1
+  }
+done
+
+cat > "$OUT" <<YAML
+# Counters shown on the landing page.
+#
+# The four GitHub fields are rewritten by .github/ci/refresh-site-counters.sh,
+# which runs weekly from .github/workflows/refresh-site-counters.yml. Editing
+# them by hand works but will be overwritten on the next run.
+stars: ${stars}
+forks: ${forks}
+contributors: ${contributors}
+releases: ${releases}
+
+# The GitHub API cannot answer for this one, so it is maintained by hand and
+# the refresh script carries it through untouched.
+discord: ${discord}
+YAML
+
+echo "stars=${stars} forks=${forks} contributors=${contributors} releases=${releases} discord=${discord}"

.github/workflows/bump_deps.yaml

@@ -110,10 +110,14 @@ jobs:
             variable: "LOCATEANYTHING_VERSION"
             branch: "master"
             file: "backend/go/locate-anything-cpp/Makefile"
-          - repository: "ServeurpersoCom/qwentts.cpp"
-            variable: "QWEN3TTS_CPP_VERSION"
-            branch: "master"
-            file: "backend/go/qwen3-tts-cpp/Makefile"
+          # qwentts.cpp is held, not tracked: upstream master hangs in synthesis
+          # (see the comment on QWEN3TTS_CPP_VERSION in the backend Makefile).
+          # Leaving it here would re-bump the pin back onto the hang every night.
+          # Restore this entry once the upstream fix lands.
+          # - repository: "ServeurpersoCom/qwentts.cpp"
+          #   variable: "QWEN3TTS_CPP_VERSION"
+          #   branch: "master"
+          #   file: "backend/go/qwen3-tts-cpp/Makefile"
           - repository: "ServeurpersoCom/omnivoice.cpp"
             variable: "OMNIVOICE_VERSION"
             branch: "master"

.github/workflows/gh-pages.yml

@@ -1,4 +1,4 @@
-name: Deploy docs to GitHub Pages
+name: Deploy site to GitHub Pages
 
 on:
   push:
@@ -6,9 +6,11 @@ on:
       - master
     paths:
       - 'docs/**'
+      - 'website/**'
       - 'gallery/**'
       - 'images/**'
       - '.github/ci/modelslist.go'
+      - '.github/ci/gen-redirects.sh'
       - '.github/workflows/gh-pages.yml'
   workflow_dispatch:
 
@@ -23,7 +25,20 @@ concurrency:
 
 jobs:
   build:
-    runs-on: ubuntu-latest
+    # Self-hosted. This workflow is push-to-master + workflow_dispatch only, so
+    # it never executes pull-request code and a fork cannot reach the runner
+    # with untrusted changes. The repository guard keeps forks (whose own master
+    # pushes would otherwise queue forever against a label they do not have) on
+    # the hosted pool.
+    #
+    # Why: the GitHub-hosted pool is shared account-wide and has repeatedly
+    # starved (2026-07-31: 35 consecutive minutes at zero scheduled jobs, while
+    # arc-runner-set kept completing work throughout). Publishing the site is
+    # small, frequent, and must not sit behind a saturated hosted queue.
+    #
+    # Needs only git, tar and curl on the runner: setup-go and actions-hugo
+    # fetch their own toolchains, and no step uses sudo, apt, make or unzip.
+    runs-on: ${{ github.repository == 'mudler/LocalAI' && 'arc-runner-set' || 'ubuntu-latest' }}
     env:
       HUGO_VERSION: "0.146.3"
     steps:
@@ -36,7 +51,16 @@ jobs:
       - name: Setup Go
         uses: actions/setup-go@v5
         with:
-          go-version: '1.22'
+          # Track go.mod rather than a literal. Pinned at 1.22 this installed a
+          # toolchain older than the module's `go 1.26.0`, so the `go run` below
+          # downloaded the real one from proxy.golang.org on every run. That
+          # fetch is not always reachable from the runner and the deploy failed
+          # on five of eight consecutive master pushes with:
+          #   go: download go1.26.0: ... connect: network is unreachable
+          #   ##[error]Command failed: go env GOPATH
+          # Installing the version the module asks for removes the download
+          # instead of depending on it succeeding.
+          go-version-file: go.mod
           cache: false
 
       - name: Setup Hugo
@@ -49,25 +73,46 @@ jobs:
         id: pages
         uses: actions/configure-pages@v6
 
+      # The gallery page is generated from the model index and shipped as a
+      # static asset of the docs site, so it has to exist before Hugo runs.
       - name: Generate gallery
         run: go run ./.github/ci/modelslist.go ./gallery/index.yaml > docs/static/gallery.html
 
-      - name: Build site
+      # Two Hugo sites, one Pages artifact: the main site owns the root,
+      # the docs site is nested under /docs/.
+      - name: Build the main site
+        working-directory: website
+        run: hugo --minify --baseURL "${{ steps.pages.outputs.base_url }}/"
+
+      - name: Build documentation site
         working-directory: docs
         run: |
           mkdir -p layouts/_default
-          hugo --minify --baseURL "${{ steps.pages.outputs.base_url }}/"
+          hugo --minify --baseURL "${{ steps.pages.outputs.base_url }}/docs/"
+
+      - name: Merge documentation into the main site
+        run: |
+          mkdir -p website/public/docs
+          cp -R docs/public/. website/public/docs/
+
+      # Keeps the pre-split URLs alive; see the script header.
+      - name: Generate legacy URL redirects
+        run: .github/ci/gen-redirects.sh website/public "${{ steps.pages.outputs.base_url }}/"
 
       - name: Upload artifact
         uses: actions/upload-pages-artifact@v5
         with:
-          path: docs/public
+          path: website/public
 
   deploy:
     environment:
       name: github-pages
       url: ${{ steps.deployment.outputs.page_url }}
-    runs-on: ubuntu-latest
+    # Same routing as build: a hosted slot for a ~10s deploy is exactly the kind
+    # of job that should not block on a starved pool. deploy-pages authenticates
+    # with the job's OIDC token (id-token: write above), which self-hosted
+    # runners issue the same way hosted ones do.
+    runs-on: ${{ github.repository == 'mudler/LocalAI' && 'arc-runner-set' || 'ubuntu-latest' }}
     needs: build
     steps:
       - name: Deploy to GitHub Pages

.github/workflows/lint.yml

@@ -21,8 +21,41 @@ concurrency:
 
 jobs:
   golangci-lint:
+    # Self-hosted for PUSH only, and only in the canonical repo.
+    #
+    # This workflow also runs on pull_request, which for a fork PR means
+    # executing untrusted contributor code. That must never land on a
+    # self-hosted runner, so anything that is not a push to mudler/LocalAI stays
+    # on the ephemeral hosted pool. Pushes to master are trusted code that has
+    # already been reviewed and merged.
+    #
+    # Why at all: the hosted pool is shared account-wide and starved for 35
+    # straight minutes on 2026-07-31 while arc-runner-set kept completing jobs.
+    # Lint is small and runs on every commit, so it is a good candidate to move
+    # off the contended pool.
+    # REVERTED to hosted: the arc-runner-set image has git, curl, unzip, tar,
+    # ldd and python3, but NOT make (nor gcc). Measured on run 30637392862,
+    # where the preflight below named both. Re-route here once the runner image
+    # ships a C toolchain and make; the preflight stays so the next attempt
+    # fails by name in one second instead of opaquely mid-build.
     runs-on: ubuntu-latest
     steps:
+      - name: Preflight - required host tools
+        # The hosted images ship these; a self-hosted container image may not.
+        # Check up front so a missing tool reports itself by name instead of
+        # surfacing as an opaque failure inside `make protogen-go` (which needs
+        # curl + unzip for protoc) or `make lint`.
+        run: |
+          missing=""
+          for t in git curl unzip make tar; do
+            command -v "$t" >/dev/null 2>&1 || missing="$missing $t"
+          done
+          echo "runner: ${RUNNER_NAME:-unknown}  os: $(uname -sm)"
+          if [ -n "$missing" ]; then
+            echo "::error::missing required tools on this runner:$missing"
+            exit 1
+          fi
+          echo "all required tools present"
       - uses: actions/checkout@v7
         with:
           # Full history so golangci-lint's new-from-merge-base can reach
@@ -55,8 +88,30 @@ jobs:
     # container build (a missing transitive dep, a partial cuDNN family). Their
     # shell tests need nothing but bash + gcc + ldd, so run them on every PR
     # rather than waiting on a multi-GB cross-arch backend image build.
+    #
+    # Push-only self-hosted routing, same fork-safety reasoning as
+    # golangci-lint above.
+    # REVERTED to hosted: the arc-runner-set image has git, curl, unzip, tar,
+    # ldd and python3, but NOT make (nor gcc). Measured on run 30637392862,
+    # where the preflight below named both. Re-route here once the runner image
+    # ships a C toolchain and make; the preflight stays so the next attempt
+    # fails by name in one second instead of opaquely mid-build.
     runs-on: ubuntu-latest
     steps:
+      - name: Preflight - required host tools
+        # This job additionally needs a C toolchain: the packaging-script tests
+        # compile a throwaway binary and inspect it with ldd.
+        run: |
+          missing=""
+          for t in git make gcc ldd python3; do
+            command -v "$t" >/dev/null 2>&1 || missing="$missing $t"
+          done
+          echo "runner: ${RUNNER_NAME:-unknown}  os: $(uname -sm)"
+          if [ -n "$missing" ]; then
+            echo "::error::missing required tools on this runner:$missing"
+            exit 1
+          fi
+          echo "all required tools present"
       - uses: actions/checkout@v7
       - name: run packaging script tests
         run: make test-build-scripts

.github/workflows/refresh-site-counters.yml

@@ -0,0 +1,44 @@
+name: Refresh site counters
+
+# The landing page shows a star count, a contributor count and a release
+# count. They were typed in by hand, so they drifted the moment somebody
+# forgot. This pulls the real numbers once a week and commits them only when
+# they have actually moved, which in turn triggers the usual Pages deploy.
+
+on:
+  schedule:
+    # Mondays, 06:17 UTC. Off the hour on purpose, since the scheduler queues
+    # everything that asks for :00 and drops what it cannot run.
+    - cron: '17 6 * * 1'
+  workflow_dispatch:
+
+permissions:
+  contents: write
+
+concurrency:
+  group: refresh-site-counters
+  cancel-in-progress: false
+
+jobs:
+  refresh:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Read the counts off the GitHub API
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: ./.github/ci/refresh-site-counters.sh
+
+      - name: Commit only if something moved
+        run: |
+          if git diff --quiet -- website/data/stats.yaml; then
+            echo "counters unchanged, nothing to commit"
+            exit 0
+          fi
+          git diff --unified=0 -- website/data/stats.yaml
+          git config user.name  "github-actions[bot]"
+          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
+          git add website/data/stats.yaml
+          git commit -m "chore(website): refresh the counters"
+          git push

.github/workflows/test-extra.yml

@@ -33,6 +33,7 @@ jobs:
       llama-cpp: ${{ steps.detect.outputs.llama-cpp }}
       ik-llama-cpp: ${{ steps.detect.outputs.ik-llama-cpp }}
       turboquant: ${{ steps.detect.outputs.turboquant }}
+      buun-llama-cpp: ${{ steps.detect.outputs['buun-llama-cpp'] }}
       vllm: ${{ steps.detect.outputs.vllm }}
       sglang: ${{ steps.detect.outputs.sglang }}
       acestep-cpp: ${{ steps.detect.outputs.acestep-cpp }}
@@ -716,6 +717,30 @@ jobs:
       - name: Build turboquant backend image and run gRPC e2e tests
         run: |
           make test-extra-backend-turboquant
+  tests-buun-llama-cpp-grpc:
+    needs: detect-changes
+    if: needs.detect-changes.outputs['buun-llama-cpp'] == 'true' || needs.detect-changes.outputs.run-all == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 90
+    steps:
+      - name: Clone
+        uses: actions/checkout@v6
+        with:
+          submodules: true
+      - name: Setup Go
+        uses: actions/setup-go@v5
+        with:
+          go-version: '1.25.4'
+      # Exercises the buun-llama-cpp (fork-of-a-fork) backend with the
+      # fork-specific TurboQuant/TCQ KV-cache types. BACKEND_TEST_CACHE_TYPE_V
+      # is set to turbo3 so the test round-trips through the fork's KV
+      # allow-list — picking a stock llama.cpp type would only re-test the
+      # shared code path. DFlash speculative decoding is not exercised here
+      # because the one known public target/drafter pair (Qwen3.5-27B) is too
+      # large for CI.
+      - name: Build buun-llama-cpp backend image and run gRPC e2e tests
+        run: |
+          make test-extra-backend-buun-llama-cpp
   # tests-vllm-grpc is currently disabled in CI.
   #
   # The prebuilt vllm CPU wheel is compiled with AVX-512 VNNI/BF16

.gitignore

@@ -62,6 +62,11 @@ prepare
 /ggml-metal.metal
 docs/static/gallery.html
 
+# Hugo build output and lock files (docs/ and website/)
+docs/public/
+website/public/
+.hugo_build.lock
+
 # Protobuf generated files
 *.pb.go
 *pb2.py
@@ -119,3 +124,8 @@ formal-verification/out/
 # package directory itself and untrack the source.
 /apexentries
 /.github/ci/apexentries/apexentries
+
+# Runtime state written by `local-ai run` when it is started from the repo
+# root, which is what a contributor testing a build does. Nothing under here is
+# source: it is the instance's own models, outputs, traces and identity.
+/data/

ADOPTERS.md

@@ -0,0 +1,48 @@
+# Adopters
+
+Organisations running LocalAI, listed by the people who run it.
+
+If your organisation uses LocalAI and you are happy to say so publicly, open a
+pull request adding a row to the table below. That pull request is how we know
+we have permission to list you, which is why we do not add anybody ourselves.
+
+You do not need to be a large company, and you do not need to disclose anything
+sensitive. A sentence on what you use it for is more useful to other readers
+than a logo.
+
+## How to add yourself
+
+1. Add a row to the table, in alphabetical order.
+2. Use your organisation's usual name and a link to your site.
+3. Say briefly what you use LocalAI for, and whether it is in production.
+4. Open the pull request from an account that makes it plausible you speak for
+   the organisation, or say in the description who you are. We may ask.
+
+To be removed, open a pull request deleting your row, or email
+[info@localai.io](mailto:info@localai.io). We will not ask why.
+
+## Who is using LocalAI
+
+<!-- Keep alphabetical. Columns: Organisation | What for | Status -->
+
+| Organisation | What they use it for | Status |
+|---|---|---|
+| _Your organisation here_ | | |
+
+## What this list is not
+
+This is not a list of everyone who has ever starred the repository, and it is
+not a list of the employers of people who have contributed a patch. Both of
+those are easy to scrape and neither means what a logo wall implies.
+
+The website shows two separate things, both of which are checkable without
+anybody's permission:
+
+- **Engineers from these companies have contributed code.** Evidence is the
+  commit history plus the employer on that person's public GitHub profile. It
+  is a claim about a person, not about their employer.
+- **These projects integrate LocalAI.** Evidence is a reference to LocalAI in
+  that project's own repository or documentation.
+
+Those two lists live in [`website/data/ecosystem.yaml`](website/data/ecosystem.yaml).
+This file is the third, stronger thing: organisations that chose to say so.

AGENTS.md

@@ -32,6 +32,7 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
 | [.agents/adding-gallery-models.md](.agents/adding-gallery-models.md) | Adding GGUF models from HuggingFace to the model gallery |
 | [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) | LocalAI Assistant chat modality — adding admin tools to the in-process MCP server, editing skill prompts, keeping REST + MCP + skills in sync |
 | [.agents/backend-signing.md](.agents/backend-signing.md) | Backend OCI image signing (keyless cosign + sigstore-go) — producer-side CI setup, consumer-side gallery `verification:` block, strict mode (`LOCALAI_REQUIRE_BACKEND_INTEGRITY`), revocation via `not_before` |
+| [.agents/preparing-a-release.md](.agents/preparing-a-release.md) | Cutting a release: PR labels, `RELEASE_NOTES_vX.Y.Z.md`, the blog post under `website/content/blog/`, and the demo clips under `website/static/media/` |
 
 ## Quick Reference
 
@@ -42,6 +43,7 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
 - **Docs (docs-with-code rule)**: When you change user-facing behavior (API endpoints, CLI flags, config keys, or features), update the corresponding page under `docs/content/` in the SAME change, not as a follow-up. A user-facing change without a matching docs update is incomplete. See also the documentation conventions in [.agents/coding-style.md](.agents/coding-style.md).
 - **New API endpoints**: LocalAI advertises its capability surface in several independent places — swagger `@Tags`, `/api/instructions` registry, auth `RouteFeatureRegistry`, React UI `capabilities.js`, docs. Read [.agents/api-endpoints-and-auth.md](.agents/api-endpoints-and-auth.md) and follow its checklist — missing any surface means clients, admins, and the UI won't know the endpoint exists.
 - **Admin endpoints → MCP tool**: every admin endpoint that an admin would manage conversationally (install/list/edit/toggle/upgrade) MUST also be exposed as an MCP tool in `pkg/mcp/localaitools/`. The LocalAI Assistant chat modality and the standalone `local-ai mcp-server` consume that package; drift between REST and MCP is a real risk. Read [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) — the `TestToolHTTPRouteMappingComplete` test fails until you wire the new tool and update the route map.
+- **Releases ship with a post and clips**: a release is not done at the tag. It needs labelled PRs, `RELEASE_NOTES_vX.Y.Z.md`, a blog post under `website/content/blog/`, and a short demo clip in `website/static/media/` for each notable feature. See [.agents/preparing-a-release.md](.agents/preparing-a-release.md).
 - **Build**: Inspect `Makefile` and `.github/workflows/` — ask the user before running long builds
 - **Backend OS coverage**: a new backend must target every OS it can build for, not just Linux. `.github/backend-matrix.yml` has two matrices — `include:` (Linux) and `includeDarwin:` (macOS / Apple Silicon). Most C/C++/GGML and many Python backends build on Darwin too — wire the `includeDarwin` entry + `backend/index.yaml` `metal:` entries, or say in the PR why an OS is unsupported. See the darwin checklist in [.agents/adding-backends.md](.agents/adding-backends.md).
 - **Gallery variant ranking**: a gallery entry can declare `variants` (alternative builds of the same weights), and LocalAI ranks the ones a host can run by engine preference first, size second. A new backend that should be preferred on some hardware must be listed in `engineNamePreferenceRules` in `pkg/system/capabilities.go`; the sibling `backendBuildTagPreferenceRules` speaks build tags rather than engine names, and using the wrong table matches nothing without erroring. See [.agents/adding-backends.md](.agents/adding-backends.md).

Makefile

@@ -1,4 +1,5 @@
 # Disable parallel execution for backend builds
+.NOTPARALLEL: backends/buun-llama-cpp
 .NOTPARALLEL: backends/diffusers backends/llama-cpp backends/turboquant backends/bonsai backends/outetts backends/piper backends/stablediffusion-ggml backends/trellis2cpp backends/trellis2cpp-darwin backends/whisper backends/crispasr backends/parakeet-cpp backends/moss-transcribe-cpp backends/faster-whisper backends/silero-vad backends/local-store backends/valkey-store backends/cloud-proxy backends/huggingface backends/rfdetr backends/rfdetr-cpp backends/insightface backends/speaker-recognition backends/kitten-tts backends/kokoro backends/chatterbox backends/llama-cpp-darwin backends/neutts build-darwin-python-backend build-darwin-go-backend backends/mlx backends/diffuser-darwin backends/mlx-vlm backends/mlx-audio backends/mlx-distributed backends/stablediffusion-ggml-darwin backends/vllm backends/vllm-omni backends/longcat-video backends/sglang backends/moonshine backends/pocket-tts backends/qwen-tts backends/faster-qwen3-tts backends/qwen-asr backends/nemo backends/voxcpm backends/whisperx backends/ace-step backends/acestep-cpp backends/fish-speech backends/voxtral backends/opus backends/trl backends/llama-cpp-quantization backends/kokoros backends/sam3-cpp backends/qwen3-tts-cpp backends/moss-tts-cpp backends/magpie-tts-cpp backends/vllm-cpp backends/omnivoice-cpp backends/vibevoice-cpp backends/localvqe backends/tinygrad backends/sherpa-onnx backends/ds4 backends/ds4-darwin backends/liquid-audio backends/supertonic backends/depth-anything-cpp backends/privacy-filter backends/privacy-filter-darwin backends/audio-cpp backends/audio-cpp-darwin
 
 GOCMD=go
@@ -748,6 +749,19 @@ test-extra-backend-bonsai: docker-build-bonsai
 	BACKEND_TEST_MODEL_URL=https://huggingface.co/prism-ml/Bonsai-8B-gguf/resolve/main/Bonsai-8B-Q1_0.gguf \
 	$(MAKE) test-extra-backend
 
+## buun-llama-cpp: exercises the fork-of-a-fork backend (spiritbuun/buun-llama-cpp)
+## with the *TurboQuant/TCQ-specific* KV-cache types (turbo3 for V). Same rationale
+## as turboquant above: picking a standard llama.cpp type would only re-test the
+## shared code path. buun inherits turboquant's turbo2/turbo3/turbo4 and adds
+## turbo2_tcq / turbo3_tcq on top. DFlash speculative decoding is not exercised
+## here because no small DFlash drafter model exists (the known public pair is
+## Qwen3.5-27B, ~54 GB).
+test-extra-backend-buun-llama-cpp: docker-build-buun-llama-cpp
+	BACKEND_IMAGE=local-ai-backend:buun-llama-cpp \
+	BACKEND_TEST_CACHE_TYPE_K=q8_0 \
+	BACKEND_TEST_CACHE_TYPE_V=turbo3 \
+	$(MAKE) test-extra-backend
+
 ## Audio transcription wrapper for the llama-cpp backend.
 ## Drives the new AudioTranscription / AudioTranscriptionStream RPCs against
 ## ggml-org/Qwen3-ASR-0.6B-GGUF (a small ASR model that requires its mmproj
@@ -1284,6 +1298,11 @@ BACKEND_PRIVACY_FILTER = privacy-filter|privacy-filter|.|false|false
 # against apt gRPC/protobuf rather than a prebuilt base-grpc image; the reason
 # is on the audio-cpp block in .github/backend-matrix.yml.
 BACKEND_AUDIO_CPP = audio-cpp|audio-cpp|.|false|false
+# buun-llama-cpp is a fork-of-a-fork (spiritbuun/buun-llama-cpp forks
+# TheTom/llama-cpp-turboquant) that adds DFlash block-diffusion speculative
+# decoding and extra TCQ KV-cache variants on top of TurboQuant. Same thin
+# wrapper pattern as turboquant — reuses backend/cpp/llama-cpp grpc-server.
+BACKEND_BUUN_LLAMA_CPP = buun-llama-cpp|buun-llama-cpp|.|false|false
 
 # Golang backends
 BACKEND_PIPER = piper|golang|.|false|true
@@ -1388,6 +1407,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_BONSAI)))
 $(eval $(call generate-docker-build-target,$(BACKEND_DS4)))
 $(eval $(call generate-docker-build-target,$(BACKEND_PRIVACY_FILTER)))
 $(eval $(call generate-docker-build-target,$(BACKEND_AUDIO_CPP)))
+$(eval $(call generate-docker-build-target,$(BACKEND_BUUN_LLAMA_CPP)))
 $(eval $(call generate-docker-build-target,$(BACKEND_PIPER)))
 $(eval $(call generate-docker-build-target,$(BACKEND_LOCAL_STORE)))
 $(eval $(call generate-docker-build-target,$(BACKEND_VALKEY_STORE)))
@@ -1456,7 +1476,7 @@ $(eval $(call generate-docker-build-target,$(BACKEND_SUPERTONIC)))
 docker-save-%: backend-images
 	docker save local-ai-backend:$* -o backend-images/$*.tar
 
-docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-magpie-tts-cpp docker-build-vllm-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-privacy-filter docker-build-trellis2cpp docker-build-valkey-store docker-build-audio-cpp
+docker-build-backends: docker-build-llama-cpp docker-build-ik-llama-cpp docker-build-turboquant docker-build-buun-llama-cpp docker-build-bonsai docker-build-ds4 docker-build-rerankers docker-build-vllm docker-build-vllm-omni docker-build-longcat-video docker-build-sglang docker-build-transformers docker-build-outetts docker-build-diffusers docker-build-kokoro docker-build-faster-whisper docker-build-crispasr docker-build-coqui docker-build-chatterbox docker-build-vibevoice docker-build-liquid-audio docker-build-moonshine docker-build-pocket-tts docker-build-qwen-tts docker-build-fish-speech docker-build-faster-qwen3-tts docker-build-qwen-asr docker-build-nemo docker-build-voxcpm docker-build-whisperx docker-build-ace-step docker-build-acestep-cpp docker-build-voxtral docker-build-mlx-distributed docker-build-trl docker-build-llama-cpp-quantization docker-build-tinygrad docker-build-kokoros docker-build-sam3-cpp docker-build-rfdetr-cpp docker-build-qwen3-tts-cpp docker-build-moss-tts-cpp docker-build-magpie-tts-cpp docker-build-vllm-cpp docker-build-omnivoice-cpp docker-build-vibevoice-cpp docker-build-localvqe docker-build-insightface docker-build-speaker-recognition docker-build-sherpa-onnx docker-build-cloud-proxy docker-build-supertonic docker-build-depth-anything-cpp docker-build-moss-transcribe-cpp docker-build-privacy-filter docker-build-trellis2cpp docker-build-valkey-store docker-build-audio-cpp
 
 ########################################################
 ### Mock Backend for E2E Tests
@@ -1548,7 +1568,12 @@ swagger:
 gen-assets:
 	$(GOCMD) run core/dependencies_manager/manager.go webui_static.yaml core/http/static/assets
 
-## Documentation
+## Documentation and website
+# The published site is two Hugo sites: website/ owns the root, docs/ is nested
+# under /docs/. Serve them separately while editing; use `make site` to get the
+# merged tree (including the legacy URL redirects) that GitHub Pages deploys.
+SITE_BASE_URL?=http://localhost:8000
+
 docs/layouts/_default:
 	mkdir -p docs/layouts/_default
 
@@ -1560,12 +1585,30 @@ docs/public: docs/layouts/_default docs/static/gallery.html
 
 docs-clean:
 	rm -rf docs/public
+	rm -rf website/public
 	rm -rf docs/static/gallery.html
 
 .PHONY: docs
 docs: docs/static/gallery.html
 	cd docs && hugo serve
 
+.PHONY: website
+website:
+	cd website && hugo serve
+
+.PHONY: site
+site: docs/static/gallery.html
+	rm -rf website/public docs/public
+	cd website && hugo --minify --baseURL "$(SITE_BASE_URL)/"
+	cd docs && hugo --minify --baseURL "$(SITE_BASE_URL)/docs/"
+	mkdir -p website/public/docs
+	cp -R docs/public/. website/public/docs/
+	./.github/ci/gen-redirects.sh website/public "$(SITE_BASE_URL)/"
+
+.PHONY: site-serve
+site-serve: site
+	cd website/public && python3 -m http.server 8000
+
 ########################################################
 ## Platform-specific builds
 ########################################################

README.md

@@ -161,7 +161,7 @@ local-ai run https://gist.githubusercontent.com/.../phi-2.yaml
 local-ai run oci://localai/phi-2:latest
 ```
 
-To test a running LocalAI server from the terminal, open an interactive chat session from another shell. Inside the prompt, `/models` lists installed models and `/model <name>` switches between them.
+To work with a running LocalAI server from the terminal, start the built-in agent from another shell. It answers questions, reads your files and runs commands on your machine, asking you to approve anything that changes state. Inside a session, `/models` lists installed models and `/model <name>` switches between them. See the [Terminal agent](https://localai.io/docs/features/terminal-agent/) docs.
 
 ```bash
 # Terminal 1
@@ -238,7 +238,7 @@ Most backends wrap a best-in-class upstream engine. A handful of them are native
 | [magpie-tts.cpp](https://github.com/mudler/magpie-tts.cpp) | C++/GGML port of NVIDIA's Magpie TTS Multilingual 357M: 22.05 kHz mono text-to-speech in 5 voices and 9+ languages, with the NanoCodec neural codec and tokenizer/G2P embedded in a single GGUF |
 | [ced.cpp](https://github.com/localai-org/ced.cpp) | C++/GGML port of the CED audio-tagging models: sound-event classification (527-class AudioSet) over REST and the realtime API for live recognition |
 | [voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp) | Speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion), replacing the Python speaker-recognition backend |
-| [voxtral-tts.c](https://github.com/mudler/voxtral-tts.c) | Voxtral Realtime 4B speech-to-text in pure C |
+| [voxtral-tts.c](https://github.com/mudler/voxtral-tts.c) | Mistral Voxtral-4B-TTS text-to-speech in pure C: 20 preset voices across 9 languages, 24 kHz WAV output, no dependencies beyond libc |
 | [vibevoice.cpp](https://github.com/mudler/vibevoice.cpp) | Native port of Microsoft VibeVoice for TTS (voice cloning) and long-form ASR with speaker diarization |
 | [rf-detr.cpp](https://github.com/localai-org/rf-detr.cpp) | Native RF-DETR object detection and instance segmentation |
 | [locate-anything.cpp](https://github.com/mudler/locate-anything.cpp) | Open-vocabulary object detection and visual grounding (LocateAnything-3B) |

backend/Dockerfile.buun-llama-cpp

@@ -0,0 +1,290 @@
+ARG BASE_IMAGE=ubuntu:24.04
+ARG GRPC_BASE_IMAGE=${BASE_IMAGE}
+
+
+# The grpc target does one thing, it builds and installs GRPC.  This is in it's own layer so that it can be effectively cached by CI.
+# You probably don't need to change anything here, and if you do, make sure that CI is adjusted so that the cache continues to work.
+FROM ${GRPC_BASE_IMAGE} AS grpc
+
+# This is a bit of a hack, but it's required in order to be able to effectively cache this layer in CI
+ARG GRPC_MAKEFLAGS="-j4 -Otarget"
+ARG GRPC_VERSION=v1.65.0
+ARG CMAKE_FROM_SOURCE=false
+# CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
+ARG CMAKE_VERSION=3.31.10
+
+ENV MAKEFLAGS=${GRPC_MAKEFLAGS}
+
+WORKDIR /build
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+        ca-certificates \
+        build-essential curl libssl-dev \
+        git wget && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+# Install CMake (the version in 22.04 is too old)
+RUN <<EOT bash
+    if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
+        curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
+    else
+        apt-get update && \
+        apt-get install -y \
+            cmake && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
+# We install GRPC to a different prefix here so that we can copy in only the build artifacts later
+# saves several hundred MB on the final docker image size vs copying in the entire GRPC source tree
+# and running make install in the target container
+RUN git clone --recurse-submodules --jobs 4 -b ${GRPC_VERSION} --depth 1 --shallow-submodules https://github.com/grpc/grpc && \
+    mkdir -p /build/grpc/cmake/build && \
+    cd /build/grpc/cmake/build && \
+    sed -i "216i\  TESTONLY" "../../third_party/abseil-cpp/absl/container/CMakeLists.txt" && \
+    cmake -DgRPC_INSTALL=ON -DgRPC_BUILD_TESTS=OFF -DCMAKE_INSTALL_PREFIX:PATH=/opt/grpc ../.. && \
+    make && \
+    make install && \
+    rm -rf /build
+
+FROM ${BASE_IMAGE} AS builder
+ARG CMAKE_FROM_SOURCE=false
+ARG CMAKE_VERSION=3.31.10
+# We can target specific CUDA ARCHITECTURES like --build-arg CUDA_DOCKER_ARCH='75;86;89;120'
+ARG CUDA_DOCKER_ARCH
+ENV CUDA_DOCKER_ARCH=${CUDA_DOCKER_ARCH}
+ARG CMAKE_ARGS
+ENV CMAKE_ARGS=${CMAKE_ARGS}
+ARG BACKEND=rerankers
+ARG BUILD_TYPE
+ENV BUILD_TYPE=${BUILD_TYPE}
+ARG CUDA_MAJOR_VERSION
+ARG CUDA_MINOR_VERSION
+ARG SKIP_DRIVERS=false
+ENV CUDA_MAJOR_VERSION=${CUDA_MAJOR_VERSION}
+ENV CUDA_MINOR_VERSION=${CUDA_MINOR_VERSION}
+ENV DEBIAN_FRONTEND=noninteractive
+ARG TARGETARCH
+ARG TARGETVARIANT
+ARG GO_VERSION=1.25.4
+ARG UBUNTU_VERSION=2404
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+        build-essential \
+        ccache git \
+        ca-certificates \
+        make \
+        pkg-config libcurl4-openssl-dev \
+        curl unzip \
+        libssl-dev wget && \
+    apt-get clean && \
+    rm -rf /var/lib/apt/lists/*
+
+# Cuda
+ENV PATH=/usr/local/cuda/bin:${PATH}
+
+# HipBLAS requirements
+ENV PATH=/opt/rocm/bin:${PATH}
+
+
+# Vulkan requirements
+RUN <<EOT bash
+    if [ "${BUILD_TYPE}" = "vulkan" ] && [ "${SKIP_DRIVERS}" = "false" ]; then
+        apt-get update && \
+        apt-get install -y  --no-install-recommends \
+            software-properties-common pciutils wget gpg-agent && \
+        apt-get install -y libglm-dev cmake libxcb-dri3-0 libxcb-present0 libpciaccess0 \
+            libpng-dev libxcb-keysyms1-dev libxcb-dri3-dev libx11-dev g++ gcc \
+            libwayland-dev libxrandr-dev libxcb-randr0-dev libxcb-ewmh-dev \
+            git python-is-python3 bison libx11-xcb-dev liblz4-dev libzstd-dev \
+            ocaml-core ninja-build pkg-config libxml2-dev wayland-protocols python3-jsonschema \
+            clang-format qtbase5-dev qt6-base-dev libxcb-glx0-dev sudo xz-utils
+        if [ "amd64" = "$TARGETARCH" ]; then
+            wget "https://sdk.lunarg.com/sdk/download/1.4.335.0/linux/vulkansdk-linux-x86_64-1.4.335.0.tar.xz" && \
+            tar -xf vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
+            rm vulkansdk-linux-x86_64-1.4.335.0.tar.xz && \
+            mkdir -p /opt/vulkan-sdk && \
+            mv 1.4.335.0 /opt/vulkan-sdk/ && \
+            cd /opt/vulkan-sdk/1.4.335.0 && \
+            ./vulkansdk --no-deps --maxjobs \
+                vulkan-loader \
+                vulkan-validationlayers \
+                vulkan-extensionlayer \
+                vulkan-tools \
+                shaderc && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/bin/* /usr/bin/ && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/lib/* /usr/lib/x86_64-linux-gnu/ && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/include/* /usr/include/ && \
+            cp -rfv /opt/vulkan-sdk/1.4.335.0/x86_64/share/* /usr/share/ && \
+            rm -rf /opt/vulkan-sdk
+        fi
+        if [ "arm64" = "$TARGETARCH" ]; then
+            mkdir vulkan && cd vulkan && \
+            curl -L -o vulkan-sdk.tar.xz https://github.com/mudler/vulkan-sdk-arm/releases/download/1.4.335.0/vulkansdk-ubuntu-24.04-arm-1.4.335.0.tar.xz && \
+            tar -xvf vulkan-sdk.tar.xz && \
+            rm vulkan-sdk.tar.xz && \
+            cd 1.4.335.0 && \
+            cp -rfv aarch64/bin/* /usr/bin/ && \
+            cp -rfv aarch64/lib/* /usr/lib/aarch64-linux-gnu/ && \
+            cp -rfv aarch64/include/* /usr/include/ && \
+            cp -rfv aarch64/share/* /usr/share/ && \
+            cd ../.. && \
+            rm -rf vulkan
+        fi
+        ldconfig && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
+# CuBLAS requirements
+RUN <<EOT bash
+    if ( [ "${BUILD_TYPE}" = "cublas" ] || [ "${BUILD_TYPE}" = "l4t" ] ) && [ "${SKIP_DRIVERS}" = "false" ]; then
+        apt-get update && \
+        apt-get install -y  --no-install-recommends \
+            software-properties-common pciutils
+        if [ "amd64" = "$TARGETARCH" ]; then
+            curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/x86_64/cuda-keyring_1.1-1_all.deb
+        fi
+        if [ "arm64" = "$TARGETARCH" ]; then
+            if [ "${CUDA_MAJOR_VERSION}" = "13" ]; then
+                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/sbsa/cuda-keyring_1.1-1_all.deb
+            else
+                curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu${UBUNTU_VERSION}/arm64/cuda-keyring_1.1-1_all.deb
+            fi
+        fi
+        dpkg -i cuda-keyring_1.1-1_all.deb && \
+        rm -f cuda-keyring_1.1-1_all.deb && \
+        apt-get update && \
+        apt-get install -y --no-install-recommends \
+            cuda-nvcc-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcufft-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcurand-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcublas-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcusparse-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} \
+            libcusolver-dev-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
+        if [ "${CUDA_MAJOR_VERSION}" = "13" ] && [ "arm64" = "$TARGETARCH" ]; then
+            apt-get install -y --no-install-recommends \
+            libcufile-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libcudnn9-cuda-${CUDA_MAJOR_VERSION} cuda-cupti-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION} libnvjitlink-${CUDA_MAJOR_VERSION}-${CUDA_MINOR_VERSION}
+        fi
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
+
+# https://github.com/NVIDIA/Isaac-GR00T/issues/343
+RUN <<EOT bash
+    if [ "${BUILD_TYPE}" = "cublas" ] && [ "${TARGETARCH}" = "arm64" ]; then
+        wget https://developer.download.nvidia.com/compute/cudss/0.6.0/local_installers/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
+        dpkg -i cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0_0.6.0-1_arm64.deb && \
+        cp /var/cudss-local-tegra-repo-ubuntu${UBUNTU_VERSION}-0.6.0/cudss-*-keyring.gpg /usr/share/keyrings/ && \
+        apt-get update && apt-get -y install cudss cudss-cuda-${CUDA_MAJOR_VERSION} && \
+        wget https://developer.download.nvidia.com/compute/nvpl/25.5/local_installers/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
+        dpkg -i nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5_1.0-1_arm64.deb && \
+        cp /var/nvpl-local-repo-ubuntu${UBUNTU_VERSION}-25.5/nvpl-*-keyring.gpg /usr/share/keyrings/ && \
+        apt-get update && apt-get install -y nvpl
+    fi
+EOT
+
+# If we are building with clblas support, we need the libraries for the builds
+RUN if [ "${BUILD_TYPE}" = "clblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
+        apt-get update && \
+        apt-get install -y --no-install-recommends \
+            libclblast-dev && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/* \
+    ; fi
+
+RUN if [ "${BUILD_TYPE}" = "hipblas" ] && [ "${SKIP_DRIVERS}" = "false" ]; then \
+        apt-get update && \
+        apt-get install -y --no-install-recommends \
+            hipblas-dev \
+            rocblas-dev && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/* && \
+        # I have no idea why, but the ROCM lib packages don't trigger ldconfig after they install, which results in local-ai and others not being able
+        # to locate the libraries. We run ldconfig ourselves to work around this packaging deficiency
+        ldconfig && \
+        # Log which GPU architectures have rocBLAS kernel support
+        echo "rocBLAS library data architectures:" && \
+        (ls /opt/rocm*/lib/rocblas/library/Kernels* 2>/dev/null || ls /opt/rocm*/lib64/rocblas/library/Kernels* 2>/dev/null) | grep -oP 'gfx[0-9a-z+-]+' | sort -u || \
+        echo "WARNING: No rocBLAS kernel data found" \
+    ; fi
+
+RUN echo "TARGETARCH: $TARGETARCH"
+
+# We need protoc installed, and the version in 22.04 is too old.  We will create one as part installing the GRPC build below
+# but that will also being in a newer version of absl which stablediffusion cannot compile with.  This version of protoc is only
+# here so that we can generate the grpc code for the stablediffusion build
+RUN <<EOT bash
+    if [ "amd64" = "$TARGETARCH" ]; then
+        curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-x86_64.zip -o protoc.zip && \
+        unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+        rm protoc.zip
+    fi
+    if [ "arm64" = "$TARGETARCH" ]; then
+        curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v27.1/protoc-27.1-linux-aarch_64.zip -o protoc.zip && \
+        unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
+        rm protoc.zip
+    fi
+EOT
+
+# Install CMake (the version in 22.04 is too old)
+RUN <<EOT bash
+    if [ "${CMAKE_FROM_SOURCE}" = "true" ]; then
+        curl -L -s https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}.tar.gz -o cmake.tar.gz && tar xvf cmake.tar.gz && cd cmake-${CMAKE_VERSION} && ./configure && make && make install
+    else
+        apt-get update && \
+        apt-get install -y \
+            cmake && \
+        apt-get clean && \
+        rm -rf /var/lib/apt/lists/*
+    fi
+EOT
+
+COPY --from=grpc /opt/grpc /usr/local
+
+
+COPY . /LocalAI
+
+RUN <<'EOT' bash
+set -euxo pipefail
+
+if [[ -n "${CUDA_DOCKER_ARCH:-}" ]]; then
+  CUDA_ARCH_ESC="${CUDA_DOCKER_ARCH//;/\\;}"
+  export CMAKE_ARGS="${CMAKE_ARGS:-} -DCMAKE_CUDA_ARCHITECTURES=${CUDA_ARCH_ESC}"
+  echo "CMAKE_ARGS(env) = ${CMAKE_ARGS}"
+  rm -rf /LocalAI/backend/cpp/buun-llama-cpp-*-build
+fi
+
+cd /LocalAI/backend/cpp/buun-llama-cpp
+
+if [ "${TARGETARCH}" = "arm64" ] || [ "${BUILD_TYPE}" = "hipblas" ]; then
+  make buun-llama-cpp-fallback
+  make buun-llama-cpp-grpc
+  make buun-llama-cpp-rpc-server
+else
+  make buun-llama-cpp-avx
+  make buun-llama-cpp-avx2
+  make buun-llama-cpp-avx512
+  make buun-llama-cpp-fallback
+  make buun-llama-cpp-grpc
+  make buun-llama-cpp-rpc-server
+fi
+EOT
+
+
+# Copy libraries using a script to handle architecture differences
+RUN make -BC /LocalAI/backend/cpp/buun-llama-cpp package
+
+
+FROM scratch
+
+
+# Copy all available binaries (the build process only creates the appropriate ones for the target architecture)
+COPY --from=builder /LocalAI/backend/cpp/buun-llama-cpp/package/. ./

backend/backend.proto

@@ -15,6 +15,7 @@ service Backend {
   rpc PredictStream(PredictOptions) returns (stream Reply) {}
   rpc Embedding(PredictOptions) returns (EmbeddingResult) {}
   rpc GenerateImage(GenerateImageRequest) returns (Result) {}
+  rpc UpscaleImage(UpscaleImageRequest) returns (Result) {}
   rpc GenerateVideo(GenerateVideoRequest) returns (Result) {}
   rpc Generate3D(Generate3DRequest) returns (Result) {}
   rpc AudioTranscription(TranscriptRequest) returns (TranscriptResult) {}
@@ -637,6 +638,12 @@ message GenerateImageRequest {
   string ModelIdentity = 13;
 }
 
+message UpscaleImageRequest {
+  string src = 1;   // input image path
+  string dst = 2;   // output image path
+  int32  scale = 3; // upscale factor (e.g. 2 or 4)
+}
+
 message GenerateVideoRequest {
   string prompt = 1;
   string negative_prompt = 2;  // Negative prompt for video generation

backend/cpp/audio-cpp/Makefile

@@ -9,7 +9,7 @@
 # recipe is a make target (not a prepare.sh) so 'make purge && make' is a clean
 # rebuild and so the bump bot can see the pin.
 
-AUDIO_CPP_VERSION?=e800d435d130dc776baf6f3e6129bb62b1495c89
+AUDIO_CPP_VERSION?=5a8312ef7b8aa7cf14e9a24ac568cabd8725d68a
 AUDIO_CPP_REPO?=https://github.com/0xShug0/audio.cpp
 
 CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))

backend/cpp/bonsai/Makefile

@@ -1,7 +1,7 @@
 
 # Pinned to the HEAD of the `prism` branch on https://github.com/PrismML-Eng/llama.cpp.
 # Auto-bumped nightly by .github/workflows/bump_deps.yaml.
-BONSAI_VERSION?=7529fdaaf99ffdc5ca71ace9c7409a56b27ad92f
+BONSAI_VERSION?=9ca265a57f85f2117942490f421f64a226dd9847
 LLAMA_REPO?=https://github.com/PrismML-Eng/llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/bonsai/run.sh

@@ -40,6 +40,27 @@ else
 	if [ -d "$CURDIR/lib/hipblaslt/library" ]; then
 		export HIPBLASLT_TENSILE_LIBPATH="$CURDIR"/lib/hipblaslt/library
 	fi
+	# Backends built for Intel GPUs carry a copy of the Intel graphics driver,
+	# and libze_loader is only there in those builds. Level Zero looks for a
+	# driver on its own, so point it at the copy that came with this backend: it
+	# was built against the same C library, while the machine's own driver may
+	# not have been, and loading that one can crash on start.
+	#
+	# Anything the user set is left alone, so a machine with a graphics card
+	# newer than the driver carried here can still be told to use its own.
+	# Nothing is said about OpenCL: no OpenCL driver is carried, so anything we
+	# set there would leave OpenCL worse off than the machine's own setup.
+	if [ -e "$CURDIR/lib/libze_loader.so.1" ]; then
+		if [ -e "$CURDIR/lib/libze_intel_gpu.so.1" ] && [ -z "${ZE_ENABLE_ALT_DRIVERS:-}" ]; then
+			export ZE_ENABLE_ALT_DRIVERS="$CURDIR"/lib/libze_intel_gpu.so.1
+		fi
+		# Ask the driver how much graphics memory is free. Without this, the
+		# backend reads zero on an integrated graphics chip, because such a chip
+		# shares the system memory instead of having its own.
+		if [ -z "${ZES_ENABLE_SYSMAN:-}" ]; then
+			export ZES_ENABLE_SYSMAN=1
+		fi
+	fi
 fi
 
 # If there is a lib/ld.so, use it

backend/cpp/buun-llama-cpp/Makefile

@@ -0,0 +1,92 @@
+
+# Pinned to the HEAD of master on https://github.com/spiritbuun/buun-llama-cpp.
+# Auto-bumped nightly by .github/workflows/bump_deps.yaml.
+BUUN_LLAMA_VERSION?=22464d0848b87c5d56b52fdf6af2e5da46bf803e
+LLAMA_REPO?=https://github.com/spiritbuun/buun-llama-cpp
+
+CMAKE_ARGS?=
+BUILD_TYPE?=
+NATIVE?=false
+ONEAPI_VARS?=/opt/intel/oneapi/setvars.sh
+TARGET?=--target grpc-server
+JOBS?=$(shell nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 1)
+ARCH?=$(shell uname -m)
+
+CURRENT_MAKEFILE_DIR := $(dir $(abspath $(lastword $(MAKEFILE_LIST))))
+LLAMA_CPP_DIR := $(CURRENT_MAKEFILE_DIR)/../llama-cpp
+
+GREEN := \033[0;32m
+RESET := \033[0m
+
+# buun-llama-cpp is a llama.cpp fork-of-a-fork (spiritbuun/buun-llama-cpp forked
+# TheTom/llama-cpp-turboquant, which itself forked ggml-org/llama.cpp). Rather
+# than duplicating grpc-server.cpp / CMakeLists.txt / prepare.sh we reuse the
+# ones in backend/cpp/llama-cpp, and only swap which repo+sha the fetch step
+# pulls. Each flavor target copies ../llama-cpp into a sibling
+# ../buun-llama-cpp-<flavor>-build directory, then invokes llama-cpp's own
+# build-llama-cpp-grpc-server with LLAMA_REPO/LLAMA_VERSION overridden to point
+# at the fork.
+PATCHES_DIR := $(CURRENT_MAKEFILE_DIR)/patches
+
+# Each flavor target:
+#   1. copies backend/cpp/llama-cpp/ (grpc-server.cpp + prepare.sh + CMakeLists.txt + Makefile)
+#      into a sibling buun-llama-cpp-<flavor>-build directory;
+#   2. clones the buun fork into buun-llama-cpp-<flavor>-build/llama.cpp via the
+#      copy's own `llama.cpp` target, overriding LLAMA_REPO/LLAMA_VERSION;
+#   3. applies patches from backend/cpp/buun-llama-cpp/patches/ to the cloned
+#      fork sources (for backporting upstream commits the fork hasn't pulled);
+#   4. runs the copy's `grpc-server` target, which produces the binary we copy
+#      up as buun-llama-cpp-<flavor>.
+define buun-llama-cpp-build
+	rm -rf $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-$(1)-build
+	cp -rf $(LLAMA_CPP_DIR) $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-$(1)-build
+	# Stock llama.cpp patches target upstream and may not apply to this fork.
+	# The buun-specific compatibility series is applied explicitly below.
+	rm -rf $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-$(1)-build/patches
+	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-$(1)-build purge
+	# Augment the copied grpc-server.cpp's KV-cache allow-list with the
+	# fork's turbo2/turbo3/turbo4/turbo2_tcq/turbo3_tcq types and wire up the
+	# DFlash-specific option handlers (tree_budget / draft_topk). We patch the
+	# *copy*, never the original under backend/cpp/llama-cpp/, so the stock
+	# llama-cpp build stays compiling against vanilla upstream.
+	bash $(CURRENT_MAKEFILE_DIR)/patch-grpc-server.sh $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-$(1)-build/grpc-server.cpp
+	bash $(LLAMA_CPP_DIR)/disable-score-task.sh $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-$(1)-build/grpc-server.cpp
+	$(info $(GREEN)I buun-llama-cpp build info:$(1)$(RESET))
+	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BUUN_LLAMA_VERSION) \
+	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-$(1)-build llama.cpp
+	bash $(CURRENT_MAKEFILE_DIR)/apply-patches.sh $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-$(1)-build/llama.cpp $(PATCHES_DIR)
+	CMAKE_ARGS="$(CMAKE_ARGS) $(2)" TARGET="$(3)" \
+	LLAMA_REPO=$(LLAMA_REPO) LLAMA_VERSION=$(BUUN_LLAMA_VERSION) \
+	$(MAKE) -C $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-$(1)-build grpc-server
+	cp -rfv $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-$(1)-build/grpc-server buun-llama-cpp-$(1)
+endef
+
+buun-llama-cpp-avx2:
+	$(call buun-llama-cpp-build,avx2,-DGGML_AVX=on -DGGML_AVX2=on -DGGML_AVX512=off -DGGML_FMA=on -DGGML_F16C=on,--target grpc-server)
+
+buun-llama-cpp-avx512:
+	$(call buun-llama-cpp-build,avx512,-DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=on -DGGML_FMA=on -DGGML_F16C=on,--target grpc-server)
+
+buun-llama-cpp-avx:
+	$(call buun-llama-cpp-build,avx,-DGGML_AVX=on -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off,--target grpc-server)
+
+buun-llama-cpp-fallback:
+	$(call buun-llama-cpp-build,fallback,-DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off,--target grpc-server)
+
+buun-llama-cpp-grpc:
+	$(call buun-llama-cpp-build,grpc,-DGGML_RPC=ON -DGGML_AVX=off -DGGML_AVX2=off -DGGML_AVX512=off -DGGML_FMA=off -DGGML_F16C=off -DGGML_BMI2=off,--target grpc-server --target rpc-server)
+
+buun-llama-cpp-rpc-server: buun-llama-cpp-grpc
+	cp -rf $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-grpc-build/llama.cpp/build/bin/rpc-server buun-llama-cpp-rpc-server
+
+package:
+	bash package.sh
+
+test:
+	bash test-patch-grpc-server.sh
+
+purge:
+	rm -rf $(CURRENT_MAKEFILE_DIR)/../buun-llama-cpp-*-build
+	rm -rf buun-llama-cpp-* package
+
+clean: purge

backend/cpp/buun-llama-cpp/apply-patches.sh

@@ -0,0 +1,50 @@
+#!/bin/bash
+# Apply the buun-llama-cpp patch series to a cloned buun-llama-cpp checkout.
+#
+# buun-llama-cpp is a fork-of-a-fork that branched off upstream llama.cpp
+# before some API changes the shared backend/cpp/llama-cpp/grpc-server.cpp
+# depends on. We carry those upstream commits as patch files under
+# backend/cpp/buun-llama-cpp/patches/ and apply them here so the reused
+# grpc-server source compiles against the fork unmodified.
+#
+# Drop the corresponding patch from patches/ whenever the fork catches up with
+# upstream — the build will fail fast if a patch stops applying, which is the
+# signal to retire it.
+
+set -euo pipefail
+
+if [[ $# -ne 2 ]]; then
+    echo "usage: $0 <llama.cpp-src-dir> <patches-dir>" >&2
+    exit 2
+fi
+
+SRC_DIR=$1
+PATCHES_DIR=$2
+
+if [[ ! -d "$SRC_DIR" ]]; then
+    echo "source dir does not exist: $SRC_DIR" >&2
+    exit 2
+fi
+
+if [[ ! -d "$PATCHES_DIR" ]]; then
+    echo "no patches dir at $PATCHES_DIR, nothing to apply"
+    exit 0
+fi
+
+shopt -s nullglob
+patches=("$PATCHES_DIR"/*.patch)
+shopt -u nullglob
+
+if [[ ${#patches[@]} -eq 0 ]]; then
+    echo "no .patch files in $PATCHES_DIR, nothing to apply"
+    exit 0
+fi
+
+cd "$SRC_DIR"
+
+for patch in "${patches[@]}"; do
+    echo "==> applying $patch"
+    git apply --verbose "$patch"
+done
+
+echo "all buun-llama-cpp patches applied successfully"

backend/cpp/buun-llama-cpp/package.sh

@@ -0,0 +1,57 @@
+#!/bin/bash
+
+# Script to copy the appropriate libraries based on architecture
+# This script is used in the final stage of the Dockerfile
+
+set -e
+
+CURDIR=$(dirname "$(realpath $0)")
+REPO_ROOT="${CURDIR}/../../.."
+
+# Create lib directory
+mkdir -p $CURDIR/package/lib
+
+cp -avrf $CURDIR/buun-llama-cpp-* $CURDIR/package/
+cp -rfv $CURDIR/run.sh $CURDIR/package/
+
+# Detect architecture and copy appropriate libraries
+if [ -f "/lib64/ld-linux-x86-64.so.2" ]; then
+    # x86_64 architecture
+    echo "Detected x86_64 architecture, copying x86_64 libraries..."
+    cp -arfLv /lib64/ld-linux-x86-64.so.2 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/x86_64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/x86_64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/x86_64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/x86_64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+elif [ -f "/lib/ld-linux-aarch64.so.1" ]; then
+    # ARM64 architecture
+    echo "Detected ARM64 architecture, copying ARM64 libraries..."
+    cp -arfLv /lib/ld-linux-aarch64.so.1 $CURDIR/package/lib/ld.so
+    cp -arfLv /lib/aarch64-linux-gnu/libc.so.6 $CURDIR/package/lib/libc.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgcc_s.so.1 $CURDIR/package/lib/libgcc_s.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libstdc++.so.6 $CURDIR/package/lib/libstdc++.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libm.so.6 $CURDIR/package/lib/libm.so.6
+    cp -arfLv /lib/aarch64-linux-gnu/libgomp.so.1 $CURDIR/package/lib/libgomp.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libdl.so.2 $CURDIR/package/lib/libdl.so.2
+    cp -arfLv /lib/aarch64-linux-gnu/librt.so.1 $CURDIR/package/lib/librt.so.1
+    cp -arfLv /lib/aarch64-linux-gnu/libpthread.so.0 $CURDIR/package/lib/libpthread.so.0
+else
+    echo "Error: Could not detect architecture"
+    exit 1
+fi
+
+# Package GPU libraries based on BUILD_TYPE
+GPU_LIB_SCRIPT="${REPO_ROOT}/scripts/build/package-gpu-libs.sh"
+if [ -f "$GPU_LIB_SCRIPT" ]; then
+    echo "Packaging GPU libraries for BUILD_TYPE=${BUILD_TYPE:-cpu}..."
+    source "$GPU_LIB_SCRIPT" "$CURDIR/package/lib"
+    package_gpu_libs
+fi
+
+echo "Packaging completed successfully"
+ls -liah $CURDIR/package/
+ls -liah $CURDIR/package/lib/

backend/cpp/buun-llama-cpp/patch-grpc-server.sh

@@ -0,0 +1,196 @@
+#!/bin/bash
+# Patch the shared backend/cpp/llama-cpp/grpc-server.cpp *copy* used by the
+# buun-llama-cpp build to account for three gaps between upstream and the fork:
+#
+#   1. Augment the kv_cache_types[] allow-list so `LoadModel` accepts the
+#      fork-specific `turbo2` / `turbo3` / `turbo4` cache types plus the buun
+#      additions `turbo2_tcq` / `turbo3_tcq`.
+#
+#   2. Adapt the post-refactor speculative-decoding fields and option handlers
+#      to the fork's legacy flat common_params_speculative layout, while adding
+#      buun-exclusive tree_budget / draft_topk support.
+#      These reference struct fields (common_params.speculative.tree_budget
+#      and .draft_topk) that only exist in buun's common/common.h — adding
+#      them to the shared backend/cpp/llama-cpp/grpc-server.cpp would break
+#      the stock llama-cpp build, so we inject them only into the buun copy.
+#
+#   3. Replace `get_media_marker()` (added upstream in ggml-org/llama.cpp#21962,
+#      server-side random per-instance marker) with the legacy "<__media__>"
+#      literal. The fork branched before that PR, so server-common.cpp has no
+#      get_media_marker symbol. The fork's mtmd_default_marker() still returns
+#      "<__media__>", and Go-side tooling falls back to that sentinel when the
+#      backend does not expose media_marker, so substituting the literal keeps
+#      behavior identical on the buun path.
+#
+# We patch the *copy* sitting in buun-llama-cpp-<flavor>-build/, never the
+# original under backend/cpp/llama-cpp/, so the stock llama-cpp build keeps
+# compiling against vanilla upstream.
+#
+# Idempotent: skips each insertion if its marker is already present (so re-runs
+# of the same build dir don't double-insert).
+
+set -euo pipefail
+
+if [[ $# -ne 1 ]]; then
+    echo "usage: $0 <grpc-server.cpp>" >&2
+    exit 2
+fi
+
+SRC=$1
+
+if [[ ! -f "$SRC" ]]; then
+    echo "grpc-server.cpp not found at $SRC" >&2
+    exit 2
+fi
+
+if grep -q 'GGML_TYPE_TURBO2_TCQ' "$SRC"; then
+    echo "==> $SRC already has buun cache types, skipping KV allow-list patch"
+else
+    echo "==> patching $SRC to allow turbo2/turbo3/turbo4/turbo2_tcq/turbo3_tcq KV-cache types"
+
+    # Insert the five TURBO entries right after the first `    GGML_TYPE_Q5_1,`
+    # line (the kv_cache_types[] allow-list). Using awk because the builder
+    # image does not ship python3, and GNU sed's multi-line `a\` quoting is
+    # awkward.
+    awk '
+        /^    GGML_TYPE_Q5_1,$/ && !done {
+            print
+            print "    // buun-llama-cpp fork extras — added by patch-grpc-server.sh"
+            print "    GGML_TYPE_TURBO2_0,"
+            print "    GGML_TYPE_TURBO3_0,"
+            print "    GGML_TYPE_TURBO4_0,"
+            print "    GGML_TYPE_TURBO2_TCQ,"
+            print "    GGML_TYPE_TURBO3_TCQ,"
+            done = 1
+            next
+        }
+        { print }
+        END {
+            if (!done) {
+                print "patch-grpc-server.sh: anchor `    GGML_TYPE_Q5_1,` not found" > "/dev/stderr"
+                exit 1
+            }
+        }
+    ' "$SRC" > "$SRC.tmp"
+    mv "$SRC.tmp" "$SRC"
+
+    echo "==> KV allow-list patch OK"
+fi
+
+if grep -q 'buun-llama-cpp legacy speculative options' "$SRC"; then
+    echo "==> $SRC already has legacy speculative option handlers, skipping"
+else
+    echo "==> replacing modern speculative option handlers with the fork-compatible set"
+
+    # Replace the whole speculative option section. The fork predates chained
+    # speculative types and the nested draft/ngram families, so retaining any
+    # modern-only handler makes the copied server fail at compile time.
+    awk '
+        /} else if \(!strcmp\(optname, "spec_type"\)/ && !done {
+            print "        // buun-llama-cpp legacy speculative options"
+            print "        } else if (!strcmp(optname, \"spec_type\") || !strcmp(optname, \"speculative_type\")) {"
+            print "            auto type = common_speculative_type_from_name(optval_str.substr(0, optval_str.find(\",\")));"
+            print "            if (type != COMMON_SPECULATIVE_TYPE_COUNT) params.speculative.type = type;"
+            print "        } else if (!strcmp(optname, \"spec_n_max\") || !strcmp(optname, \"draft_max\")) {"
+            print "            if (optval != NULL) { try { params.speculative.n_max = std::stoi(optval_str); } catch (...) {} }"
+            print "        } else if (!strcmp(optname, \"spec_n_min\") || !strcmp(optname, \"draft_min\")) {"
+            print "            if (optval != NULL) { try { params.speculative.n_min = std::stoi(optval_str); } catch (...) {} }"
+            print "        } else if (!strcmp(optname, \"spec_p_min\") || !strcmp(optname, \"draft_p_min\")) {"
+            print "            if (optval != NULL) { try { params.speculative.p_min = std::stof(optval_str); } catch (...) {} }"
+            print "        } else if (!strcmp(optname, \"spec_p_split\")) {"
+            print "            if (optval != NULL) { try { params.speculative.p_split = std::stof(optval_str); } catch (...) {} }"
+            print "        } else if (!strcmp(optname, \"spec_ngram_size_n\") || !strcmp(optname, \"ngram_size_n\")) {"
+            print "            if (optval != NULL) { try { params.speculative.ngram_size_n = (uint16_t)std::stoi(optval_str); } catch (...) {} }"
+            print "        } else if (!strcmp(optname, \"spec_ngram_size_m\") || !strcmp(optname, \"ngram_size_m\")) {"
+            print "            if (optval != NULL) { try { params.speculative.ngram_size_m = (uint16_t)std::stoi(optval_str); } catch (...) {} }"
+            print "        } else if (!strcmp(optname, \"spec_ngram_min_hits\") || !strcmp(optname, \"ngram_min_hits\")) {"
+            print "            if (optval != NULL) { try { params.speculative.ngram_min_hits = (uint16_t)std::stoi(optval_str); } catch (...) {} }"
+            print "        } else if (!strcmp(optname, \"draft_gpu_layers\")) {"
+            print "            if (optval != NULL) { try { params.speculative.n_gpu_layers = std::stoi(optval_str); } catch (...) {} }"
+            print "        } else if (!strcmp(optname, \"tree_budget\")) {"
+            print "            if (optval != NULL) { try { params.speculative.tree_budget = std::stoi(optval_str); } catch (...) {} }"
+            print "        } else if (!strcmp(optname, \"draft_topk\")) {"
+            print "            if (optval != NULL) { try { params.speculative.draft_topk = std::stoi(optval_str); } catch (...) {} }"
+            skipping = 1
+            next
+        }
+        skipping && /^    }$/ { skipping = 0; done = 1; print; next }
+        !skipping { print }
+        END {
+            if (!done) {
+                print "patch-grpc-server.sh: speculative option section not found" > "/dev/stderr"
+                exit 1
+            }
+        }
+    ' "$SRC" > "$SRC.tmp"
+    mv "$SRC.tmp" "$SRC"
+
+    echo "==> legacy speculative option-handler patch OK"
+fi
+
+# The modern server initializes a vector of speculative types when DraftModel
+# is present. The fork still exposes a single enum value.
+awk '
+    /const bool no_spec_type = params\.speculative\.types\.empty\(\)/ && !done {
+        print "        if (params.speculative.type == COMMON_SPECULATIVE_TYPE_NONE) {"
+        print "            params.speculative.type = COMMON_SPECULATIVE_TYPE_DRAFT;"
+        print "        }"
+        skipping = 1
+        next
+    }
+    skipping && /^        }$/ { skipping = 0; done = 1; next }
+    !skipping { print }
+' "$SRC" > "$SRC.tmp"
+mv "$SRC.tmp" "$SRC"
+
+# Map supported post-refactor fields back to the names used by the pinned fork.
+sed -E \
+    -e 's/params\.speculative\.draft\.mparams\.path/params.speculative.mparams_dft.path/g' \
+    -e 's/params\.speculative\.draft\.n_gpu_layers/params.speculative.n_gpu_layers/g' \
+    -e 's/ctx_server\.impl->model_tgt/ctx_server.impl->model/g' \
+    -e '/params\.cache_idle_slots =/d' \
+    -e '/params\.split_mode = LLAMA_SPLIT_MODE_TENSOR;/d' \
+    -e '/params\.speculative\.draft\.tensor_buft_overrides/d' \
+    "$SRC" > "$SRC.tmp"
+mv "$SRC.tmp" "$SRC"
+
+if ! grep -q '^#define LOCALAI_TURBOQUANT_NO_CHECKPOINT_MIN_STEP' "$SRC"; then
+    sed '0,/^#include/{s/^#include/#define LOCALAI_TURBOQUANT_NO_CHECKPOINT_MIN_STEP 1\n\n#include/}' "$SRC" > "$SRC.tmp"
+    mv "$SRC.tmp" "$SRC"
+fi
+
+if grep -qE 'ctx_server\.get_meta\(\)\.logit_bias_eog|params_base\.sampling\.logit_bias_eog,' "$SRC"; then
+    echo "==> patching $SRC to drop the logit_bias_eog arg from params_from_json_cmpl() callsites (buun still uses the pre-refactor 4-arg signature)"
+    # Upstream llama.cpp refactored params_from_json_cmpl to take a precomputed
+    # logit_bias_eog vector after buun's 2026-04-05 fork-point — simultaneously
+    # adding server_context_meta::logit_bias_eog as the supplier. Buun carries
+    # neither change: its params_from_json_cmpl is still 4-arg, and internally
+    # derives logit_bias_eog from the common_params it's passed. So we just
+    # delete the argument line entirely — the remaining 4 args match buun's
+    # signature and the resulting behavior matches upstream bit-for-bit
+    # (upstream's 5th arg is the same data buun derives internally).
+    #
+    # Guard is broad so this works whether the line has been run through this
+    # block before (leaving params_base.sampling.logit_bias_eog,) or not
+    # (leaving the original ctx_server.get_meta().logit_bias_eog,).
+    sed -E '/^[[:space:]]+(ctx_server\.get_meta\(\)\.logit_bias_eog|params_base\.sampling\.logit_bias_eog),$/d' "$SRC" > "$SRC.tmp"
+    mv "$SRC.tmp" "$SRC"
+    echo "==> logit_bias_eog arg drop OK"
+else
+    echo "==> $SRC has no logit_bias_eog arg line, skipping"
+fi
+
+if grep -q 'get_media_marker()' "$SRC"; then
+    echo "==> patching $SRC to replace get_media_marker() with legacy \"<__media__>\" literal"
+    # Only one call site today (ModelMetadata), but replace all occurrences to
+    # stay robust if upstream adds more. Use a temp file to avoid relying on
+    # sed -i portability (the builder image uses GNU sed, but keeping this
+    # consistent with the awk block above).
+    sed 's/get_media_marker()/"<__media__>"/g' "$SRC" > "$SRC.tmp"
+    mv "$SRC.tmp" "$SRC"
+    echo "==> get_media_marker() substitution OK"
+else
+    echo "==> $SRC has no get_media_marker() call, skipping media-marker patch"
+fi
+
+echo "==> all patches applied"

backend/cpp/buun-llama-cpp/patches/0001-fattn-atomicAdd-double-shim.patch

@@ -0,0 +1,46 @@
+Subject: [PATCH] ggml-cuda/fattn: provide atomicAdd(double*,double) shim for pre-sm_60
+
+Buun's Q² calibration path in ggml_cuda_turbo_scale_q calls
+  atomicAdd(&d_q_channel_sq_fattn[threadIdx.x], (double)(val * val));
+but native double atomicAdd is only available on compute capability 6.0
+and newer. Compiling against a CUDA arch list that includes older
+architectures (LocalAI's CUDA 12 Docker image builds for the full
+published arch range) fails with:
+
+    fattn.cu(812): error: no instance of overloaded function "atomicAdd"
+      matches the argument list, argument types are: (double *, double)
+
+Add the canonical CUDA-programming-guide shim at the top of fattn.cu so
+pre-sm_60 codegen has a definition to call. On sm_60+ the native CUDA
+intrinsic is used and the shim is elided via __CUDA_ARCH__.
+
+--- a/ggml/src/ggml-cuda/fattn.cu
++++ b/ggml/src/ggml-cuda/fattn.cu
+@@ -7,6 +7,27 @@
+
+ #include <atomic>
+
++// Pre-sm_60 double atomicAdd shim. Native double atomicAdd(double*,double)
++// is only available on CUDA compute capability 6.0+ (see CUDA C Programming
++// Guide, B.15 Atomic Functions). Buun's Q² calibration path below calls
++// atomicAdd with a double*; without this definition, nvcc fails to find a
++// matching overload whenever the compile target list includes pre-sm_60
++// architectures. The standard CAS loop implementation below matches the
++// semantics of the native intrinsic.
++#if defined(__CUDA_ARCH__) && __CUDA_ARCH__ < 600
++static __device__ double atomicAdd(double * address, double val) {
++    unsigned long long int * address_as_ull = (unsigned long long int *)address;
++    unsigned long long int old = *address_as_ull;
++    unsigned long long int assumed;
++    do {
++        assumed = old;
++        old = atomicCAS(address_as_ull, assumed,
++                        __double_as_longlong(val + __longlong_as_double(assumed)));
++    } while (assumed != old);
++    return __longlong_as_double(old);
++}
++#endif
++
+ // InnerQ: update the fattn-side inverse scale array from host (all devices)
+ void turbo_innerq_update_fattn_scales(const float * scale_inv) {
+     int cur_device;

backend/cpp/buun-llama-cpp/patches/0002-argmax-shfl-xor-sync-add-width.patch

@@ -0,0 +1,32 @@
+Subject: [PATCH] ggml-cuda/argmax: pass WARP_SIZE to the top-K __shfl_xor_sync calls
+
+Two __shfl_xor_sync calls in the top-K intra-warp merge drop the `width`
+argument and rely on the CUDA default (warpSize). Every other call in
+the same file already passes WARP_SIZE explicitly, and the HIP/ROCm
+compatibility shim at ggml/src/ggml-cuda/vendors/hip.h:33 is a 4-arg
+function-like macro — so the 3-arg form fails to preprocess when
+building with hipcc against ROCm:
+
+    argmax.cu:265: error: too few arguments provided to function-like
+      macro invocation
+    note: macro '__shfl_xor_sync' defined here:
+      #define __shfl_xor_sync(mask, var, laneMask, width) \
+              __shfl_xor(var, laneMask, width)
+
+Align the two call sites with the rest of the file by passing WARP_SIZE
+explicitly. On CUDA the generated code is unchanged (warpSize is the
+default); on HIP it now matches the macro's arity.
+
+--- a/ggml/src/ggml-cuda/argmax.cu
++++ b/ggml/src/ggml-cuda/argmax.cu
+@@ -262,8 +262,8 @@
+     // Each step: lane gets partner's min element, if it beats our min, replace and re-heapify
+     for (int offset = WARP_SIZE / 2; offset > 0; offset >>= 1) {
+         for (int i = 0; i < K; i++) {
+-            float partner_val = __shfl_xor_sync(0xFFFFFFFF, heap_val[i], offset);
+-            int partner_idx = __shfl_xor_sync(0xFFFFFFFF, heap_idx[i], offset);
++            float partner_val = __shfl_xor_sync(0xFFFFFFFF, heap_val[i], offset, WARP_SIZE);
++            int partner_idx = __shfl_xor_sync(0xFFFFFFFF, heap_idx[i], offset, WARP_SIZE);
+             if (partner_val > heap_val[0]) {
+                 heap_val[0] = partner_val;
+                 heap_idx[0] = partner_idx;

backend/cpp/buun-llama-cpp/patches/0003-hip-add-memcpy-symbol-aliases.patch

@@ -0,0 +1,24 @@
+Subject: [PATCH] ggml-cuda/vendors/hip: alias cudaMemcpy{To,From}Symbol to hip counterparts
+
+Buun's Q² calibration + TCQ codebook upload paths in fattn.cu use
+cudaMemcpyToSymbol / cudaMemcpyFromSymbol. The HIP-compat header in
+ggml/src/ggml-cuda/vendors/hip.h already aliases the scalar cudaMemcpy
+family (cudaMemcpy, cudaMemcpyAsync, cudaMemcpy2DAsync, …) but is
+missing the symbol variants. Building with hipcc therefore fails with
+15+ "use of undeclared identifier 'cudaMemcpyToSymbol'" errors.
+
+Add the two missing aliases alongside the existing memcpy block. HIP
+provides hipMemcpy{To,From}Symbol with the same signature as CUDA's
+equivalents, so this is a straight name substitution.
+
+--- a/ggml/src/ggml-cuda/vendors/hip.h
++++ b/ggml/src/ggml-cuda/vendors/hip.h
+@@ -85,6 +85,8 @@
+ #define cudaMemcpyDeviceToDevice hipMemcpyDeviceToDevice
+ #define cudaMemcpyDeviceToHost hipMemcpyDeviceToHost
+ #define cudaMemcpyHostToDevice hipMemcpyHostToDevice
++#define cudaMemcpyToSymbol hipMemcpyToSymbol
++#define cudaMemcpyFromSymbol hipMemcpyFromSymbol
+ #define cudaMemcpyKind hipMemcpyKind
+ #define cudaMemset hipMemset
+ #define cudaMemsetAsync hipMemsetAsync

backend/cpp/buun-llama-cpp/patches/0004-fattn-fwht128-shfl-xor-sync-add-width.patch

@@ -0,0 +1,36 @@
+Subject: [PATCH] ggml-cuda/fattn: pass WARP_SIZE to fwht128 __shfl_xor_sync calls
+
+Same issue as the argmax top-K fix: two __shfl_xor_sync call sites in
+the FWHT-128 butterfly kernels (ggml_cuda_fwht128 and fwht128_store_half)
+use the 3-arg CUDA form and omit the `width` argument that the HIP
+function-like macro in vendors/hip.h:33 requires. Hipcc fails with:
+
+    fattn.cu:512: too few arguments provided to function-like macro
+      invocation
+    note: macro '__shfl_xor_sync' defined here:
+      #define __shfl_xor_sync(mask, var, laneMask, width) \
+              __shfl_xor(var, laneMask, width)
+
+Add WARP_SIZE to both calls. CUDA codegen is unchanged (warpSize is the
+default); HIP now matches the macro arity.
+
+--- a/ggml/src/ggml-cuda/fattn.cu
++++ b/ggml/src/ggml-cuda/fattn.cu
+@@ -509,7 +509,7 @@
+     // Intra-warp passes: shuffle xor with stride h, no smem, no sync.
+     #pragma unroll
+     for (int h = 1; h <= 16; h *= 2) {
+-        const float other = __shfl_xor_sync(0xFFFFFFFF, val, h);
++        const float other = __shfl_xor_sync(0xFFFFFFFF, val, h, WARP_SIZE);
+         val = (tid & h) ? (other - val) : (val + other);
+     }
+
+@@ -533,7 +533,7 @@
+ static __device__ __forceinline__ void fwht128_store_half(
+         float val, half * dst_base) {
+     const int tid = threadIdx.x;
+-    const float neighbor = __shfl_xor_sync(0xFFFFFFFF, val, 1);
++    const float neighbor = __shfl_xor_sync(0xFFFFFFFF, val, 1, WARP_SIZE);
+     if ((tid & 1) == 0) {
+         const half2 packed = __floats2half2_rn(val, neighbor);
+         *((half2 *)(dst_base + tid)) = packed;

backend/cpp/buun-llama-cpp/run.sh

@@ -0,0 +1,65 @@
+#!/bin/bash
+set -ex
+
+# Get the absolute current dir where the script is located
+CURDIR=$(dirname "$(realpath $0)")
+
+cd /
+
+echo "CPU info:"
+grep -e "model\sname" /proc/cpuinfo | head -1
+grep -e "flags" /proc/cpuinfo | head -1
+
+BINARY=buun-llama-cpp-fallback
+
+if grep -q -e "\savx\s" /proc/cpuinfo ; then
+	echo "CPU:    AVX    found OK"
+	if [ -e $CURDIR/buun-llama-cpp-avx ]; then
+		BINARY=buun-llama-cpp-avx
+	fi
+fi
+
+if grep -q -e "\savx2\s" /proc/cpuinfo ; then
+	echo "CPU:    AVX2   found OK"
+	if [ -e $CURDIR/buun-llama-cpp-avx2 ]; then
+		BINARY=buun-llama-cpp-avx2
+	fi
+fi
+
+# Check avx 512
+if grep -q -e "\savx512f\s" /proc/cpuinfo ; then
+	echo "CPU:    AVX512F found OK"
+	if [ -e $CURDIR/buun-llama-cpp-avx512 ]; then
+		BINARY=buun-llama-cpp-avx512
+	fi
+fi
+
+if [ -n "$LLAMACPP_GRPC_SERVERS" ]; then
+	if [ -e $CURDIR/buun-llama-cpp-grpc ]; then
+		BINARY=buun-llama-cpp-grpc
+	fi
+fi
+
+# Extend ld library path with the dir where this script is located/lib
+if [ "$(uname)" == "Darwin" ]; then
+	export DYLD_LIBRARY_PATH=$CURDIR/lib:$DYLD_LIBRARY_PATH
+else
+	export LD_LIBRARY_PATH=$CURDIR/lib:$LD_LIBRARY_PATH
+	# Tell rocBLAS where to find TensileLibrary data (GPU kernel tuning files)
+	if [ -d "$CURDIR/lib/rocblas/library" ]; then
+		export ROCBLAS_TENSILE_LIBPATH=$CURDIR/lib/rocblas/library
+	fi
+fi
+
+# If there is a lib/ld.so, use it
+if [ -f $CURDIR/lib/ld.so ]; then
+	echo "Using lib/ld.so"
+	echo "Using binary: $BINARY"
+	exec $CURDIR/lib/ld.so $CURDIR/$BINARY "$@"
+fi
+
+echo "Using binary: $BINARY"
+exec $CURDIR/$BINARY "$@"
+
+# We should never reach this point, however just in case we do, run fallback
+exec $CURDIR/buun-llama-cpp-fallback "$@"

backend/cpp/buun-llama-cpp/test-patch-grpc-server.sh

@@ -0,0 +1,34 @@
+#!/bin/bash
+
+set -euo pipefail
+
+SCRIPT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)
+SOURCE="$SCRIPT_DIR/../llama-cpp/grpc-server.cpp"
+TMP_DIR=$(mktemp -d)
+trap 'rm -rf "$TMP_DIR"' EXIT
+
+cp "$SOURCE" "$TMP_DIR/grpc-server.cpp"
+bash "$SCRIPT_DIR/patch-grpc-server.sh" "$TMP_DIR/grpc-server.cpp"
+bash "$SCRIPT_DIR/../llama-cpp/disable-score-task.sh" "$TMP_DIR/grpc-server.cpp"
+bash "$SCRIPT_DIR/patch-grpc-server.sh" "$TMP_DIR/grpc-server.cpp"
+
+for unsupported in \
+    'params.cache_idle_slots' \
+    'params.speculative.types' \
+    'params.speculative.draft.' \
+    'common_speculative_types_from_names' \
+    'COMMON_SPECULATIVE_TYPE_DRAFT_SIMPLE' \
+    'ctx_server.impl->model_tgt'; do
+    if grep -Fq "$unsupported" "$TMP_DIR/grpc-server.cpp"; then
+        echo "unsupported buun API remains: $unsupported" >&2
+        exit 1
+    fi
+done
+
+grep -Fq '#define LOCALAI_LLAMA_CPP_NO_SCORE_TASK 1' "$TMP_DIR/grpc-server.cpp"
+grep -Fq '#define LOCALAI_TURBOQUANT_NO_CHECKPOINT_MIN_STEP 1' "$TMP_DIR/grpc-server.cpp"
+grep -Fq 'params.speculative.mparams_dft.path = request->draftmodel();' "$TMP_DIR/grpc-server.cpp"
+grep -Fq 'params.speculative.type = COMMON_SPECULATIVE_TYPE_DRAFT;' "$TMP_DIR/grpc-server.cpp"
+grep -Fq 'ctx_server.impl->model' "$TMP_DIR/grpc-server.cpp"
+
+echo "buun grpc-server compatibility transform passed"

backend/cpp/ik-llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-IK_LLAMA_VERSION?=6647db9c27760044950fd6f99060456ae3d15df3
+IK_LLAMA_VERSION?=cb9147fd0d9c08a9a84eee5ac405a73f4e10e3e1
 LLAMA_REPO?=https://github.com/ikawrakow/ik_llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/Makefile

@@ -1,5 +1,5 @@
 
-LLAMA_VERSION?=1cbfd1988311775425d36c0ce066590f7d3049cf
+LLAMA_VERSION?=a7a6d0d269c896218b6c78e0933bd6a17519d3f6
 LLAMA_REPO?=https://github.com/ggerganov/llama.cpp
 
 CMAKE_ARGS?=

backend/cpp/llama-cpp/patches/0001-add-minimax-m3-chat-parser.patch

@@ -1,225 +0,0 @@
-# MiniMax-M3 chat-template parser, vendored from upstream llama.cpp PR #24523.
-#
-# Upstream has since merged the *model* half of #24523 (LLM_ARCH_MINIMAX_M3,
-# src/models/minimax-m3.cpp, the gguf-py constants and conversion/minimax.py), so
-# only the chat half is carried here: M3's namespace token "]<]minimax[>[" collides
-# with the autoparser's markup delimiters, so common/chat.cpp needs a dedicated
-# template detection + PEG parser that upstream does not have yet.
-#
-# Rebased against LLAMA_VERSION 0d47ea7427463093e69128bf2c2f9cd06b3ee5b3, which also
-# renamed common_chat_params::thinking_end_tag to thinking_end_tags (a vector).
-# LLAMA_VERSION is auto-bumped nightly; if a bump rejects this patch, re-vendor from
-# #24523 — or, once the chat half merges upstream, delete this file.
-# See https://github.com/mudler/LocalAI/issues/10820 and PR #10837.
-diff --git a/common/chat.cpp b/common/chat.cpp
-index 7a6e7238c..2dd015a2e 100644
---- a/common/chat.cpp
-+++ b/common/chat.cpp
-@@ -2121,6 +2121,191 @@ static common_chat_params common_chat_params_init_deepseek_v3_2(const common_cha
-     return data;
- }
- 
-+static common_chat_params common_chat_params_init_minimax_m3(const common_chat_template &    tmpl,
-+                                                             const autoparser::generation_params & inputs) {
-+    common_chat_params data;
-+
-+    data.prompt             = common_chat_template_direct_apply_impl(tmpl, inputs);
-+    data.generation_prompt  = common_chat_template_generation_prompt_impl(tmpl, inputs);
-+    data.format             = COMMON_CHAT_FORMAT_PEG_NATIVE;
-+    data.supports_thinking  = true;
-+    data.thinking_start_tag = "<mm:think>";
-+    data.thinking_end_tags  = {"</mm:think>"};
-+
-+    // M3 prefixes every tool tag with the namespace token "]<]minimax[>[";
-+    // params use the parameter name as the tag (<file_path>...</file_path>).
-+    const std::string NS           = "]<]minimax[>[";
-+    const std::string THINK_START  = "<mm:think>";
-+    const std::string THINK_END    = "</mm:think>";
-+    const std::string FC_START     = NS + "<tool_call>";
-+    const std::string FC_END       = NS + "</tool_call>";
-+    const std::string INVOKE_END   = NS + "</invoke>";
-+
-+    data.preserved_tokens   = {
-+        NS,
-+        "<tool_call>",
-+        "</tool_call>",
-+        THINK_START,
-+        THINK_END,
-+    };
-+
-+    auto has_tools           = inputs.tools.is_array() && !inputs.tools.empty();
-+    auto has_response_format = !inputs.json_schema.is_null() && inputs.json_schema.is_object();
-+    auto extract_reasoning   = inputs.reasoning_format != COMMON_REASONING_FORMAT_NONE;
-+    auto include_grammar     = has_response_format || (has_tools && inputs.tool_choice != COMMON_CHAT_TOOL_CHOICE_NONE);
-+
-+    const std::string GEN_PROMPT = data.generation_prompt;
-+
-+    if (inputs.has_continuation()) {
-+        const auto & msg = inputs.continue_msg;
-+
-+        data.generation_prompt = GEN_PROMPT + THINK_START + msg.reasoning_content;
-+        if (inputs.continue_final_message == COMMON_CHAT_CONTINUATION_CONTENT) {
-+            data.generation_prompt += THINK_END + msg.render_content();
-+        }
-+
-+        data.prompt += data.generation_prompt;
-+    }
-+
-+    auto parser = build_chat_peg_parser([&](common_chat_peg_builder & p) {
-+        auto generation_prompt = p.literal(GEN_PROMPT);
-+        auto end = p.end();
-+
-+        auto reasoning = p.eps();
-+        // M3 can emit a bare </mm:think> (no opener) after tool results; keep the opener optional.
-+        if (extract_reasoning && inputs.enable_thinking) {
-+            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.reasoning(p.until(THINK_END)) + THINK_END);
-+        } else if (extract_reasoning) {
-+            reasoning = p.optional(p.optional(p.literal(THINK_START)) + p.until(THINK_END) + p.literal(THINK_END));
-+        }
-+
-+        if (has_response_format) {
-+            auto response_format = p.rule("response-format",
-+                p.literal("```json") + p.space() +
-+                p.content(p.schema(p.json(), "response-format-schema", inputs.json_schema)) +
-+                p.space() + p.literal("```"));
-+            return generation_prompt + reasoning + response_format + end;
-+        }
-+
-+        if (!has_tools || inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_NONE) {
-+            return generation_prompt + reasoning + p.content(p.rest()) + end;
-+        }
-+
-+        auto tool_choice = p.choice();
-+        foreach_function(inputs.tools, [&](const json & tool) {
-+            const auto & function = tool.at("function");
-+            std::string  name     = function.at("name");
-+            auto params   = function.contains("parameters") ? function.at("parameters") : json::object();
-+            const auto & props    = params.contains("properties") ? params.at("properties") : json::object();
-+
-+            std::set<std::string> required;
-+            if (params.contains("required")) {
-+                params.at("required").get_to(required);
-+            }
-+
-+            auto schema_info = common_schema_info();
-+            schema_info.resolve_refs(params);
-+
-+            std::vector<common_peg_parser> required_parsers;
-+            std::vector<common_peg_parser> optional_parsers;
-+            for (const auto & [param_name, param_schema] : props.items()) {
-+                bool is_required = required.find(param_name) != required.end();
-+                bool is_string   = schema_info.resolves_to_string(param_schema);
-+
-+                const std::string p_close = NS + "</" + param_name + ">";
-+
-+                auto arg = p.tool_arg(
-+                    p.tool_arg_open(
-+                        p.literal(NS + "<") +
-+                        p.tool_arg_name(p.literal(param_name)) +
-+                        p.literal(">")) +
-+                    (is_string
-+                         ? p.ac(p.tool_arg_string_value(p.until(p_close)) +
-+                                p.tool_arg_close(p.literal(p_close)), p_close)
-+                         : p.tool_arg_json_value(p.schema(p.json(),
-+                                                          "tool-" + name + "-arg-" + param_name + "-schema",
-+                                                          param_schema, false)) +
-+                           p.tool_arg_close(p.literal(p_close))));
-+
-+                auto named_arg = p.rule("tool-" + name + "-arg-" + param_name, arg);
-+                if (is_required) {
-+                    required_parsers.push_back(named_arg);
-+                } else {
-+                    optional_parsers.push_back(named_arg);
-+                }
-+            }
-+
-+            common_peg_parser args_seq = p.eps();
-+            for (size_t i = 0; i < required_parsers.size(); i++) {
-+                if (i > 0) {
-+                    args_seq = args_seq + p.space();
-+                }
-+                args_seq = args_seq + required_parsers[i];
-+            }
-+
-+            if (!optional_parsers.empty()) {
-+                common_peg_parser any_opt = p.choice();
-+                for (const auto & opt : optional_parsers) {
-+                    any_opt |= opt;
-+                }
-+                args_seq = args_seq + p.repeat(p.space() + any_opt, 0, -1);
-+            }
-+
-+            common_peg_parser invoke_body = args_seq;
-+            auto func_parser = p.tool(
-+                p.tool_open(p.literal(NS + "<invoke name=\"") +
-+                            p.tool_name(p.literal(name)) + p.literal("\">")) +
-+                p.space() + invoke_body + p.space() +
-+                p.tool_close(p.literal(INVOKE_END)));
-+
-+            tool_choice |= p.rule("tool-" + name, func_parser);
-+        });
-+
-+        auto require_tools = inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED;
-+
-+        common_peg_parser tool_calls = p.eps();
-+        if (inputs.parallel_tool_calls) {
-+            tool_calls = p.trigger_rule("tool-call",
-+                p.literal(FC_START) + p.space() + tool_choice +
-+                p.zero_or_more(p.space() + tool_choice) + p.space() + p.literal(FC_END));
-+        } else {
-+            tool_calls = p.trigger_rule("tool-call",
-+                p.literal(FC_START) + p.space() + tool_choice + p.space() + p.literal(FC_END));
-+        }
-+
-+        if (!require_tools) {
-+            tool_calls = p.optional(tool_calls);
-+        }
-+
-+        auto content_before_tools = p.content(p.until(FC_START));
-+        return generation_prompt + reasoning + content_before_tools + tool_calls + end;
-+    });
-+
-+    data.parser = parser.save();
-+
-+    if (include_grammar) {
-+        data.grammar_lazy = !(has_response_format || (has_tools && inputs.tool_choice == COMMON_CHAT_TOOL_CHOICE_REQUIRED));
-+        data.grammar      = build_grammar([&](const common_grammar_builder & builder) {
-+            foreach_function(inputs.tools, [&](const json & tool) {
-+                const auto & function = tool.at("function");
-+                auto         schema   = function.contains("parameters") ? function.at("parameters") : json::object();
-+                builder.resolve_refs(schema);
-+            });
-+            if (has_response_format) {
-+                auto schema = inputs.json_schema;
-+                builder.resolve_refs(schema);
-+            }
-+            parser.build_grammar(builder, data.grammar_lazy);
-+        });
-+
-+        data.grammar_triggers = {
-+            { COMMON_GRAMMAR_TRIGGER_TYPE_WORD, FC_START },
-+        };
-+    }
-+
-+    return data;
-+}
-+
- // Cohere2 MoE (a.k.a. "North Code") parser.
- //
- // The assistant turn is fully marker-wrapped:
-@@ -2707,6 +2892,15 @@ std::optional<common_chat_params> common_chat_try_specialized_template(
-         return common_chat_params_init_gigachat_v3(tmpl, params);
-     }
- 
-+    // MiniMax-M3: the namespace token "]<]minimax[>[" collides with the autoparser's
-+    // markup delimiters, so detect the template and use a dedicated parser.
-+    if (src.find("]<]minimax[>[") != std::string::npos &&
-+        src.find("<tool_call>") != std::string::npos &&
-+        src.find("<invoke name=") != std::string::npos) {
-+        LOG_DBG("Using specialized template: MiniMax-M3\n");
-+        return common_chat_params_init_minimax_m3(tmpl, params);
-+    }
-+
-     // DeepSeek V3.2/V4 format detection: template defines dsml_token and uses it for tool calls.
-     // The template source contains the token as a variable assignment, not as a literal in markup.
-     // V3.2 names the tool call block "function_calls", V4 names it "tool_calls".

backend/cpp/llama-cpp/run.sh

@@ -12,10 +12,11 @@ grep -e "flags" /proc/cpuinfo | head -1
 
 BINARY=llama-cpp-fallback
 
-# CPU images (x86, arm64, darwin) ship a single llama-cpp-cpu-all built with ggml
+# CPU images and most x86 GPU images ship a single llama-cpp-cpu-all built with ggml
 # CPU_ALL_VARIANTS: ggml's backend registry dlopens the best libggml-cpu-*.so for this
-# host, so no shell-side AVX probing. GPU images (cublas/sycl/vulkan/hipblas) ship only
-# llama-cpp-fallback (the accelerator does the compute), so fall back to it when absent.
+# host, so no shell-side AVX probing. GPU arm64 images still ship llama-cpp-fallback
+# until their builder toolchains support ggml's complete arm variant matrix, and so do
+# the SYCL images, whose icpx compiler hangs on the sapphirerapids variant.
 if [ -e "$CURDIR"/llama-cpp-cpu-all ]; then
 	BINARY=llama-cpp-cpu-all
 fi
@@ -42,6 +43,27 @@ else
 	if [ -d "$CURDIR/lib/hipblaslt/library" ]; then
 		export HIPBLASLT_TENSILE_LIBPATH="$CURDIR"/lib/hipblaslt/library
 	fi
+	# Backends built for Intel GPUs carry a copy of the Intel graphics driver,
+	# and libze_loader is only there in those builds. Level Zero looks for a
+	# driver on its own, so point it at the copy that came with this backend: it
+	# was built against the same C library, while the machine's own driver may
+	# not have been, and loading that one can crash on start.
+	#
+	# Anything the user set is left alone, so a machine with a graphics card
+	# newer than the driver carried here can still be told to use its own.
+	# Nothing is said about OpenCL: no OpenCL driver is carried, so anything we
+	# set there would leave OpenCL worse off than the machine's own setup.
+	if [ -e "$CURDIR/lib/libze_loader.so.1" ]; then
+		if [ -e "$CURDIR/lib/libze_intel_gpu.so.1" ] && [ -z "${ZE_ENABLE_ALT_DRIVERS:-}" ]; then
+			export ZE_ENABLE_ALT_DRIVERS="$CURDIR"/lib/libze_intel_gpu.so.1
+		fi
+		# Ask the driver how much graphics memory is free. Without this,
+		# llama.cpp reads zero on an integrated graphics chip, because such a
+		# chip shares the system memory instead of having its own.
+		if [ -z "${ZES_ENABLE_SYSMAN:-}" ]; then
+			export ZES_ENABLE_SYSMAN=1
+		fi
+	fi
 fi
 
 # If there is a lib/ld.so, use it
@@ -55,4 +77,4 @@ echo "Using binary: $BINARY"
 exec "$CURDIR"/$BINARY "$@"
 
 # We should never reach this point, however just in case we do, run fallback
-exec "$CURDIR"/llama-cpp-fallback "$@"
+exec "$CURDIR"/llama-cpp-fallback "$@"

backend/cpp/turboquant/Makefile

@@ -1,7 +1,7 @@
 
 # Pinned to the HEAD of feature/turboquant-kv-cache on https://github.com/TheTom/llama-cpp-turboquant.
 # Auto-bumped nightly by .github/workflows/bump_deps.yaml.
-TURBOQUANT_VERSION?=c26cbdffcf6fc9b7430cd6b117757e9a3f70b7ea
+TURBOQUANT_VERSION?=8a891f4b566efdbd3cea92fafee3227a0a267683
 LLAMA_REPO?=https://github.com/TheTom/llama-cpp-turboquant
 
 CMAKE_ARGS?=

backend/cpp/turboquant/run.sh

@@ -12,9 +12,12 @@ grep -e "flags" /proc/cpuinfo | head -1
 
 BINARY=turboquant-fallback
 
-# x86/arm64 ship a single turboquant-cpu-all built with ggml CPU_ALL_VARIANTS: ggml's
+# CPU images and most x86 GPU images ship a single turboquant-cpu-all built with ggml
+# CPU_ALL_VARIANTS: ggml's
 # backend registry dlopens the best libggml-cpu-*.so for this host, so no shell-side
-# probing. ROCm ships only turboquant-fallback, so fall back to it when cpu-all is absent.
+# probing. GPU arm64 images still ship turboquant-fallback until their builder toolchains
+# support ggml's complete arm variant matrix, and so do the SYCL images, whose icpx
+# compiler hangs on the sapphirerapids variant.
 if [ -e "$CURDIR"/turboquant-cpu-all ]; then
 	BINARY=turboquant-cpu-all
 fi
@@ -40,6 +43,27 @@ else
 	if [ -d "$CURDIR/lib/hipblaslt/library" ]; then
 		export HIPBLASLT_TENSILE_LIBPATH="$CURDIR"/lib/hipblaslt/library
 	fi
+	# Backends built for Intel GPUs carry a copy of the Intel graphics driver,
+	# and libze_loader is only there in those builds. Level Zero looks for a
+	# driver on its own, so point it at the copy that came with this backend: it
+	# was built against the same C library, while the machine's own driver may
+	# not have been, and loading that one can crash on start.
+	#
+	# Anything the user set is left alone, so a machine with a graphics card
+	# newer than the driver carried here can still be told to use its own.
+	# Nothing is said about OpenCL: no OpenCL driver is carried, so anything we
+	# set there would leave OpenCL worse off than the machine's own setup.
+	if [ -e "$CURDIR/lib/libze_loader.so.1" ]; then
+		if [ -e "$CURDIR/lib/libze_intel_gpu.so.1" ] && [ -z "${ZE_ENABLE_ALT_DRIVERS:-}" ]; then
+			export ZE_ENABLE_ALT_DRIVERS="$CURDIR"/lib/libze_intel_gpu.so.1
+		fi
+		# Ask the driver how much graphics memory is free. Without this, the
+		# backend reads zero on an integrated graphics chip, because such a chip
+		# shares the system memory instead of having its own.
+		if [ -z "${ZES_ENABLE_SYSMAN:-}" ]; then
+			export ZES_ENABLE_SYSMAN=1
+		fi
+	fi
 fi
 
 # If there is a lib/ld.so, use it

backend/go/crispasr/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # CrispASR version (release tag)
 CRISPASR_REPO?=https://github.com/CrispStrobe/CrispASR
-CRISPASR_VERSION?=4e863bae52aa76a875e4aca57db54ae6d4145c5c
+CRISPASR_VERSION?=fcb79282a6bc52e13d858026c42b24fb6e63c97a
 SO_TARGET?=libgocrispasr.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/go/qwen3-tts-cpp/Makefile

@@ -7,6 +7,16 @@ GO_TAGS?=
 JOBS?=$(shell nproc --ignore=1)
 
 # qwentts.cpp version
+#
+# Held at 35ebe537 rather than tracking latest: abab6b3 hangs in synthesis.
+# TTS() never returns from the native call, so tests-qwen3-tts-cpp goes from
+# ~5 minutes to the 20 minute Go test timeout. Reproduced on master on
+# 2026-08-01 and again on re-run, and the bump PR (#11241) was merged with
+# this same check already red.
+#
+# The regression is in 35ebe537..abab6b3, three upstream commits whose only
+# functional change is 26dd8adb, "predictor: unroll the frame into one cgraph
+# and sample in standard ops". Restore the bump once that is fixed upstream.
 QWEN3TTS_REPO?=https://github.com/ServeurpersoCom/qwentts.cpp
 QWEN3TTS_CPP_VERSION?=35ebe5376b82a0a59d008586d55bbe623d449011
 SO_TARGET?=libgoqwen3ttscpp.so

backend/go/rfdetr-cpp/Makefile

@@ -11,7 +11,7 @@ JOBS?=$(shell nproc --ignore=1)
 # build; leaving this on `master` always picks up the latest C-API surface
 # (incl. the per-detection accessor functions used by gorfdetrcpp.go).
 RFDETR_REPO?=https://github.com/localai-org/rf-detr.cpp.git
-RFDETR_VERSION?=65c0ffcc9a9bc9dae38252f63d0417c9845a6cf7
+RFDETR_VERSION?=98d0f381b832ef08a608b65c7dd78db066ed8b9a
 
 ifeq ($(NATIVE),false)
 	CMAKE_ARGS+=-DGGML_NATIVE=OFF

backend/go/stablediffusion-ggml/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # stablediffusion.cpp (ggml)
 STABLEDIFFUSION_GGML_REPO?=https://github.com/leejet/stable-diffusion.cpp
-STABLEDIFFUSION_GGML_VERSION?=e92e86fb11b3028ac9edaf63d93709801d106b12
+STABLEDIFFUSION_GGML_VERSION?=db99efdd6d2a43c7937fd55b3359206c680a75b0
 
 CMAKE_ARGS+=-DGGML_MAX_NAME=128
 

backend/go/whisper/Makefile

@@ -8,7 +8,7 @@ JOBS?=$(shell nproc --ignore=1)
 
 # whisper.cpp version
 WHISPER_REPO?=https://github.com/ggml-org/whisper.cpp
-WHISPER_CPP_VERSION?=a630b35c6fc02c8879f751ec3f39a61327f01dc7
+WHISPER_CPP_VERSION?=2ca53bb45e38748d07b310eeb36245a7157ac882
 SO_TARGET?=libgowhisper.so
 
 CMAKE_ARGS+=-DBUILD_SHARED_LIBS=OFF

backend/python/diffusers/backend.py

@@ -883,6 +883,34 @@ class BackendServicer(backend_pb2_grpc.BackendServicer):
 
         return backend_pb2.Result(message="Media generated", success=True)
 
+    def UpscaleImage(self, request, context):
+        try:
+            if not request.src:
+                return backend_pb2.Result(success=False, message="No source image provided")
+            if not request.dst:
+                return backend_pb2.Result(success=False, message="No destination path provided")
+
+            scale = request.scale if request.scale > 0 else 2
+            image = Image.open(request.src).convert("RGB")
+
+            # If the loaded pipeline supports upscaling (e.g. StableDiffusionUpscalePipeline),
+            # use it; otherwise fall back to high-quality Lanczos resize.
+            if self.pipe is not None and self.PipelineType in ("StableDiffusionUpscalePipeline", "StableDiffusionLatentUpscalePipeline"):
+                print(f"UpscaleImage: using diffusers upscale pipeline ({self.PipelineType})", file=sys.stderr)
+                upscaled = self.pipe(prompt="", image=image).images[0]
+            else:
+                # Fallback: high-quality Lanczos resize
+                print(f"UpscaleImage: no upscale pipeline loaded, using Lanczos resize (scale={scale})", file=sys.stderr)
+                new_w = image.width * scale
+                new_h = image.height * scale
+                upscaled = image.resize((new_w, new_h), Image.LANCZOS)
+
+            upscaled.save(request.dst)
+            return backend_pb2.Result(message="Image upscaled", success=True)
+        except Exception as e:
+            print(f"UpscaleImage error: {e}", file=sys.stderr)
+            return backend_pb2.Result(success=False, message=str(e))
+
     def GenerateVideo(self, request, context):
         try:
             prompt = request.prompt

backend/python/sglang/requirements-cublas12-after.txt

@@ -15,3 +15,12 @@ sglang[all]>=0.5.11
 # load-bearing for flash-attn-4, and this is the narrower change. Raise the
 # bound once 0.46.0 final ships.
 nvidia-modelopt<0.46
+
+# Same failure mode as the nvidia-modelopt bound above, via a different
+# package. sglang -> flashinfer-python -> cuda-tile, unbounded, and the
+# global --prerelease=allow resolves it to 1.6.0rc3, whose build backend
+# imports wheel_stub without declaring it in build-system.requires. With
+# --no-build-isolation nothing installs it and the build dies with
+# "No module named 'wheel_stub'". 1.5.0 is the newest stable release.
+# Raise the bound once 1.6.0 final ships.
+cuda-tile<1.6

backend/python/sglang/requirements-cublas13-after.txt

@@ -15,3 +15,12 @@ sglang[all]>=0.5.11
 # load-bearing for flash-attn-4, and this is the narrower change. Raise the
 # bound once 0.46.0 final ships.
 nvidia-modelopt<0.46
+
+# Same failure mode as the nvidia-modelopt bound above, via a different
+# package. sglang -> flashinfer-python -> cuda-tile, unbounded, and the
+# global --prerelease=allow resolves it to 1.6.0rc3, whose build backend
+# imports wheel_stub without declaring it in build-system.requires. With
+# --no-build-isolation nothing installs it and the build dies with
+# "No module named 'wheel_stub'". 1.5.0 is the newest stable release.
+# Raise the bound once 1.6.0 final ships.
+cuda-tile<1.6

backend/python/sglang/requirements-l4t13-after.txt

@@ -13,3 +13,12 @@
 # FunctionCallParser, ReasoningParser); the [all] extras are optional
 # accelerators not required at import time.
 sglang>=0.5.11
+
+# Same failure mode the cublas profiles carry an nvidia-modelopt bound for,
+# reached through a different package. sglang -> flashinfer-python ->
+# cuda-tile, unbounded, and the global --prerelease=allow resolves it to
+# 1.6.0rc3, whose build backend imports wheel_stub without declaring it in
+# build-system.requires. With --no-build-isolation nothing installs it and
+# the build dies with "No module named 'wheel_stub'". 1.5.0 is the newest
+# stable release. Raise the bound once 1.6.0 final ships.
+cuda-tile<1.6

cmd/local-ai/main.go

@@ -1,6 +1,7 @@
 package main
 
 import (
+	"errors"
 	"os"
 	"path/filepath"
 
@@ -107,6 +108,13 @@ For documentation and support:
 	// Run the thing!
 	err = ctx.Run(&cli.CLI.Context)
 	if err != nil {
+		// A command that has already told the user what went wrong returns
+		// only a status. Logging it as well would print a bare "exit status 1"
+		// underneath the explanation they just read.
+		var reported cli.ExitCodeError
+		if errors.As(err, &reported) {
+			os.Exit(reported.Code)
+		}
 		xlog.Fatal("Error running the application", "error", err)
 	}
 }

core/application/application.go

@@ -553,12 +553,17 @@ func (a *Application) start() error {
 	// once at startup and reused across chat sessions that opt in via metadata.
 	if !a.applicationConfig.DisableLocalAIAssistant {
 		holder := mcpTools.NewLocalAIAssistantHolder()
+		var nodeRegistry *nodes.NodeRegistry
+		if a.distributed != nil {
+			nodeRegistry = a.distributed.Registry
+		}
 		assistantClient := localaiInproc.New(
 			a.applicationConfig,
 			a.applicationConfig.SystemState,
 			a.backendLoader,
 			a.modelLoader,
 			a.galleryService,
+			nodeRegistry,
 		)
 		// Wire usage tracking so the assistant's get_usage_stats tool
 		// returns real data; nil values keep the tool returning a clear

core/application/startup.go

@@ -444,6 +444,13 @@ func New(opts ...config.AppOption) (*Application, error) {
 	// when gallery data refreshes instead of using a fixed TTL.
 	vram.SetGalleryGenerationFunc(gallery.GalleryGeneration)
 
+	// Fill those caches ahead of the first visitor. An estimate for an entry
+	// nobody has asked about yet costs a remote probe of its weight files, and
+	// the model gallery asks for one per row, so without this the first page
+	// spends seconds filling in its own sizes while somebody watches it.
+	// Non-blocking, and bounded: see DefaultEstimateWarmConfig.
+	gallery.WarmEstimateCache(options.Context, options.Galleries, options.SystemState, gallery.EstimateWarmConfigFromEnv())
+
 	if options.ConfigFile != "" {
 		if err := application.ModelConfigLoader().LoadMultipleModelConfigsSingleFile(options.ConfigFile, configLoaderOpts...); err != nil {
 			xlog.Error("error loading config file", "error", err)

core/backend/upscale.go

@@ -0,0 +1,37 @@
+package backend
+
+import (
+	"context"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/pkg/grpc/proto"
+	model "github.com/mudler/LocalAI/pkg/model"
+)
+
+// ImageUpscale loads the model specified in modelConfig and calls UpscaleImage
+// on the backend, writing the result to dst.
+func ImageUpscale(ctx context.Context, src, dst string, scale int, loader *model.ModelLoader, modelConfig config.ModelConfig, appConfig *config.ApplicationConfig) (func() error, error) {
+	opts := ModelOptions(modelConfig, appConfig, model.WithContext(ctx))
+	inferenceModel, err := loader.Load(opts...)
+	if err != nil {
+		recordModelLoadFailure(appConfig, modelConfig.Name, modelConfig.Backend, err, nil)
+		return nil, err
+	}
+
+	fn := func() error {
+		_, err := inferenceModel.UpscaleImage(
+			ctx,
+			&proto.UpscaleImageRequest{
+				Src:   src,
+				Dst:   dst,
+				Scale: int32(scale),
+			},
+		)
+		return err
+	}
+
+	return fn, nil
+}
+
+// ImageUpscaleFunc is a test-friendly indirection.
+var ImageUpscaleFunc = ImageUpscale

core/cli/chat/chat.go

@@ -1,30 +0,0 @@
-package chat
-
-import (
-	"context"
-	"io"
-	"strings"
-)
-
-type Options struct {
-	Model   string
-	BaseURL string
-	APIKey  string
-	In      io.Reader
-	Out     io.Writer
-}
-
-func Run(ctx context.Context, opts Options) error {
-	if opts.In == nil {
-		opts.In = strings.NewReader("")
-	}
-	if opts.Out == nil {
-		opts.Out = io.Discard
-	}
-
-	session, err := newChatSession(ctx, newLocalAIChatClient(opts.BaseURL, opts.APIKey), opts.Model)
-	if err != nil {
-		return err
-	}
-	return runTerminalChat(ctx, session, opts.In, opts.Out)
-}

core/cli/chat/chat_test.go

@@ -1,172 +0,0 @@
-package chat
-
-import (
-	"bytes"
-	"encoding/json"
-	"fmt"
-	"io"
-	"net/http"
-	"net/http/httptest"
-	"strings"
-
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-var _ = Describe("Run chat", func() {
-	It("streams a single chat response", func() {
-		var capturedModel string
-		var capturedAuth string
-		server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-			if r.URL.Path == "/v1/models" {
-				w.Header().Set("Content-Type", "application/json")
-				writeResponse(w, `{"object":"list","data":[{"id":"test-model","object":"model"}]}`)
-				return
-			}
-
-			Expect(r.URL.Path).To(Equal("/v1/chat/completions"))
-			capturedAuth = r.Header.Get("Authorization")
-
-			var body struct {
-				Model    string `json:"model"`
-				Messages []struct {
-					Role    string `json:"role"`
-					Content string `json:"content"`
-				} `json:"messages"`
-			}
-			Expect(json.NewDecoder(r.Body).Decode(&body)).To(Succeed())
-			capturedModel = body.Model
-			Expect(body.Messages).To(HaveLen(1))
-			Expect(body.Messages[0].Role).To(Equal("user"))
-			Expect(body.Messages[0].Content).To(Equal("hello"))
-
-			w.Header().Set("Content-Type", "text/event-stream")
-			writeResponse(w, "data: {\"choices\":[{\"index\":0,\"delta\":{\"content\":\"hi\"}}]}\n\n")
-			writeResponse(w, "data: {\"choices\":[{\"index\":0,\"delta\":{\"content\":\"!\"}}]}\n\n")
-			writeResponse(w, "data: [DONE]\n\n")
-		}))
-		defer server.Close()
-
-		var out bytes.Buffer
-		err := Run(GinkgoT().Context(), Options{
-			Model:   "test-model",
-			BaseURL: server.URL + "/v1",
-			APIKey:  "secret",
-			In:      strings.NewReader("hello\n/exit\n"),
-			Out:     &out,
-		})
-
-		Expect(err).ToNot(HaveOccurred())
-		Expect(capturedModel).To(Equal("test-model"))
-		Expect(capturedAuth).To(Equal("Bearer secret"))
-		Expect(out.String()).To(ContainSubstring("assistant: hi!"))
-		Expect(out.String()).To(ContainSubstring("bye"))
-	})
-
-	It("auto-selects the only available model", func() {
-		server := chatTestServer([]string{"solo"}, nil)
-		defer server.Close()
-
-		var out bytes.Buffer
-		err := Run(GinkgoT().Context(), Options{
-			BaseURL: server.URL + "/v1",
-			In:      strings.NewReader("/exit\n"),
-			Out:     &out,
-		})
-
-		Expect(err).ToNot(HaveOccurred())
-		Expect(out.String()).To(ContainSubstring("LocalAI chat (solo)"))
-	})
-
-	It("returns an actionable error when no models are installed", func() {
-		server := chatTestServer(nil, nil)
-		defer server.Close()
-
-		err := Run(GinkgoT().Context(), Options{
-			BaseURL: server.URL + "/v1",
-			In:      strings.NewReader(""),
-		})
-
-		Expect(err).To(HaveOccurred())
-		Expect(err.Error()).To(ContainSubstring("no chat models are installed"))
-		Expect(err.Error()).To(ContainSubstring("local-ai models install <model>"))
-	})
-
-	It("returns an actionable error when multiple models are available without a selection", func() {
-		server := chatTestServer([]string{"alpha", "beta"}, nil)
-		defer server.Close()
-
-		err := Run(GinkgoT().Context(), Options{
-			BaseURL: server.URL + "/v1",
-			In:      strings.NewReader(""),
-		})
-
-		Expect(err).To(HaveOccurred())
-		Expect(err.Error()).To(ContainSubstring("multiple models are available"))
-		Expect(err.Error()).To(ContainSubstring("--model"))
-		Expect(err.Error()).To(ContainSubstring("alpha"))
-		Expect(err.Error()).To(ContainSubstring("beta"))
-	})
-
-	It("lists and switches models inside the chat", func() {
-		requestedModels := []string{}
-		server := chatTestServer([]string{"alpha", "beta"}, func(model string) {
-			requestedModels = append(requestedModels, model)
-		})
-		defer server.Close()
-
-		var out bytes.Buffer
-		err := Run(GinkgoT().Context(), Options{
-			Model:   "alpha",
-			BaseURL: server.URL + "/v1",
-			In:      strings.NewReader("/models\n/model beta\nhello\n/exit\n"),
-			Out:     &out,
-		})
-
-		Expect(err).ToNot(HaveOccurred())
-		Expect(out.String()).To(ContainSubstring("* alpha"))
-		Expect(out.String()).To(ContainSubstring("  beta"))
-		Expect(out.String()).To(ContainSubstring("switched to beta; conversation cleared"))
-		Expect(requestedModels).To(Equal([]string{"beta"}))
-	})
-})
-
-func chatTestServer(models []string, onChat func(model string)) *httptest.Server {
-	return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		switch r.URL.Path {
-		case "/v1/models":
-			w.Header().Set("Content-Type", "application/json")
-			writeResponse(w, `{"object":"list","data":[`)
-			for i, model := range models {
-				if i > 0 {
-					writeResponse(w, ",")
-				}
-				writeResponsef(w, `{"id":%q,"object":"model"}`, model)
-			}
-			writeResponse(w, `]}`)
-		case "/v1/chat/completions":
-			var body struct {
-				Model string `json:"model"`
-			}
-			Expect(json.NewDecoder(r.Body).Decode(&body)).To(Succeed())
-			if onChat != nil {
-				onChat(body.Model)
-			}
-			w.Header().Set("Content-Type", "text/event-stream")
-			writeResponse(w, "data: {\"choices\":[{\"index\":0,\"delta\":{\"content\":\"ok\"}}]}\n\n")
-			writeResponse(w, "data: [DONE]\n\n")
-		default:
-			w.WriteHeader(http.StatusNotFound)
-		}
-	}))
-}
-
-func writeResponse(w io.Writer, text string) {
-	_, err := fmt.Fprint(w, text)
-	Expect(err).ToNot(HaveOccurred())
-}
-
-func writeResponsef(w io.Writer, format string, args ...any) {
-	_, err := fmt.Fprintf(w, format, args...)
-	Expect(err).ToNot(HaveOccurred())
-}

core/cli/chat/client.go

@@ -1,114 +0,0 @@
-package chat
-
-import (
-	"context"
-	"errors"
-	"fmt"
-	"io"
-	"sort"
-	"strings"
-
-	openai "github.com/sashabaranov/go-openai"
-)
-
-type chatClient interface {
-	ListModels(ctx context.Context) ([]string, error)
-	StreamChat(ctx context.Context, model string, messages []chatMessage, out io.Writer) (string, error)
-}
-
-type localAIChatClient struct {
-	client *openai.Client
-}
-
-func newLocalAIChatClient(baseURL string, apiKey string) *localAIChatClient {
-	cfg := openai.DefaultConfig(apiKey)
-	cfg.BaseURL = baseURL
-	return &localAIChatClient{client: openai.NewClientWithConfig(cfg)}
-}
-
-func (c *localAIChatClient) ListModels(ctx context.Context) ([]string, error) {
-	resp, err := c.client.ListModels(ctx)
-	if err != nil {
-		return nil, err
-	}
-
-	models := make([]string, 0, len(resp.Models))
-	for _, model := range resp.Models {
-		if model.ID != "" {
-			models = append(models, model.ID)
-		}
-	}
-	sort.Strings(models)
-	return models, nil
-}
-
-func (c *localAIChatClient) StreamChat(ctx context.Context, model string, messages []chatMessage, out io.Writer) (string, error) {
-	stream, err := c.client.CreateChatCompletionStream(ctx, openai.ChatCompletionRequest{
-		Model:    model,
-		Messages: openAIChatMessages(messages),
-	})
-	if err != nil {
-		return "", friendlyChatError(err, model)
-	}
-	defer func() {
-		_ = stream.Close()
-	}()
-
-	var answer strings.Builder
-	for {
-		resp, err := stream.Recv()
-		if errors.Is(err, io.EOF) {
-			break
-		}
-		if err != nil {
-			return answer.String(), friendlyChatError(err, model)
-		}
-		if len(resp.Choices) == 0 {
-			continue
-		}
-
-		token := resp.Choices[0].Delta.Content
-		if token == "" {
-			continue
-		}
-		answer.WriteString(token)
-		if _, err := fmt.Fprint(out, token); err != nil {
-			return answer.String(), err
-		}
-	}
-
-	return answer.String(), nil
-}
-
-func openAIChatMessages(messages []chatMessage) []openai.ChatCompletionMessage {
-	converted := make([]openai.ChatCompletionMessage, len(messages))
-	for i, message := range messages {
-		converted[i] = openai.ChatCompletionMessage{
-			Role:    message.Role,
-			Content: message.Content,
-		}
-	}
-	return converted
-}
-
-func friendlyChatError(err error, model string) error {
-	var apiErr *openai.APIError
-	if errors.As(err, &apiErr) {
-		switch apiErr.HTTPStatusCode {
-		case 404:
-			return fmt.Errorf("model %q is not available. Run `local-ai models list`, install a model with `local-ai models install <model>`, or switch with `/model <name>`", model)
-		case 403:
-			return fmt.Errorf("model %q is disabled. Enable it from LocalAI settings or choose another model with `/model <name>`", model)
-		}
-		if apiErr.Message != "" {
-			return errors.New(apiErr.Message)
-		}
-	}
-
-	msg := err.Error()
-	if strings.Contains(msg, "model") && strings.Contains(msg, "not found") {
-		return fmt.Errorf("model %q is not available. Run `local-ai models list`, install a model with `local-ai models install <model>`, or switch with `/model <name>`", model)
-	}
-
-	return err
-}

core/cli/chat/models.go

@@ -1,17 +0,0 @@
-package chat
-
-import "strings"
-
-func formatChatModelList(models []string, current string) string {
-	var b strings.Builder
-	for _, model := range models {
-		prefix := "  "
-		if model == current {
-			prefix = "* "
-		}
-		b.WriteString(prefix)
-		b.WriteString(model)
-		b.WriteByte('\n')
-	}
-	return b.String()
-}

core/cli/chat/paths.go

@@ -0,0 +1,153 @@
+package chat
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+
+	"gopkg.in/yaml.v3"
+)
+
+// stateDirMode matches the mode nib uses for the same directory. The directory
+// holds an API key, so it stays owner-only.
+const stateDirMode = 0o700
+
+// configFileMode keeps the config owner-only: nib stores the user's API key in
+// it alongside the keys written here.
+const configFileMode = 0o600
+
+// StateDir resolves where the chat agent keeps its config, plugins, and
+// skills. This is user-scoped rather than server-scoped: chat is a client that
+// may target a remote LocalAI, so it does not belong under LOCALAI_CONFIG_DIR.
+func StateDir(override string) (string, error) {
+	if override != "" {
+		return override, nil
+	}
+	if xdg := os.Getenv("XDG_CONFIG_HOME"); xdg != "" {
+		return filepath.Join(xdg, "localai", "chat"), nil
+	}
+	home, err := os.UserHomeDir()
+	if err != nil {
+		return "", fmt.Errorf("resolving home directory for the agent state dir: %w", err)
+	}
+	return filepath.Join(home, ".config", "localai", "chat"), nil
+}
+
+// ConfigPath is the agent's config file inside dir.
+func ConfigPath(dir string) string { return filepath.Join(dir, "config.yaml") }
+
+// EnsureStateDir creates dir and, on first run only, seeds a config file
+// pointing at baseURL. It deliberately does not seed a model: a baked-in model
+// name goes stale as soon as the user installs a different one.
+//
+// The config file is machine-managed from here on: nib rewrites it whenever it
+// self-configures, so hand-written comments in it do not survive.
+func EnsureStateDir(dir, baseURL string) error {
+	if err := os.MkdirAll(dir, stateDirMode); err != nil {
+		return fmt.Errorf("creating agent state dir %s: %w", dir, err)
+	}
+	path := ConfigPath(dir)
+	if _, err := os.Stat(path); err == nil {
+		return nil // already configured; never overwrite the user's file
+	} else if !os.IsNotExist(err) {
+		return fmt.Errorf("checking agent config %s: %w", path, err)
+	}
+
+	seed := map[string]string{"base_url": baseURL}
+	data, err := yaml.Marshal(seed)
+	if err != nil {
+		return fmt.Errorf("encoding seed agent config: %w", err)
+	}
+	if err := writeConfigFile(path, data); err != nil {
+		return fmt.Errorf("writing seed agent config: %w", err)
+	}
+	return nil
+}
+
+// PersistModel records the chosen model in the agent config, preserving every
+// other key the user may have set, including the api_key nib writes there.
+//
+// The file is machine-managed: this overlays the model onto the parsed keys and
+// re-marshals, which drops comments. That is deliberate rather than an
+// oversight, because nib's own save path does the same thing and would erase
+// them on its next write regardless.
+func PersistModel(dir, model string) error {
+	// PersistModel is callable before EnsureStateDir, so it cannot assume the
+	// directory exists.
+	if err := os.MkdirAll(dir, stateDirMode); err != nil {
+		return fmt.Errorf("creating agent state dir %s: %w", dir, err)
+	}
+	path := ConfigPath(dir)
+
+	values := map[string]any{}
+	// #nosec G304 -- path is the fixed config.yaml name under the user-selected
+	// chat state directory; selecting that directory is the documented override.
+	data, err := os.ReadFile(path)
+	if err != nil && !os.IsNotExist(err) {
+		return fmt.Errorf("reading agent config %s: %w", path, err)
+	}
+	if err == nil {
+		if err := yaml.Unmarshal(data, &values); err != nil {
+			return fmt.Errorf("parsing agent config %s: %w", path, err)
+		}
+	}
+	values["model"] = model
+
+	out, err := yaml.Marshal(values)
+	if err != nil {
+		return fmt.Errorf("encoding agent config: %w", err)
+	}
+	if err := writeConfigFile(path, out); err != nil {
+		return fmt.Errorf("writing agent config: %w", err)
+	}
+	return nil
+}
+
+// writeConfigFile replaces path with data atomically: it writes a temporary
+// file next to the target and renames it over the target. Writing the target in
+// place would truncate it first, so an interrupted or out-of-disk write would
+// leave a half-written config and destroy the api_key nib keeps in the same
+// file. The temporary file must share the directory because rename is only
+// atomic within one filesystem.
+func writeConfigFile(path string, data []byte) error {
+	dir := filepath.Dir(path)
+
+	// A randomized name rather than a fixed config.yaml.tmp, so two concurrent
+	// writers cannot corrupt each other's temporary file.
+	tmp, err := os.CreateTemp(dir, "config.yaml.*.tmp")
+	if err != nil {
+		return fmt.Errorf("creating temp file in %s: %w", dir, err)
+	}
+	tmpPath := tmp.Name()
+	renamed := false
+	defer func() {
+		if !renamed {
+			// Leave no litter behind on any failure path.
+			_ = os.Remove(tmpPath)
+		}
+	}()
+
+	if _, err := tmp.Write(data); err != nil {
+		_ = tmp.Close()
+		return fmt.Errorf("writing %s: %w", tmpPath, err)
+	}
+	// Flush before the rename: renaming a file whose contents are still only in
+	// the page cache can still lose them across a crash.
+	if err := tmp.Sync(); err != nil {
+		_ = tmp.Close()
+		return fmt.Errorf("syncing %s: %w", tmpPath, err)
+	}
+	if err := tmp.Close(); err != nil {
+		return fmt.Errorf("closing %s: %w", tmpPath, err)
+	}
+	// CreateTemp already asks for 0600, but the umask can only ever clear bits,
+	// so set the mode explicitly rather than inheriting whatever survived.
+	if err := os.Chmod(tmpPath, configFileMode); err != nil {
+		return fmt.Errorf("setting mode on %s: %w", tmpPath, err)
+	}
+	if err := os.Rename(tmpPath, path); err != nil {
+		return fmt.Errorf("replacing %s: %w", path, err)
+	}
+	renamed = true
+	return nil
+}

core/cli/chat/paths_test.go

@@ -0,0 +1,186 @@
+package chat
+
+import (
+	"os"
+	"path/filepath"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	"gopkg.in/yaml.v3"
+)
+
+// richConfig stands in for a config nib has already taken ownership of: a
+// comment, a secret, and a nested block. A flat scalar alone would not catch a
+// writer that mangles structure or drops a key it does not know about.
+const richConfig = `# hand written note
+base_url: http://x.invalid/v1
+api_key: secret-token
+mcp_servers:
+  files:
+    command: mcp-files
+    args:
+      - --root
+      - /tmp
+`
+
+var _ = Describe("Agent state directory", func() {
+	Describe("StateDir", func() {
+		It("prefers an explicit override", func() {
+			Expect(StateDir("/custom/dir")).To(Equal("/custom/dir"))
+		})
+
+		It("uses XDG_CONFIG_HOME when set", func() {
+			tmp := GinkgoT().TempDir()
+			GinkgoT().Setenv("XDG_CONFIG_HOME", tmp)
+			Expect(StateDir("")).To(Equal(filepath.Join(tmp, "localai", "chat")))
+		})
+
+		It("falls back to ~/.config/localai/chat", func() {
+			tmp := GinkgoT().TempDir()
+			GinkgoT().Setenv("XDG_CONFIG_HOME", "")
+			GinkgoT().Setenv("HOME", tmp)
+			Expect(StateDir("")).To(Equal(filepath.Join(tmp, ".config", "localai", "chat")))
+		})
+
+		It("fails when neither XDG_CONFIG_HOME nor a home directory is resolvable", func() {
+			GinkgoT().Setenv("XDG_CONFIG_HOME", "")
+			GinkgoT().Setenv("HOME", "")
+
+			dir, err := StateDir("")
+			Expect(err).To(HaveOccurred())
+			Expect(err.Error()).To(ContainSubstring("agent state dir"))
+			// No silent fallback to a relative path: writing an API key into the
+			// working directory would be worse than refusing.
+			Expect(dir).To(BeEmpty())
+		})
+	})
+
+	Describe("EnsureStateDir", func() {
+		It("creates the directory and seeds base_url on first run", func() {
+			dir := filepath.Join(GinkgoT().TempDir(), "chat")
+			Expect(EnsureStateDir(dir, "http://127.0.0.1:8080/v1")).To(Succeed())
+
+			data, err := os.ReadFile(ConfigPath(dir))
+			Expect(err).ToNot(HaveOccurred())
+			Expect(string(data)).To(ContainSubstring("base_url: http://127.0.0.1:8080/v1"))
+			// A model must NOT be seeded: it goes stale as soon as the user
+			// installs a different one.
+			Expect(string(data)).ToNot(ContainSubstring("model:"))
+		})
+
+		It("keeps the seeded config and its directory owner-only", func() {
+			dir := filepath.Join(GinkgoT().TempDir(), "chat")
+			Expect(EnsureStateDir(dir, "http://127.0.0.1:8080/v1")).To(Succeed())
+
+			// nib writes the user's api_key into this same file, so the modes are
+			// load-bearing, not cosmetic.
+			config, err := os.Stat(ConfigPath(dir))
+			Expect(err).ToNot(HaveOccurred())
+			Expect(config.Mode().Perm()).To(Equal(os.FileMode(0o600)))
+
+			state, err := os.Stat(dir)
+			Expect(err).ToNot(HaveOccurred())
+			Expect(state.Mode().Perm()).To(Equal(os.FileMode(0o700)))
+		})
+
+		It("leaves an existing config byte-for-byte untouched", func() {
+			dir := GinkgoT().TempDir()
+			Expect(os.WriteFile(ConfigPath(dir), []byte(richConfig), 0o600)).To(Succeed())
+
+			Expect(EnsureStateDir(dir, "http://127.0.0.1:8080/v1")).To(Succeed())
+
+			data, err := os.ReadFile(ConfigPath(dir))
+			Expect(err).ToNot(HaveOccurred())
+			// Byte-exact against a fixture carrying a comment and a nested block:
+			// an implementation that "preserves" by re-marshaling through a map
+			// fails here rather than passing on a flat scalar.
+			Expect(string(data)).To(Equal(richConfig))
+		})
+	})
+
+	Describe("PersistModel", func() {
+		It("adds a model to an existing config, preserving other keys", func() {
+			dir := GinkgoT().TempDir()
+			Expect(os.WriteFile(ConfigPath(dir), []byte("base_url: http://x.invalid/v1\n"), 0o600)).To(Succeed())
+
+			Expect(PersistModel(dir, "chosen-model")).To(Succeed())
+
+			data, err := os.ReadFile(ConfigPath(dir))
+			Expect(err).ToNot(HaveOccurred())
+			Expect(string(data)).To(ContainSubstring("base_url: http://x.invalid/v1"))
+			Expect(string(data)).To(ContainSubstring("model: chosen-model"))
+		})
+
+		It("replaces an existing model rather than duplicating the key", func() {
+			dir := GinkgoT().TempDir()
+			Expect(os.WriteFile(ConfigPath(dir), []byte("model: old\nbase_url: http://x.invalid/v1\n"), 0o600)).To(Succeed())
+
+			Expect(PersistModel(dir, "new")).To(Succeed())
+
+			data, err := os.ReadFile(ConfigPath(dir))
+			Expect(err).ToNot(HaveOccurred())
+			Expect(string(data)).To(ContainSubstring("model: new"))
+			Expect(string(data)).ToNot(ContainSubstring("model: old"))
+		})
+
+		It("preserves secrets and nested blocks it does not understand", func() {
+			dir := GinkgoT().TempDir()
+			Expect(os.WriteFile(ConfigPath(dir), []byte(richConfig), 0o600)).To(Succeed())
+
+			Expect(PersistModel(dir, "chosen-model")).To(Succeed())
+
+			data, err := os.ReadFile(ConfigPath(dir))
+			Expect(err).ToNot(HaveOccurred())
+
+			var got map[string]any
+			Expect(yaml.Unmarshal(data, &got)).To(Succeed())
+			Expect(got).To(HaveKeyWithValue("model", "chosen-model"))
+			Expect(got).To(HaveKeyWithValue("base_url", "http://x.invalid/v1"))
+			// Losing this key logs the user out of their own server.
+			Expect(got).To(HaveKeyWithValue("api_key", "secret-token"))
+			Expect(got).To(HaveKeyWithValue("mcp_servers",
+				HaveKeyWithValue("files", And(
+					HaveKeyWithValue("command", "mcp-files"),
+					HaveKeyWithValue("args", ConsistOf("--root", "/tmp")),
+				)),
+			))
+
+			// Documented, accepted behavior rather than an aspiration: the overlay
+			// re-marshals, so comments do not survive. nib's own save path erases
+			// them too, so preserving them here would buy nothing.
+			Expect(string(data)).ToNot(ContainSubstring("# hand written note"))
+		})
+
+		It("keeps the rewritten config owner-only and leaves no temp file behind", func() {
+			dir := GinkgoT().TempDir()
+			Expect(os.WriteFile(ConfigPath(dir), []byte(richConfig), 0o600)).To(Succeed())
+
+			Expect(PersistModel(dir, "chosen-model")).To(Succeed())
+
+			info, err := os.Stat(ConfigPath(dir))
+			Expect(err).ToNot(HaveOccurred())
+			Expect(info.Mode().Perm()).To(Equal(os.FileMode(0o600)))
+
+			// The atomic write stages through a sibling temp file; it must not
+			// survive a successful write.
+			entries, err := os.ReadDir(dir)
+			Expect(err).ToNot(HaveOccurred())
+			names := []string{}
+			for _, entry := range entries {
+				names = append(names, entry.Name())
+			}
+			Expect(names).To(ConsistOf("config.yaml"))
+		})
+
+		It("creates the state directory when it does not exist yet", func() {
+			// Task 4 may persist a picked model before anything else has run.
+			dir := filepath.Join(GinkgoT().TempDir(), "chat")
+
+			Expect(PersistModel(dir, "chosen-model")).To(Succeed())
+
+			data, err := os.ReadFile(ConfigPath(dir))
+			Expect(err).ToNot(HaveOccurred())
+			Expect(string(data)).To(ContainSubstring("model: chosen-model"))
+		})
+	})
+})

core/cli/chat/probe.go

@@ -0,0 +1,86 @@
+package chat
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"net/http"
+	"net/url"
+
+	openai "github.com/sashabaranov/go-openai"
+)
+
+var (
+	// ErrUnreachable means nothing answered at the endpoint. Callers use this
+	// to decide whether offering to start a server makes sense.
+	ErrUnreachable = errors.New("no LocalAI server reachable")
+	// ErrUnauthorized means the server answered but rejected the credentials.
+	ErrUnauthorized = errors.New("LocalAI server rejected the API key")
+)
+
+// Probe lists the models the endpoint advertises. It classifies the two
+// failures that need different advice: nothing listening, and bad credentials.
+//
+// The returned list is what the server advertises, verbatim and in server
+// order. LocalAI happily lists non-model entries it finds in the models
+// directory (stray archives, dotfiles), and guessing which advertised IDs are
+// real belongs to whoever presents them, not here.
+func Probe(ctx context.Context, baseURL, apiKey string) ([]string, error) {
+	cfg := openai.DefaultConfig(apiKey)
+	cfg.BaseURL = baseURL
+
+	resp, err := openai.NewClientWithConfig(cfg).ListModels(ctx)
+	if err != nil {
+		if status, answered := responseStatus(err); answered {
+			if status == http.StatusUnauthorized || status == http.StatusForbidden {
+				return nil, fmt.Errorf("%w: %w", ErrUnauthorized, err)
+			}
+			// The server answered, so it is up; surface its error as-is.
+			return nil, fmt.Errorf("listing models at %s: %w", baseURL, err)
+		}
+		// A caller who cancelled the probe learned nothing about the endpoint,
+		// so claiming it is unreachable would send them to fix a server that
+		// may be fine. A deadline is left alone: an endpoint that cannot answer
+		// within the probe's budget is unreachable for our purposes.
+		var urlErr *url.Error
+		if errors.As(err, &urlErr) && !errors.Is(err, context.Canceled) {
+			// Only a failure to complete the round trip means nothing is
+			// listening. A reply we could not parse is a different problem,
+			// so it falls through to the generic error below.
+			return nil, fmt.Errorf("%w at %s: %w", ErrUnreachable, baseURL, err)
+		}
+		return nil, fmt.Errorf("listing models at %s: %w", baseURL, err)
+	}
+
+	models := make([]string, 0, len(resp.Models))
+	for _, m := range resp.Models {
+		if m.ID != "" {
+			models = append(models, m.ID)
+		}
+	}
+	return models, nil
+}
+
+// responseStatus reports the HTTP status a failed call came back with, and
+// whether there was one at all.
+//
+// go-openai splits this across two types depending on the error body, and both
+// occur against a real LocalAI: it returns *openai.APIError when the body
+// parses as an OpenAI error envelope, which is what LocalAI's normal error
+// handler sends, and *openai.RequestError when it does not, which is what
+// LocalAI sends when started with opaque errors, since that handler replies
+// with a bare status and no body.
+func responseStatus(err error) (int, bool) {
+	// *RequestError is checked first because it is the outer type when
+	// go-openai nests one error inside the other; the inner value in that case
+	// carries no status.
+	var reqErr *openai.RequestError
+	if errors.As(err, &reqErr) {
+		return reqErr.HTTPStatusCode, true
+	}
+	var apiErr *openai.APIError
+	if errors.As(err, &apiErr) {
+		return apiErr.HTTPStatusCode, true
+	}
+	return 0, false
+}

core/cli/chat/probe_test.go

@@ -0,0 +1,169 @@
+package chat
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"time"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("Probe", func() {
+	It("returns the advertised models", func() {
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			w.Header().Set("Content-Type", "application/json")
+			Expect(json.NewEncoder(w).Encode(map[string]any{
+				"object": "list",
+				"data": []map[string]string{
+					{"id": "model-a", "object": "model"},
+					{"id": "model-b", "object": "model"},
+				},
+			})).To(Succeed())
+		}))
+		defer srv.Close()
+
+		models, err := Probe(context.Background(), srv.URL+"/v1", "")
+		Expect(err).ToNot(HaveOccurred())
+		Expect(models).To(Equal([]string{"model-a", "model-b"}))
+	})
+
+	It("reports an unreachable server distinguishably", func() {
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}))
+		url := srv.URL
+		srv.Close() // nothing is listening now
+
+		_, err := Probe(context.Background(), url+"/v1", "")
+		Expect(err).To(HaveOccurred())
+		Expect(errors.Is(err, ErrUnreachable)).To(BeTrue(), "want ErrUnreachable, got %v", err)
+	})
+
+	It("reports an auth failure distinguishably", func() {
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			w.WriteHeader(http.StatusUnauthorized)
+		}))
+		defer srv.Close()
+
+		_, err := Probe(context.Background(), srv.URL+"/v1", "bad-key")
+		Expect(err).To(HaveOccurred())
+		Expect(errors.Is(err, ErrUnauthorized)).To(BeTrue(), "want ErrUnauthorized, got %v", err)
+	})
+
+	// LocalAI's normal error handler replies with an OpenAI error envelope, and
+	// its opaque-errors handler replies with a bare status and no body. Those
+	// reach the client as two different go-openai types, so both have to be
+	// classified the same way.
+	It("reports an auth failure carrying an error envelope distinguishably", func() {
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			w.Header().Set("Content-Type", "application/json")
+			w.WriteHeader(http.StatusUnauthorized)
+			Expect(json.NewEncoder(w).Encode(map[string]any{
+				"error": map[string]any{"message": "invalid api key", "code": http.StatusUnauthorized},
+			})).To(Succeed())
+		}))
+		defer srv.Close()
+
+		_, err := Probe(context.Background(), srv.URL+"/v1", "bad-key")
+		Expect(err).To(HaveOccurred())
+		Expect(errors.Is(err, ErrUnauthorized)).To(BeTrue(), "want ErrUnauthorized, got %v", err)
+	})
+
+	It("does not call a server that answered with an error unreachable", func() {
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			w.WriteHeader(http.StatusInternalServerError)
+		}))
+		defer srv.Close()
+
+		_, err := Probe(context.Background(), srv.URL+"/v1", "")
+		Expect(err).To(HaveOccurred())
+		Expect(errors.Is(err, ErrUnreachable)).To(BeFalse(), "a server that replied is not unreachable, got %v", err)
+		Expect(errors.Is(err, ErrUnauthorized)).To(BeFalse(), "500 is not an auth failure, got %v", err)
+	})
+
+	// Pointing chat at some other service that happens to be listening is a
+	// different problem from nothing listening, and needs different advice.
+	It("does not call a reply it could not parse unreachable", func() {
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			w.Header().Set("Content-Type", "text/html")
+			_, err := w.Write([]byte("<html><body>not LocalAI</body></html>"))
+			Expect(err).ToNot(HaveOccurred())
+		}))
+		defer srv.Close()
+
+		_, err := Probe(context.Background(), srv.URL+"/v1", "")
+		Expect(err).To(HaveOccurred())
+		Expect(errors.Is(err, ErrUnreachable)).To(BeFalse(), "something answered, got %v", err)
+	})
+
+	It("returns every advertised id, including ones that are not models", func() {
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			w.Header().Set("Content-Type", "application/json")
+			Expect(json.NewEncoder(w).Encode(map[string]any{
+				"object": "list",
+				"data": []map[string]string{
+					{"id": "zeta", "object": "model"},
+					{"id": ".gitignore", "object": "model"},
+					{"id": "alpha", "object": "model"},
+					{"id": "voice.tar.bz2", "object": "model"},
+				},
+			})).To(Succeed())
+		}))
+		defer srv.Close()
+
+		// Verbatim and in server order: deciding which of these are real, and
+		// what order to show them in, belongs to the caller.
+		models, err := Probe(context.Background(), srv.URL+"/v1", "")
+		Expect(err).ToNot(HaveOccurred())
+		Expect(models).To(Equal([]string{"zeta", ".gitignore", "alpha", "voice.tar.bz2"}))
+	})
+
+	It("stops early when the context is already cancelled", func() {
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			w.Header().Set("Content-Type", "application/json")
+			Expect(json.NewEncoder(w).Encode(map[string]any{"object": "list", "data": []any{}})).To(Succeed())
+		}))
+		defer srv.Close()
+
+		ctx, cancel := context.WithCancel(context.Background())
+		cancel()
+
+		_, err := Probe(ctx, srv.URL+"/v1", "")
+		Expect(err).To(HaveOccurred())
+		Expect(errors.Is(err, context.Canceled)).To(BeTrue(), "want the cancellation preserved, got %v", err)
+		// A cancelled probe learned nothing about the endpoint, so it must not
+		// send the caller off to start a server that may already be running.
+		Expect(errors.Is(err, ErrUnreachable)).To(BeFalse(), "cancelling is not a verdict on the server, got %v", err)
+	})
+
+	It("reports a server that never answers as unreachable", func() {
+		release := make(chan struct{})
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			<-release
+		}))
+		defer srv.Close()
+		defer close(release)
+
+		ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
+		defer cancel()
+
+		_, err := Probe(ctx, srv.URL+"/v1", "")
+		Expect(err).To(HaveOccurred())
+		Expect(errors.Is(err, ErrUnreachable)).To(BeTrue(), "want ErrUnreachable, got %v", err)
+		Expect(errors.Is(err, context.DeadlineExceeded)).To(BeTrue(), "want the deadline preserved, got %v", err)
+	})
+
+	It("returns an empty list when the server has no models", func() {
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			w.Header().Set("Content-Type", "application/json")
+			Expect(json.NewEncoder(w).Encode(map[string]any{"object": "list", "data": []any{}})).To(Succeed())
+		}))
+		defer srv.Close()
+
+		models, err := Probe(context.Background(), srv.URL+"/v1", "")
+		Expect(err).ToNot(HaveOccurred())
+		Expect(models).To(BeEmpty())
+	})
+})

core/cli/chat/resolve.go

@@ -0,0 +1,95 @@
+package chat
+
+import (
+	"errors"
+	"fmt"
+	"slices"
+	"sort"
+	"strings"
+
+	"github.com/mudler/xlog"
+)
+
+// ModelChooser asks the user to pick one of models. It is nil when the session
+// is not interactive.
+type ModelChooser func(models []string) (string, error)
+
+// ModelRequest is everything model resolution needs.
+type ModelRequest struct {
+	Flag       string       // --model
+	Configured string       // model recorded in the agent config
+	Available  []string     // models the server advertises
+	StateDir   string       // where an interactive choice is persisted
+	Choose     ModelChooser // nil means non-interactive
+	// Notify reports a problem that is worth telling the user about but not
+	// worth failing over. Nil discards it. It exists because the one such
+	// problem here, a choice that could not be saved, changes what the user
+	// should expect next: they will be asked again. A log line does not reach
+	// them, since the agent runs at log level error by default.
+	Notify func(message string)
+}
+
+// ResolveModel picks the model for this invocation. A flag or a configured
+// value wins outright and is not persisted; only an interactive choice is
+// written back, so the prompt appears at most once.
+//
+// Available is used exactly as the server gave it. LocalAI advertises stray
+// files it finds in the models directory alongside real models, but real model
+// IDs contain dots too (lfm2.5-8b-a1b), so any client-side "looks like a
+// filename" heuristic would eventually hide a model the user has. Deciding
+// which advertised IDs are real belongs to the endpoint, not to a guess here.
+func ResolveModel(req ModelRequest) (string, error) {
+	if req.Flag != "" {
+		return req.Flag, nil
+	}
+	if req.Configured != "" {
+		return req.Configured, nil
+	}
+
+	// The server's /v1/models ordering is not stable between calls, so sort
+	// before showing or listing: the same number must mean the same model on
+	// the next run. Sort a copy; the caller's slice is not ours to reorder.
+	available := append([]string(nil), req.Available...)
+	sort.Strings(available)
+
+	switch len(available) {
+	case 0:
+		return "", errors.New("the LocalAI server has no models installed. Install one with 'local-ai models install <name>', then run 'local-ai chat' again")
+	case 1:
+		return available[0], nil
+	}
+
+	if req.Choose == nil {
+		return "", fmt.Errorf(
+			"several models are available; pick one with --model. Available: %s",
+			strings.Join(available, ", "),
+		)
+	}
+
+	chosen, err := req.Choose(available)
+	if err != nil {
+		return "", err
+	}
+	// Choose is an interface, so its answer is checked rather than trusted.
+	// What comes back is persisted and every later run starts against it, so a
+	// chooser that returns an empty string or a name of its own would record a
+	// model the server never offered and there would be nothing left to catch
+	// it.
+	if !slices.Contains(available, chosen) {
+		return "", fmt.Errorf(
+			"the model chooser answered %q, which is not one of the available models: %s",
+			chosen, strings.Join(available, ", "),
+		)
+	}
+	if req.StateDir != "" {
+		if err := PersistModel(req.StateDir, chosen); err != nil {
+			// A failure to remember the choice must not block the session: the
+			// user picked a model, so honour it and say what will happen.
+			xlog.Warn("could not save the model choice", "error", err, "model", chosen)
+			if req.Notify != nil {
+				req.Notify(fmt.Sprintf("Your choice of %s could not be saved, so this question comes back next time: %v", chosen, err))
+			}
+		}
+	}
+	return chosen, nil
+}

core/cli/chat/resolve_test.go

@@ -0,0 +1,156 @@
+package chat
+
+import (
+	"errors"
+	"os"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("ResolveModel", func() {
+	It("prefers the flag over everything", func() {
+		got, err := ResolveModel(ModelRequest{
+			Flag:       "from-flag",
+			Configured: "from-config",
+			Available:  []string{"a", "b"},
+		})
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).To(Equal("from-flag"))
+	})
+
+	It("uses the configured model when no flag is given", func() {
+		got, err := ResolveModel(ModelRequest{
+			Configured: "from-config",
+			Available:  []string{"a", "b"},
+		})
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).To(Equal("from-config"))
+	})
+
+	It("auto-selects when the server offers exactly one model", func() {
+		got, err := ResolveModel(ModelRequest{Available: []string{"only-one"}})
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).To(Equal("only-one"))
+	})
+
+	It("errors and lists the options when several models exist and there is no chooser", func() {
+		_, err := ResolveModel(ModelRequest{Available: []string{"a", "b"}})
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("a"))
+		Expect(err.Error()).To(ContainSubstring("b"))
+		Expect(err.Error()).To(ContainSubstring("--model"))
+	})
+
+	It("sorts before offering, so the same number means the same model next run", func() {
+		var offered []string
+		available := []string{"zeta", "alpha", "mid"}
+		_, err := ResolveModel(ModelRequest{
+			Available: available,
+			StateDir:  GinkgoT().TempDir(),
+			Choose: func(models []string) (string, error) {
+				offered = models
+				return models[0], nil
+			},
+		})
+		Expect(err).ToNot(HaveOccurred())
+		// The server's /v1/models ordering is unstable between calls.
+		Expect(offered).To(Equal([]string{"alpha", "mid", "zeta"}))
+		// Sorting must happen on a copy: the caller still owns this slice, and
+		// reordering it under them would move whatever they index into it.
+		Expect(available).To(Equal([]string{"zeta", "alpha", "mid"}))
+	})
+
+	It("lists models in sorted order in the several-models error", func() {
+		_, err := ResolveModel(ModelRequest{Available: []string{"zeta", "alpha"}})
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("alpha, zeta"))
+	})
+
+	It("asks the chooser when several models exist, and persists the answer", func() {
+		dir := GinkgoT().TempDir()
+		got, err := ResolveModel(ModelRequest{
+			Available: []string{"a", "b"},
+			StateDir:  dir,
+			Choose:    func(models []string) (string, error) { return models[1], nil },
+		})
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).To(Equal("b"))
+
+		data, err := os.ReadFile(ConfigPath(dir))
+		Expect(err).ToNot(HaveOccurred())
+		Expect(string(data)).To(ContainSubstring("model: b"))
+	})
+
+	// The answer is persisted and every later run starts against it, and
+	// ModelChooser is exported, so the invariant has to hold for choosers this
+	// package did not write.
+	DescribeTable("refuses an answer the chooser was not offered",
+		func(answer string) {
+			dir := GinkgoT().TempDir()
+			got, err := ResolveModel(ModelRequest{
+				Available: []string{"alpha", "zeta"},
+				StateDir:  dir,
+				Choose:    func([]string) (string, error) { return answer, nil },
+			})
+			Expect(err).To(HaveOccurred())
+			Expect(got).To(BeEmpty())
+			Expect(err.Error()).To(ContainSubstring("alpha, zeta"))
+
+			_, statErr := os.Stat(ConfigPath(dir))
+			Expect(os.IsNotExist(statErr)).To(BeTrue(), "nothing may be recorded for an answer that was refused")
+		},
+		Entry("nothing at all", ""),
+		Entry("a model the server never offered", "gamma"),
+		Entry("an offered model with stray whitespace", " alpha"),
+		Entry("an offered model in the wrong case", "Alpha"),
+	)
+
+	It("notifies, and still honours the choice, when it cannot be persisted", func() {
+		dir := GinkgoT().TempDir()
+		// A directory where the config file belongs: the write fails for any
+		// user, including root.
+		Expect(os.MkdirAll(ConfigPath(dir), 0o700)).To(Succeed())
+
+		var notices []string
+		got, err := ResolveModel(ModelRequest{
+			Available: []string{"a", "b"},
+			StateDir:  dir,
+			Choose:    func(models []string) (string, error) { return models[0], nil },
+			Notify:    func(message string) { notices = append(notices, message) },
+		})
+		Expect(err).ToNot(HaveOccurred())
+		Expect(got).To(Equal("a"))
+		Expect(notices).To(HaveLen(1))
+		Expect(notices[0]).To(ContainSubstring("a"))
+		Expect(notices[0]).To(ContainSubstring("could not be saved"))
+	})
+
+	It("says nothing when the choice was saved", func() {
+		var notices []string
+		_, err := ResolveModel(ModelRequest{
+			Available: []string{"a", "b"},
+			StateDir:  GinkgoT().TempDir(),
+			Choose:    func(models []string) (string, error) { return models[0], nil },
+			Notify:    func(message string) { notices = append(notices, message) },
+		})
+		Expect(err).ToNot(HaveOccurred())
+		Expect(notices).To(BeEmpty())
+	})
+
+	It("propagates a chooser cancellation", func() {
+		cancelled := errors.New("cancelled")
+		_, err := ResolveModel(ModelRequest{
+			Available: []string{"a", "b"},
+			StateDir:  GinkgoT().TempDir(),
+			Choose:    func([]string) (string, error) { return "", cancelled },
+		})
+		Expect(errors.Is(err, cancelled)).To(BeTrue())
+	})
+
+	It("errors with an install hint when the server has no models", func() {
+		_, err := ResolveModel(ModelRequest{Available: nil})
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("local-ai models install"))
+	})
+})

core/cli/chat/run.go

@@ -0,0 +1,475 @@
+package chat
+
+import (
+	"bufio"
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"os/signal"
+	"strconv"
+	"strings"
+	"syscall"
+	"time"
+
+	"github.com/mudler/nib/app"
+	nibcmd "github.com/mudler/nib/cmd"
+	nibconfig "github.com/mudler/nib/config"
+	nibtypes "github.com/mudler/nib/types"
+	"golang.org/x/term"
+)
+
+// Options is everything the chat command passes down from its flags.
+type Options struct {
+	Args     []string // forwarded to the agent verbatim
+	Endpoint string   // the server root, e.g. http://127.0.0.1:8080
+	BaseURL  string   // the API base, e.g. http://127.0.0.1:8080/v1
+	APIKey   string
+	Model    string
+	StateDir string
+	TraceDir string
+	Yolo     bool
+	// ProbeTimeout bounds each check of the server. Zero means
+	// defaultProbeTimeout.
+	ProbeTimeout time.Duration
+
+	In     io.Reader
+	Out    io.Writer
+	ErrOut io.Writer
+}
+
+// ExitStatus reports the status the process should exit with for an agent run
+// that failed, and whether err is such a failure.
+//
+// nib writes what went wrong to the error stream itself and hands back nothing
+// but a code, so an error that satisfies this has already been explained to the
+// user and must not be reported a second time. The refusal to open a
+// full-screen session on a stdin that cannot be read arrives this way, and it
+// is the one a user is most likely to meet: 'echo q | local-ai chat' names
+// --cli, and burying that under a second message would hide the fix.
+func ExitStatus(err error) (int, bool) {
+	var exit app.ExitError
+	if errors.As(err, &exit) {
+		return exit.Code, true
+	}
+	return 0, false
+}
+
+// shutdownSignals end the session. SIGHUP is one of them because this is a
+// terminal program: once the terminal is gone there is nobody left to talk to,
+// and a server started for the session has to go with it.
+var shutdownSignals = []os.Signal{os.Interrupt, syscall.SIGTERM, syscall.SIGHUP}
+
+// shutdownContext derives a context that is cancelled when the process is
+// asked to stop.
+//
+// Without it a signal kills this process where it stands, skipping every
+// deferred call, and a 'local-ai run' started for the session is reparented to
+// init with nothing left that knows to shut it down. An interactive Ctrl+C is
+// safe on its own, because the child shares this process' foreground process
+// group and the terminal signals all of it, but a SIGTERM from a supervisor or
+// a script reaches only this process.
+//
+// Since nib v0.5.1 cancelling this context does end the session: RunTUI passes
+// it to bubbletea, which unwinds the program and reports the context's own
+// error. The server is still stopped on cancellation rather than on the way
+// out (see runSession), because registering here removes SIGHUP's default
+// terminate disposition, and a guarantee about a server this process owns is
+// not worth resting on how promptly a third party unwinds its interface.
+//
+// A handler rather than SysProcAttr.Pdeathsig on the child: Pdeathsig is
+// Linux-only, and in Go it is delivered when the OS thread that forked exits
+// rather than when the process does, so it can fire on a perfectly healthy
+// parent. Setpgid is not an alternative either, since taking the child out of
+// the foreground process group is what would break the Ctrl+C that works
+// today. SIGKILL stays uncovered, as it must: nothing in the process can
+// observe it.
+func shutdownContext(parent context.Context) (context.Context, context.CancelFunc) {
+	return signal.NotifyContext(parent, shutdownSignals...)
+}
+
+// Run starts the agent: resolve where state lives, make sure a server is
+// reachable, pick a model, then hand off to nib.
+func Run(ctx context.Context, opts Options) error {
+	ctx, stop := shutdownContext(ctx)
+	defer stop()
+
+	p, err := prepare(ctx, opts, isTerminal(opts.In))
+	if err != nil {
+		return err
+	}
+	// A server this process started belongs to this session, and Stop is
+	// nil-safe and idempotent, so one defer covers both cases and costs nothing
+	// when runSession has already stopped it.
+	defer p.server.Stop()
+
+	return runSession(ctx, p.server, func(ctx context.Context) error {
+		return runAgent(ctx, p.dir, p.model, opts)
+	})
+}
+
+// runSession hands the terminal to agent, and stops a server started for this
+// session as soon as the context is cancelled rather than when agent returns.
+//
+// The difference matters because the deferred Stop in Run is only reached once
+// agent returns, and how long that takes is nib's business rather than ours.
+// nib v0.5.1 does unwind the TUI on a cancelled context, so it does return; a
+// SIGHUP no longer leaves the interface on screen with the server behind it,
+// which it did before, when bubbletea's own SIGINT and SIGTERM handler was the
+// only thing that ever quit the program and registering for SIGHUP had removed
+// the default disposition that used to end the process. Watching the context
+// keeps the guarantee independent of what the agent does with it.
+func runSession(ctx context.Context, server *StartedServer, agent func(context.Context) error) error {
+	returned := make(chan struct{})
+	defer close(returned)
+
+	go func() {
+		select {
+		case <-ctx.Done():
+			server.Stop()
+		case <-returned:
+		}
+	}()
+
+	return agent(ctx)
+}
+
+// preparation is what the agent needs once the environment is ready: where its
+// state lives, which model to talk to, and the server this process started on
+// the user's behalf, if any.
+type preparation struct {
+	dir    string
+	model  string
+	server *StartedServer
+}
+
+// prepare does everything that has to happen before the agent takes over the
+// terminal. It is split out of Run because all of it is testable and none of
+// what follows is: once app.Run has the terminal there is no seam left.
+//
+// interactive says whether there is a user to prompt. It is a parameter rather
+// than a second read of opts.In so the prompts can be driven over a pipe.
+func prepare(ctx context.Context, opts Options, interactive bool) (_ *preparation, err error) {
+	dir, dirErr := StateDir(opts.StateDir)
+	if dirErr != nil {
+		return nil, dirErr
+	}
+	if err := EnsureStateDir(dir, opts.BaseURL); err != nil {
+		return nil, err
+	}
+
+	if isLocalOnlyArgs(opts.Args) {
+		return &preparation{dir: dir}, nil
+	}
+
+	// One prompter for every question this run asks; see its doc comment for
+	// why the reader cannot be rebuilt per question.
+	var prompts *prompter
+	if interactive {
+		prompts = newPrompter(opts.In, opts.ErrOut)
+	}
+
+	var started *StartedServer
+	defer func() {
+		// Nothing after the spawn may leave a server behind: the caller only
+		// learns about it through a successful return.
+		if err != nil {
+			started.Stop()
+		}
+	}()
+
+	models, err := probeModels(ctx, opts)
+	if err != nil {
+		if errors.Is(err, ErrUnauthorized) {
+			return nil, fmt.Errorf("the LocalAI server at %s rejected the API key. Pass --api-key or set LOCALAI_API_KEY", opts.Endpoint)
+		}
+		if !errors.Is(err, ErrUnreachable) {
+			return nil, err
+		}
+
+		var confirm Confirmer
+		if interactive {
+			confirm = prompts.yesNo
+		}
+		var startErr error
+		started, startErr = OfferToStart(ctx, StartOptions{
+			Endpoint: opts.Endpoint,
+			Confirm:  confirm,
+			Stderr:   opts.ErrOut,
+		})
+		if startErr != nil {
+			err = startErr
+			if errors.Is(startErr, ErrDeclined) {
+				err = fmt.Errorf("no LocalAI server at %s. Start one with 'local-ai run', or point elsewhere with --endpoint", opts.Endpoint)
+			}
+			return nil, err
+		}
+		say(opts.ErrOut, "Started a temporary LocalAI server; it stops when you exit. Use 'local-ai run' for a persistent one.\n")
+
+		if models, err = probeModels(ctx, opts); err != nil {
+			return nil, err
+		}
+	}
+
+	var chooser ModelChooser
+	if interactive {
+		chooser = prompts.choose
+	}
+	model, err := ResolveModel(ModelRequest{
+		Flag:       opts.Model,
+		Configured: configuredModel(dir),
+		Available:  models,
+		StateDir:   dir,
+		Choose:     chooser,
+		Notify:     func(message string) { say(opts.ErrOut, "%s\n", message) },
+	})
+	if err != nil {
+		return nil, err
+	}
+
+	return &preparation{dir: dir, model: model, server: started}, nil
+}
+
+func runAgent(ctx context.Context, dir, model string, opts Options) error {
+	return app.Run(ctx, agentOptions(dir, model, opts))
+}
+
+// agentOptions builds the request handed to nib. It is split out of runAgent
+// because app.Run takes the terminal and cannot be called from a test, while
+// what is asked of it is exactly the part worth pinning.
+//
+// The stream fields are the interesting ones, and they are not symmetric.
+//
+// nib reads a non-nil stream as "the embedder wants this used", and refuses
+// every mode but --cli when such a stream is not a terminal, because the
+// full-screen interface renders on /dev/tty and would otherwise ignore it in
+// silence. Nil means "not injected": nib falls back to the process stream and
+// behaves as standalone nib does.
+//
+// Stdin is passed through as it comes. A piped or redirected stdin really is
+// ignored by the interface, so the refusal is the honest answer there, and it
+// is the one users meet: 'echo q | local-ai chat' says to re-run with --cli
+// rather than opening a full-screen session that will never read the question.
+//
+// Stdout is different, and the process stream is deliberately sent as nil. The
+// interface does write to stdout even when it is a pipe: that is the whole of
+// nib's shell-capture idiom, out=$(local-ai chat --height 50%), which is what
+// the Ctrl+Space widget emitted by --init is built on. Injecting os.Stdout
+// there would refuse the widget for a stream nib was going to use anyway.
+//
+// The test is identity with os.Stdout rather than whether it happens to be a
+// terminal, which means a shell redirect goes the same way as the widget:
+// 'local-ai chat > out.txt' no longer refuses either, and renders on /dev/tty
+// with the capture line landing in the file. That is not a second decision, it
+// is the same one. Both are the process stdout as the shell handed it over,
+// differing only in being a pipe rather than a regular file, which nib's gate
+// does not look at and should not. Refusing one would refuse the other.
+//
+// What stays injected, and so stays subject to the refusal, is a writer some
+// in-process caller chose for itself rather than inherited: a bytes.Buffer, or
+// an *os.File it opened. The specs rely on that.
+//
+// Stderr is never gated by nib, so it is passed through unchanged.
+//
+// The config values go through Overrides rather than Defaults, and that is not
+// a detail. Defaults are seeds: they sit BENEATH the config file, so the file
+// silently undoes them. Everything here is a decision this invocation already
+// made on the user's behalf, and a flag that the file can undo is not a flag.
+// It was not a rare case either, since EnsureStateDir writes base_url on the
+// first run and an interactive choice writes model, so from the second run on
+// the file carried a value for both and --endpoint and --model did nothing.
+//
+// The one asymmetry to plan around is that nib cannot tell "set to the zero
+// value" from "not set", so an override only ever raises a field. --yolo can
+// turn approval off, but nothing on the command line can turn it back on over
+// an approval_mode: auto in the file; that needs a config edit. Same shape for
+// the strings, which is what makes an unset --api-key or --trace-dir leave the
+// file's value standing, as it should.
+//
+// nib's own --trace-dir and --yolo, and their NIB_TRACE_DIR and NIB_YOLO twins,
+// are resolved after the config load and so still outrank these. That is
+// deliberate upstream: they are instructions to nib rather than ambient
+// environment.
+func agentOptions(dir, model string, opts Options) app.Options {
+	// Model is the model this run resolved, which already prefers --model and
+	// falls back to the file's own model, so the override restates the file's
+	// value rather than fighting it whenever no flag was given.
+	//
+	// BaseURL is the endpoint this run probed, offered to start a server for,
+	// and seeded the config with. Handing nib a different one is precisely the
+	// split that made --endpoint a no-op, so the agent talks to the server
+	// LocalAI checked. Pointing somewhere else for good is LOCALAI_CHAT_ENDPOINT
+	// or --endpoint, not a hand-edited base_url the probe never reads.
+	//
+	// APIKey and TraceDir are the flags as given, empty when they were not, and
+	// an empty override leaves the file alone. TraceDir is runtime-only in nib
+	// (yaml:"-"), so no file value exists for it to beat today; it belongs here
+	// with the other flags rather than one rung down for a reason that could
+	// quietly stop being true.
+	overrides := nibtypes.Config{
+		Model:    model,
+		APIKey:   opts.APIKey,
+		BaseURL:  opts.BaseURL,
+		TraceDir: opts.TraceDir,
+	}
+	if opts.Yolo {
+		overrides.ApprovalMode = "auto"
+	}
+
+	return app.Options{
+		Args:        opts.Args,
+		ProgramName: "local-ai chat",
+		BaseDir:     dir,
+		Overrides:   overrides,
+		SkipSetup:   true,
+		SkipBareEnv: true,
+		Stdin:       opts.In,
+		Stdout:      ownStdout(opts.Out),
+		Stderr:      opts.ErrOut,
+	}
+}
+
+// ownStdout reports the writer as nib's own rather than as an injected one when
+// it is the process stdout, by answering nil for it. See agentOptions for why
+// that distinction is the difference between a working Ctrl+Space widget and a
+// refused one.
+func ownStdout(w io.Writer) io.Writer {
+	if f, ok := w.(*os.File); ok && f == os.Stdout {
+		return nil
+	}
+	return w
+}
+
+// defaultProbeTimeout bounds a check of the server. Listing models is cheap,
+// so this is long enough that a loaded server is never given up on and short
+// enough that a hung one does not leave the user staring at nothing.
+const defaultProbeTimeout = 30 * time.Second
+
+// probeModels lists what the endpoint offers, under a budget.
+func probeModels(ctx context.Context, opts Options) ([]string, error) {
+	timeout := opts.ProbeTimeout
+	if timeout <= 0 {
+		timeout = defaultProbeTimeout
+	}
+	// A real deadline rather than a cancel plus a timer. Probe reads
+	// context.Canceled as "the caller gave up", which is a statement about the
+	// caller and not about the endpoint, and only a deadline as "nothing
+	// answered in time". Expiring the budget as a cancellation would stop
+	// ErrUnreachable firing for precisely the hung servers that the offer to
+	// start one exists for.
+	probeCtx, cancel := context.WithTimeout(ctx, timeout)
+	defer cancel()
+	return Probe(probeCtx, opts.BaseURL, opts.APIKey)
+}
+
+// isLocalOnlyArgs reports whether the forwarded arguments do their work
+// without ever reaching a model, in which case demanding a running server (and
+// offering to start one) would be an obstacle rather than a service.
+//
+// Two groups qualify. The management subcommands edit nib's own state: plugin,
+// skill, and the mcp verbs that add or remove configured servers, which is
+// asked of nib rather than restated, because bare 'mcp' and its transport
+// flags do serve the agent and do need a model. The other group is the flags
+// that only print something, above all --init: its shell snippet goes into an
+// rc file, typically long before any server exists.
+func isLocalOnlyArgs(args []string) bool {
+	if len(args) == 0 {
+		return false
+	}
+	// A scan rather than a look at args[0]: the mode flags this command
+	// translates are prepended, so --init is not necessarily first. Positional
+	// text cannot be mistaken for a flag here, since nib ignores what is left
+	// after flag parsing.
+	for _, a := range args {
+		switch {
+		case a == "--init", a == "-init", strings.HasPrefix(a, "--init="), strings.HasPrefix(a, "-init="):
+			return true
+		case a == "--version", a == "-version":
+			return true
+		}
+	}
+	switch args[0] {
+	case "plugin", "skill":
+		return true
+	case "mcp":
+		return len(args) >= 2 && nibcmd.IsMCPManageSubcommand(args[1])
+	}
+	return false
+}
+
+// configuredModel reads the model already recorded in the agent config, if any.
+func configuredModel(dir string) string {
+	cfg := nibconfig.LoadWith(nibconfig.LoadOptions{BaseDir: dir, SkipBareEnv: true})
+	return cfg.Model
+}
+
+func isTerminal(in io.Reader) bool {
+	f, ok := in.(*os.File)
+	return ok && term.IsTerminal(int(f.Fd()))
+}
+
+// say writes a line of interactive chatter: a question, or a notice about
+// something that did not stop the session. A write that fails is not worth
+// failing over, and when the terminal really is gone the read that follows the
+// question says so.
+func say(w io.Writer, format string, args ...any) {
+	_, _ = fmt.Fprintf(w, format, args...)
+}
+
+// prompter asks this run's questions on the user's terminal.
+//
+// It owns the buffered reader rather than wrapping opts.In per question,
+// because bufio reads ahead: a throwaway reader for the "start a server?"
+// question swallows the model choice that was typed behind it, and the next
+// question then sees EOF. A real run asks both, one after the other.
+type prompter struct {
+	in  *bufio.Reader
+	out io.Writer
+}
+
+func newPrompter(in io.Reader, out io.Writer) *prompter {
+	return &prompter{in: bufio.NewReader(in), out: out}
+}
+
+// yesNo satisfies Confirmer. Anything that is not an explicit yes is a no, so
+// a closed stream declines rather than proceeding on the user's behalf.
+func (p *prompter) yesNo(question string) (bool, error) {
+	say(p.out, "%s [y/N]: ", question)
+	line, err := p.in.ReadString('\n')
+	if err != nil && !errors.Is(err, io.EOF) {
+		return false, fmt.Errorf("reading the answer: %w", err)
+	}
+	switch strings.ToLower(strings.TrimSpace(line)) {
+	case "y", "yes":
+		return true, nil
+	}
+	return false, nil
+}
+
+// choose satisfies ModelChooser. It answers with a list index rather than with
+// what the user typed, so the result can only ever be one of the models it was
+// offered: a model name is not something to accept unvalidated here, since
+// ResolveModel persists whatever comes back and every later run then starts
+// against it.
+func (p *prompter) choose(models []string) (string, error) {
+	if len(models) == 0 {
+		return "", errors.New("there is nothing to choose from")
+	}
+	say(p.out, "Several models are available:\n")
+	for i, m := range models {
+		say(p.out, "  %d) %s\n", i+1, m)
+	}
+	say(p.out, "Pick one [1-%d]: ", len(models))
+
+	line, err := p.in.ReadString('\n')
+	if err != nil && !errors.Is(err, io.EOF) {
+		return "", fmt.Errorf("reading the choice: %w", err)
+	}
+	answer := strings.TrimSpace(line)
+	n, err := strconv.Atoi(answer)
+	if err != nil || n < 1 || n > len(models) {
+		return "", fmt.Errorf("not a valid choice: %q. Pick a number between 1 and %d, or pass --model", answer, len(models))
+	}
+	return models[n-1], nil
+}

core/cli/chat/run_test.go

@@ -0,0 +1,629 @@
+package chat
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"path/filepath"
+	"strings"
+	"syscall"
+	"time"
+
+	"github.com/mudler/nib/app"
+	nibconfig "github.com/mudler/nib/config"
+	nibtypes "github.com/mudler/nib/types"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+// modelServer answers /v1/models with the given ids, as LocalAI does.
+func modelServer(ids ...string) *httptest.Server {
+	return httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		data := make([]map[string]string, 0, len(ids))
+		for _, id := range ids {
+			data = append(data, map[string]string{"id": id, "object": "model"})
+		}
+		w.Header().Set("Content-Type", "application/json")
+		Expect(json.NewEncoder(w).Encode(map[string]any{"object": "list", "data": data})).To(Succeed())
+	}))
+}
+
+var _ = Describe("prepare", func() {
+	var (
+		dir    string
+		errOut *bytes.Buffer
+	)
+
+	BeforeEach(func() {
+		dir = GinkgoT().TempDir()
+		errOut = &bytes.Buffer{}
+	})
+
+	// optionsFor points a run at srv, with no input to read: the default is a
+	// session nobody can be asked anything in.
+	optionsFor := func(srv *httptest.Server) Options {
+		endpoint := "http://127.0.0.1:0"
+		base := endpoint + "/v1"
+		if srv != nil {
+			endpoint, base = srv.URL, srv.URL+"/v1"
+		}
+		return Options{
+			Endpoint: endpoint,
+			BaseURL:  base,
+			StateDir: dir,
+			In:       strings.NewReader(""),
+			Out:      &bytes.Buffer{},
+			ErrOut:   errOut,
+		}
+	}
+
+	It("uses the only model the server offers", func() {
+		srv := modelServer("the-only-model")
+		defer srv.Close()
+
+		p, err := prepare(context.Background(), optionsFor(srv), false)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(p.model).To(Equal("the-only-model"))
+		Expect(p.dir).To(Equal(dir))
+		Expect(p.server).To(BeNil(), "nothing was started, so nothing is owned")
+	})
+
+	It("seeds the agent config with the endpoint on first run", func() {
+		srv := modelServer("m")
+		defer srv.Close()
+
+		_, err := prepare(context.Background(), optionsFor(srv), false)
+		Expect(err).ToNot(HaveOccurred())
+
+		data, err := os.ReadFile(ConfigPath(dir))
+		Expect(err).ToNot(HaveOccurred())
+		Expect(string(data)).To(ContainSubstring(srv.URL + "/v1"))
+	})
+
+	It("lets --model win over what the server offers", func() {
+		srv := modelServer("a", "b")
+		defer srv.Close()
+
+		opts := optionsFor(srv)
+		opts.Model = "not-listed-yet"
+		p, err := prepare(context.Background(), opts, false)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(p.model).To(Equal("not-listed-yet"))
+	})
+
+	It("advises about the API key when the server rejects it", func() {
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			w.WriteHeader(http.StatusUnauthorized)
+		}))
+		defer srv.Close()
+
+		_, err := prepare(context.Background(), optionsFor(srv), false)
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("--api-key"))
+		Expect(err.Error()).To(ContainSubstring(srv.URL))
+	})
+
+	// Not interactive means nobody can answer the offer, so the advice has to
+	// stand on its own.
+	It("advises how to start a server when none is reachable", func() {
+		srv := modelServer()
+		url := srv.URL
+		srv.Close() // nothing is listening now
+
+		opts := optionsFor(nil)
+		opts.Endpoint, opts.BaseURL = url, url+"/v1"
+		_, err := prepare(context.Background(), opts, false)
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("local-ai run"))
+		Expect(err.Error()).To(ContainSubstring(url))
+	})
+
+	// A server that accepts the connection and then never replies is the case
+	// the offer to start one exists for, so the budget has to expire as a
+	// deadline: Probe reads a cancellation as "the caller gave up" and refuses
+	// to call the endpoint unreachable on the strength of it.
+	It("treats a server that never answers as one that is not there", func(ctx SpecContext) {
+		release := make(chan struct{})
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			select {
+			case <-release:
+			case <-r.Context().Done():
+			}
+		}))
+		defer srv.Close()
+		defer close(release)
+
+		opts := optionsFor(srv)
+		opts.ProbeTimeout = 100 * time.Millisecond
+		_, err := prepare(context.Background(), opts, false)
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("local-ai run"), "want the offer-a-server advice, got %v", err)
+	}, SpecTimeout(30*time.Second))
+
+	It("asks which model to use and remembers the answer", func() {
+		srv := modelServer("zeta", "alpha")
+		defer srv.Close()
+
+		opts := optionsFor(srv)
+		opts.In = strings.NewReader("2\n")
+		p, err := prepare(context.Background(), opts, true)
+		Expect(err).ToNot(HaveOccurred())
+		// The list is sorted before it is shown, so 2 is zeta, not the second
+		// thing the server happened to name.
+		Expect(p.model).To(Equal("zeta"))
+		Expect(errOut.String()).To(ContainSubstring("1) alpha"))
+		Expect(errOut.String()).To(ContainSubstring("2) zeta"))
+
+		data, err := os.ReadFile(ConfigPath(dir))
+		Expect(err).ToNot(HaveOccurred())
+		Expect(string(data)).To(ContainSubstring("zeta"))
+	})
+
+	// The choice is prompted for once and remembered. When remembering it fails
+	// the user is about to be asked again on every future run, so they have to
+	// be told here: a log line is invisible at the default log level.
+	It("says so on the prompt when the choice cannot be remembered", func() {
+		srv := modelServer("zeta", "alpha")
+		defer srv.Close()
+
+		// A directory where the config file belongs: writable state dir,
+		// unwritable config, on any platform and as any user.
+		Expect(os.MkdirAll(ConfigPath(dir), 0o700)).To(Succeed())
+
+		opts := optionsFor(srv)
+		opts.In = strings.NewReader("1\n")
+		p, err := prepare(context.Background(), opts, true)
+
+		// Failing to remember the choice must not cost the user their session.
+		Expect(err).ToNot(HaveOccurred())
+		Expect(p.model).To(Equal("alpha"))
+		Expect(errOut.String()).To(ContainSubstring("could not be saved"), "the user has to learn they will be asked again")
+	})
+
+	It("does not ask again once a model is recorded", func() {
+		srv := modelServer("zeta", "alpha")
+		defer srv.Close()
+
+		Expect(PersistModel(dir, "alpha")).To(Succeed())
+
+		opts := optionsFor(srv)
+		opts.In = strings.NewReader("") // an answer would have nothing to read
+		p, err := prepare(context.Background(), opts, true)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(p.model).To(Equal("alpha"))
+		Expect(errOut.String()).To(BeEmpty())
+	})
+
+	It("says what to install when the server has no models", func() {
+		srv := modelServer()
+		defer srv.Close()
+
+		_, err := prepare(context.Background(), optionsFor(srv), false)
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("models install"))
+	})
+
+	Describe("arguments that only touch local state", func() {
+		unreachable := func(args ...string) Options {
+			opts := optionsFor(nil) // port 0: nothing can ever answer here
+			opts.Args = args
+			return opts
+		}
+
+		DescribeTable("skips the server entirely",
+			func(args ...string) {
+				p, err := prepare(context.Background(), unreachable(args...), false)
+				Expect(err).ToNot(HaveOccurred())
+				Expect(p.model).To(BeEmpty())
+				Expect(p.server).To(BeNil())
+			},
+			Entry("plugin", "plugin", "list"),
+			Entry("skill", "skill", "list"),
+			Entry("mcp add", "mcp", "add", "srv"),
+			Entry("mcp list", "mcp", "list"),
+			// The shell snippet is what a user puts in their rc file, long
+			// before any server exists.
+			Entry("the shell integration script", "--init", "zsh"),
+			Entry("the version", "--version"),
+		)
+
+		// Bare 'mcp' and its transport flags serve the agent over MCP, so they
+		// need a model like any other session. Only the verbs that edit the
+		// configured servers are local.
+		DescribeTable("still needs a server",
+			func(args ...string) {
+				_, err := prepare(context.Background(), unreachable(args...), false)
+				Expect(err).To(HaveOccurred())
+				Expect(err.Error()).To(ContainSubstring("local-ai run"))
+			},
+			Entry("mcp over stdio", "mcp", "--stdio"),
+			Entry("bare mcp", "mcp"),
+		)
+	})
+
+	// A reader per question would read ahead into a buffer it then discards, so
+	// the second question would see EOF whenever both answers were typed ahead.
+	// That is the shape of a real run: the offer to start a server is followed
+	// by the model prompt.
+	It("keeps reading answers from the same stream across questions", func() {
+		out := &bytes.Buffer{}
+		p := newPrompter(strings.NewReader("y\n2\n"), out)
+
+		yes, err := p.yesNo("Start one now?")
+		Expect(err).ToNot(HaveOccurred())
+		Expect(yes).To(BeTrue())
+
+		chosen, err := p.choose([]string{"alpha", "zeta"})
+		Expect(err).ToNot(HaveOccurred())
+		Expect(chosen).To(Equal("zeta"))
+	})
+
+	// Whatever the chooser returns is persisted and used for every later run,
+	// so an answer that is not one of the offered models must never come back
+	// as one.
+	Describe("the model prompt", func() {
+		offered := []string{"alpha", "zeta"}
+
+		DescribeTable("refuses an answer that is not one of the numbers shown",
+			func(answer string) {
+				chosen, err := newPrompter(strings.NewReader(answer), &bytes.Buffer{}).choose(offered)
+				Expect(err).To(HaveOccurred())
+				Expect(chosen).To(BeEmpty())
+			},
+			Entry("nothing at all", ""),
+			Entry("a blank line", "\n"),
+			Entry("only spaces", "   \n"),
+			Entry("zero", "0\n"),
+			Entry("past the end", "3\n"),
+			Entry("negative", "-1\n"),
+			Entry("a model name", "zeta\n"),
+			Entry("a number with a suffix", "1x\n"),
+		)
+
+		It("says how to answer when the answer was not a number", func() {
+			_, err := newPrompter(strings.NewReader("banana\n"), &bytes.Buffer{}).choose(offered)
+			Expect(err).To(HaveOccurred())
+			Expect(err.Error()).To(ContainSubstring("between 1 and 2"))
+			Expect(err.Error()).To(ContainSubstring("--model"))
+		})
+
+		It("returns the model shown against the number", func() {
+			chosen, err := newPrompter(strings.NewReader("1\n"), &bytes.Buffer{}).choose(offered)
+			Expect(err).ToNot(HaveOccurred())
+			Expect(chosen).To(Equal("alpha"))
+		})
+
+		It("refuses to ask when there is nothing to offer", func() {
+			chosen, err := newPrompter(strings.NewReader("1\n"), &bytes.Buffer{}).choose(nil)
+			Expect(err).To(HaveOccurred())
+			Expect(chosen).To(BeEmpty())
+		})
+	})
+
+	// A server started for this session is stopped by a deferred call, which a
+	// signal skips: the process dies where it stands and leaves 'local-ai run'
+	// reparented to init.
+	Describe("shutdown signals", func() {
+		It("ends the session when the terminal goes away", func() {
+			ctx, stop := shutdownContext(context.Background())
+			defer stop()
+
+			self, err := os.FindProcess(os.Getpid())
+			Expect(err).ToNot(HaveOccurred())
+			Expect(self.Signal(syscall.SIGHUP)).To(Succeed())
+
+			Eventually(ctx.Done()).WithTimeout(5 * time.Second).Should(BeClosed())
+			Expect(ctx.Err()).To(MatchError(context.Canceled))
+		})
+
+		// SIGINT and SIGTERM cannot be delivered here to prove the same thing:
+		// Ginkgo registers for both to abort the suite, and a signal goes to
+		// every registered listener.
+		It("also listens for an interrupt and a terminate", func() {
+			Expect(shutdownSignals).To(ContainElements(os.Signal(os.Interrupt), os.Signal(syscall.SIGTERM)))
+		})
+	})
+
+	// Cancelling the context does unwind nib's TUI since v0.5.1, but how long
+	// that takes is nib's business, and the deferred Stop in Run is only reached
+	// once the agent returns. A server this process started is ours to end, so
+	// the guarantee is made here instead, where it does not depend on the agent
+	// at all. Before v0.5.1 there was no guarantee to be had on the SIGHUP path:
+	// bubbletea's own SIGINT and SIGTERM handler was the only thing that ever
+	// quit the program, and registering for SIGHUP took away the default
+	// disposition that used to end the process.
+	Describe("runSession", func() {
+		It("stops the session's server on cancellation, without waiting for the agent", func() {
+			server, proc := stoppableServer()
+			ctx, cancel := context.WithCancel(context.Background())
+			defer cancel()
+
+			err := runSession(ctx, server, func(ctx context.Context) error {
+				cancel()
+				Eventually(func() int32 { return proc.interrupts.Load() }).
+					WithTimeout(5 * time.Second).
+					Should(BeNumerically(">", 0), "the server has to be stopped while the agent is still running")
+				return nil
+			})
+			Expect(err).ToNot(HaveOccurred())
+			Expect(proc.lastSignal.Load()).To(Equal(os.Interrupt))
+		})
+
+		It("leaves the server alone for as long as the session lasts", func() {
+			server, proc := stoppableServer()
+
+			Expect(runSession(context.Background(), server, func(context.Context) error {
+				return nil
+			})).To(Succeed())
+			Expect(proc.interrupts.Load()).To(BeZero())
+			Expect(proc.kills.Load()).To(BeZero())
+		})
+
+		It("returns what the agent returned", func() {
+			failed := errors.New("the agent gave up")
+			server, _ := stoppableServer()
+
+			Expect(runSession(context.Background(), server, func(context.Context) error {
+				return failed
+			})).To(MatchError(failed))
+		})
+
+		// Most sessions run against a server the user already had, and there is
+		// nothing to stop then.
+		It("copes with a session that started no server", func() {
+			ctx, cancel := context.WithCancel(context.Background())
+			cancel()
+
+			Expect(runSession(ctx, nil, func(context.Context) error {
+				return nil
+			})).To(Succeed())
+		})
+	})
+
+	// Which streams reach nib decides two user-visible behaviours at once, and
+	// they pull in opposite directions, so both are pinned here rather than left
+	// to whoever next edits the literal.
+	//
+	// nib refuses every mode but --cli when a stream it was handed is not a
+	// terminal. That refusal is wanted for stdin, where it is what tells someone
+	// piping a question to re-run with --cli. It is not wanted for the process
+	// stdout, where it would refuse the Ctrl+Space widget that --init emits:
+	// out=$(local-ai chat --height 50%) puts a pipe on stdout by construction,
+	// and writing the chosen command into that pipe is the entire point.
+	Describe("agentOptions", func() {
+		// optionsWithStreams is a request that differs from the next only in
+		// what it was told to read and write.
+		optionsWithStreams := func(in io.Reader, out, errOut io.Writer) Options {
+			return Options{
+				BaseURL: "http://127.0.0.1:8080/v1",
+				In:      in,
+				Out:     out,
+				ErrOut:  errOut,
+			}
+		}
+
+		Describe("stdout", func() {
+			// The regression this exists to catch: reinstating
+			// 'Stdout: opts.Out' breaks Ctrl+Space and nothing else notices.
+			It("hands nib nothing for the process stdout, so the capture widget is not refused", func() {
+				o := agentOptions(dir, "a-model", optionsWithStreams(os.Stdin, os.Stdout, os.Stderr))
+				Expect(o.Stdout).To(BeNil(), "injecting os.Stdout is what refuses out=$(local-ai chat)")
+			})
+
+			It("keeps a stdout the caller chose, which the refusal still guards", func() {
+				out := &bytes.Buffer{}
+				o := agentOptions(dir, "a-model", optionsWithStreams(os.Stdin, out, os.Stderr))
+				Expect(o.Stdout).To(BeIdenticalTo(out))
+			})
+
+			// Being an *os.File is not what makes a stream nib's own; being the
+			// process stdout is. This is a file an in-process caller opened for
+			// itself, not one a shell redirect handed over as stdout, which
+			// still arrives as os.Stdout and is still nil-ed. It was never going
+			// to receive the interface, so it stays injected and stays refused.
+			It("keeps a file that is not the process stdout", func() {
+				f, err := os.CreateTemp(GinkgoT().TempDir(), "captured")
+				Expect(err).ToNot(HaveOccurred())
+				DeferCleanup(f.Close)
+
+				o := agentOptions(dir, "a-model", optionsWithStreams(os.Stdin, f, os.Stderr))
+				Expect(o.Stdout).To(BeIdenticalTo(f))
+			})
+		})
+
+		Describe("stdin", func() {
+			// The opposite regression: nilling stdin the way stdout is nilled
+			// would silently drop the refusal that names --cli.
+			It("hands the process stdin over, so a piped session is still refused", func() {
+				o := agentOptions(dir, "a-model", optionsWithStreams(os.Stdin, os.Stdout, os.Stderr))
+				Expect(o.Stdin).To(BeIdenticalTo(os.Stdin))
+			})
+
+			It("hands over a stdin the caller chose", func() {
+				in := strings.NewReader("a question")
+				o := agentOptions(dir, "a-model", optionsWithStreams(in, os.Stdout, os.Stderr))
+				Expect(o.Stdin).To(BeIdenticalTo(in))
+			})
+		})
+
+		// nib gates stdin and stdout and nothing else, so there is no reason to
+		// hide the error stream from it.
+		It("hands the error stream over whatever it is", func() {
+			errOut := &bytes.Buffer{}
+			o := agentOptions(dir, "a-model", optionsWithStreams(os.Stdin, os.Stdout, errOut))
+			Expect(o.Stderr).To(BeIdenticalTo(errOut))
+
+			o = agentOptions(dir, "a-model", optionsWithStreams(os.Stdin, os.Stdout, os.Stderr))
+			Expect(o.Stderr).To(BeIdenticalTo(os.Stderr))
+		})
+
+		It("names the command a user would type, not the binary nib ships as", func() {
+			o := agentOptions(dir, "a-model", optionsWithStreams(os.Stdin, os.Stdout, os.Stderr))
+			Expect(o.ProgramName).To(Equal("local-ai chat"),
+				"the --init widget invokes this name, so a user has to be able to run it")
+		})
+
+		It("carries the resolved session through to nib", func() {
+			opts := optionsWithStreams(os.Stdin, os.Stdout, os.Stderr)
+			opts.Args = []string{"--cli"}
+			opts.APIKey = "a-key"
+			opts.TraceDir = "/traces"
+
+			o := agentOptions(dir, "the-model", opts)
+			Expect(o.Args).To(Equal([]string{"--cli"}))
+			Expect(o.BaseDir).To(Equal(dir))
+			Expect(o.Overrides.Model).To(Equal("the-model"))
+			Expect(o.Overrides.APIKey).To(Equal("a-key"))
+			Expect(o.Overrides.BaseURL).To(Equal("http://127.0.0.1:8080/v1"))
+			Expect(o.Overrides.TraceDir).To(Equal("/traces"))
+			// The model and the server are settled before nib starts, and the
+			// bare MODEL and API_KEY variables belong to some other tool.
+			Expect(o.SkipSetup).To(BeTrue())
+			Expect(o.SkipBareEnv).To(BeTrue())
+		})
+
+		// Defaults sit beneath the config file. Anything routed through them is
+		// accepted from the command line and then thrown away the moment the
+		// file carries the same key, which is the normal state rather than an
+		// edge case. Nothing this command resolves belongs there, so the channel
+		// stays empty and this says so: it is what fails if the block is moved
+		// back a rung.
+		It("seeds nothing, because a seed is not a flag", func() {
+			opts := optionsWithStreams(os.Stdin, os.Stdout, os.Stderr)
+			opts.APIKey = "a-key"
+			opts.TraceDir = "/traces"
+			opts.Yolo = true
+
+			Expect(agentOptions(dir, "the-model", opts).Defaults).To(Equal(nibtypes.Config{}),
+				"Defaults lose to the config file, so a value placed there is a flag that does nothing")
+		})
+
+		It("asks for automatic approval only when --yolo was given", func() {
+			opts := optionsWithStreams(os.Stdin, os.Stdout, os.Stderr)
+			Expect(agentOptions(dir, "a-model", opts).Overrides.ApprovalMode).To(BeEmpty())
+
+			opts.Yolo = true
+			Expect(agentOptions(dir, "a-model", opts).Overrides.ApprovalMode).To(Equal("auto"))
+		})
+
+		// The specs above pin what is handed over. These pin what nib does with
+		// it, which is the part that was wrong: every value below reached
+		// app.Options intact and was then discarded by the config load, so a
+		// spec that stops at the struct cannot see the bug. Resolving the config
+		// the way app.Run resolves it can.
+		Describe("the config nib actually resolves", func() {
+			// writeConfig puts a config file where nib will read it, with values
+			// that disagree with every flag under test.
+			writeConfig := func(body string) {
+				Expect(os.WriteFile(ConfigPath(dir), []byte(body), 0o600)).To(Succeed())
+			}
+
+			// resolve loads the config exactly as app.Run does, so the precedence
+			// under test is nib's own rather than a restatement of it here.
+			resolve := func(o app.Options) nibtypes.Config {
+				return nibconfig.LoadWith(nibconfig.LoadOptions{
+					BaseDir:     o.BaseDir,
+					Defaults:    o.Defaults,
+					Overrides:   o.Overrides,
+					SkipBareEnv: o.SkipBareEnv,
+				})
+			}
+
+			It("sends the requests to the endpoint the flag named, not the one on disk", func() {
+				writeConfig("base_url: http://127.0.0.1:9999/v1\n")
+
+				opts := optionsWithStreams(os.Stdin, os.Stdout, os.Stderr)
+				opts.BaseURL = "http://127.0.0.1:8080/v1"
+
+				cfg := resolve(agentOptions(dir, "a-model", opts))
+				Expect(cfg.BaseURL).To(Equal("http://127.0.0.1:8080/v1"),
+					"--endpoint probed 8080; every turn has to go there too")
+			})
+
+			It("uses the model the flag named, not the one the picker recorded", func() {
+				writeConfig("model: recorded-model\n")
+
+				cfg := resolve(agentOptions(dir, "flag-model", optionsWithStreams(os.Stdin, os.Stdout, os.Stderr)))
+				Expect(cfg.Model).To(Equal("flag-model"))
+			})
+
+			It("uses the key the flag named, not the one nib saved", func() {
+				writeConfig("api_key: saved-key\n")
+
+				opts := optionsWithStreams(os.Stdin, os.Stdout, os.Stderr)
+				opts.APIKey = "flag-key"
+
+				cfg := resolve(agentOptions(dir, "a-model", opts))
+				Expect(cfg.APIKey).To(Equal("flag-key"))
+			})
+
+			It("turns approval off for --yolo even when the file demands it", func() {
+				writeConfig("approval_mode: prompt\n")
+
+				opts := optionsWithStreams(os.Stdin, os.Stdout, os.Stderr)
+				opts.Yolo = true
+
+				cfg := resolve(agentOptions(dir, "a-model", opts))
+				Expect(cfg.ApprovalMode).To(Equal("auto"))
+			})
+
+			// The other half of the same rule, and the reason an unset flag is
+			// not a demand for the empty string: an override only ever raises a
+			// field, so what the user configured survives a run that said
+			// nothing about it.
+			It("leaves what the file configured alone when no flag was given", func() {
+				writeConfig("api_key: saved-key\napproval_mode: prompt\n")
+
+				cfg := resolve(agentOptions(dir, "a-model", optionsWithStreams(os.Stdin, os.Stdout, os.Stderr)))
+				Expect(cfg.APIKey).To(Equal("saved-key"))
+				Expect(cfg.ApprovalMode).To(Equal("prompt"))
+			})
+		})
+	})
+
+	// nib reports its own failures on the error stream and returns nothing but
+	// a status, so anything that reaches here as one has already been explained
+	// once. The refusal to open a full-screen session on a stdin that cannot be
+	// read is the one users meet: 'echo q | local-ai chat' names --cli, and a
+	// second message on top would bury the fix.
+	Describe("ExitStatus", func() {
+		It("recognises a status the agent already explained", func() {
+			code, reported := ExitStatus(app.ExitError{Code: 2})
+			Expect(reported).To(BeTrue())
+			Expect(code).To(Equal(2))
+		})
+
+		It("finds one that has been wrapped", func() {
+			code, reported := ExitStatus(fmt.Errorf("running the agent: %w", app.ExitError{Code: 1}))
+			Expect(reported).To(BeTrue())
+			Expect(code).To(Equal(1))
+		})
+
+		It("leaves an ordinary failure to be reported", func() {
+			_, reported := ExitStatus(errors.New("no LocalAI server at http://127.0.0.1:8080"))
+			Expect(reported).To(BeFalse())
+		})
+
+		It("says nothing about a run that succeeded", func() {
+			_, reported := ExitStatus(nil)
+			Expect(reported).To(BeFalse())
+		})
+	})
+
+	It("reports a state dir it cannot create", func() {
+		blocked := filepath.Join(dir, "a-file")
+		Expect(os.WriteFile(blocked, []byte("not a dir"), 0o600)).To(Succeed())
+
+		opts := optionsFor(nil)
+		opts.StateDir = filepath.Join(blocked, "chat")
+		_, err := prepare(context.Background(), opts, false)
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("agent state dir"))
+	})
+})

core/cli/chat/server.go

@@ -0,0 +1,276 @@
+package chat
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"os"
+	"os/exec"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/mudler/LocalAI/pkg/httpclient"
+)
+
+// ErrDeclined means no server was started, either because the session is not
+// interactive or because the user said no.
+var ErrDeclined = errors.New("no server started")
+
+// errServerExited means the process we spawned died before it ever reported
+// ready, so there is no point in polling out the rest of the budget.
+var errServerExited = errors.New("the LocalAI server exited before it became ready")
+
+const (
+	// defaultReadyTimeout bounds the wait for a freshly spawned server. A cold
+	// start probes hardware and may pull a backend, so the budget is generous.
+	defaultReadyTimeout = 2 * time.Minute
+	// readyPollInterval is how long to wait between readiness polls.
+	readyPollInterval = 500 * time.Millisecond
+	// readyProbeTimeout bounds a single readiness request, so one connection
+	// that hangs cannot swallow the whole budget.
+	readyProbeTimeout = 5 * time.Second
+	// shutdownGrace is how long a server we started gets to unload models and
+	// stop its backends after SIGINT before it is killed outright.
+	shutdownGrace = 10 * time.Second
+	// childOutputDrainDelay bounds how long cmd.Wait keeps copying the child's
+	// output after the child itself has exited.
+	//
+	// This is not a theoretical guard for LocalAI. 'local-ai run' spawns backend
+	// subprocesses, and they inherit the write end of the pipe exec created for
+	// the child's stderr. A backend that outlives its parent holds that pipe
+	// open, so an unbounded cmd.Wait would block on the copy goroutine long
+	// after the server itself is gone: exited would never close, Stop would burn
+	// its whole grace period even on a clean shutdown, and the waiter goroutine
+	// would leak.
+	//
+	// The value is long enough that a legitimate final burst of logs is never
+	// truncated even on a loaded machine, where the copy itself takes
+	// microseconds. It must stay strictly below shutdownGrace: at or above it,
+	// every wedged-pipe shutdown would exhaust the grace period and then SIGKILL
+	// a process that had already exited cleanly.
+	childOutputDrainDelay = 5 * time.Second
+)
+
+// Confirmer asks a yes/no question. Nil means the session is not interactive.
+type Confirmer func(question string) (bool, error)
+
+// StartOptions configures OfferToStart.
+type StartOptions struct {
+	// Endpoint is the address the user expected a server on, used in the
+	// question and polled for readiness. This is the endpoint root, not the
+	// /v1 API base URL: readiness is served at the root.
+	Endpoint string
+	// Confirm asks whether to start a server. Nil means never start.
+	Confirm Confirmer
+	// Stderr receives the child's output.
+	Stderr io.Writer
+	// Executable overrides the binary to run. Empty means os.Executable().
+	Executable string
+	// ReadyTimeout bounds the wait for readiness. Zero means defaultReadyTimeout.
+	ReadyTimeout time.Duration
+}
+
+// StartedServer is a server this process started and is responsible for.
+type StartedServer struct {
+	// exited is closed once the child has been reaped. One background waiter
+	// owns cmd.Wait: it may only be called once, and it is what closes the
+	// pipes exec created for Stdout/Stderr and joins the goroutines copying
+	// them, so calling os.Process.Wait directly instead would leak both.
+	exited chan struct{}
+	// waitErr is the child's exit status. It is written before exited is
+	// closed and must only be read after that channel is observed closed.
+	waitErr error
+
+	// proc is the child. It is an interface rather than *os.Process so that
+	// Stop's contract, in particular that the child is asked to stop exactly
+	// once however often Stop is called, can be pinned without a live process
+	// to signal. Nil means nothing was ever started.
+	proc processControl
+
+	stopOnce sync.Once
+}
+
+// processControl is the part of *os.Process that Stop needs.
+//
+// One interface rather than a pair of independent function fields: two fields
+// can be wired to each other's operation, or one left nil, and no test can tell,
+// because a fake satisfies any combination. There is nothing to swap or forget
+// here, since the sole implementation is the real process and the method names
+// carry the meaning.
+type processControl interface {
+	Signal(os.Signal) error
+	Kill() error
+}
+
+// *os.Process satisfies processControl unmodified, so production needs no
+// adapter and no nil branch: the wiring is a single assignment.
+var _ processControl = (*os.Process)(nil)
+
+// newServerCommand builds the child process. Split out from OfferToStart so the
+// process' configuration can be asserted on without spawning anything.
+func newServerCommand(bin string, stderr io.Writer) *exec.Cmd {
+	cmd := exec.Command(bin, "run")
+	// Stdin is left nil, so the child gets /dev/null: it is a background
+	// server, and sharing the terminal would have it stealing keystrokes from
+	// the agent.
+	cmd.Stdout = stderr // the child's logs are diagnostics, not chat output
+	cmd.Stderr = stderr
+	// Bound the wait for the child's output pipes; see childOutputDrainDelay.
+	cmd.WaitDelay = childOutputDrainDelay
+	return cmd
+}
+
+// OfferToStart asks whether to start a LocalAI server and, if allowed, spawns
+// one and waits for it to report ready.
+//
+// A child process rather than an in-process boot: RunCMD.Run installs its own
+// signal handling and blocks until shutdown, so re-entering it from a chat
+// session would entangle two lifecycles in one process.
+func OfferToStart(ctx context.Context, opts StartOptions) (*StartedServer, error) {
+	if opts.Confirm == nil {
+		// Not interactive. Spawning a server nobody asked for is the one thing
+		// this function must never do: in CI, in a pipeline, or under a
+		// supervisor there is no one to see it or shut it down.
+		return nil, ErrDeclined
+	}
+	ok, err := opts.Confirm(fmt.Sprintf("No LocalAI server at %s. Start one now?", opts.Endpoint))
+	if err != nil {
+		return nil, fmt.Errorf("asking whether to start a server: %w", err)
+	}
+	if !ok {
+		return nil, ErrDeclined
+	}
+
+	bin := opts.Executable
+	if bin == "" {
+		if bin, err = os.Executable(); err != nil {
+			return nil, fmt.Errorf("locating the local-ai binary: %w", err)
+		}
+	}
+
+	cmd := newServerCommand(bin, opts.Stderr)
+	if err := cmd.Start(); err != nil {
+		return nil, fmt.Errorf("starting a LocalAI server with %s: %w", bin, err)
+	}
+
+	s := &StartedServer{exited: make(chan struct{}), proc: cmd.Process}
+	go func() {
+		s.waitErr = cmd.Wait()
+		close(s.exited)
+	}()
+
+	timeout := opts.ReadyTimeout
+	if timeout <= 0 {
+		timeout = defaultReadyTimeout
+	}
+	if err := waitReady(ctx, opts.Endpoint, timeout, s.exited); err != nil {
+		if errors.Is(err, errServerExited) {
+			// Safe to read: errServerExited is only returned once exited has
+			// been observed closed, which happens after waitErr is written.
+			err = describeExit(err, s.waitErr)
+		}
+		s.Stop()
+		return nil, fmt.Errorf("%w. Run 'local-ai run' in another terminal to see why it did not come up", err)
+	}
+	return s, nil
+}
+
+// describeExit adds what is known about how the child died to exitErr, without
+// putting os/exec's plumbing in front of the user.
+//
+// waitErr is exec.ErrWaitDelay when the child exited cleanly but something it
+// spawned still held its output pipe open past childOutputDrainDelay. The
+// sentinel's own text names the WaitDelay field, which is meaningless to a
+// user, so it is translated. Nothing is swallowed: os/exec only substitutes
+// ErrWaitDelay when the process itself exited without an error of its own (see
+// Cmd.Wait, "Report an error from the copying goroutines only if the program
+// otherwise exited normally"), so it can never stand in for an *ExitError.
+func describeExit(exitErr, waitErr error) error {
+	switch {
+	case waitErr == nil:
+		return exitErr
+	case errors.Is(waitErr, exec.ErrWaitDelay):
+		return fmt.Errorf("%w, and left a subprocess of its own still running", exitErr)
+	default:
+		return fmt.Errorf("%w: %w", exitErr, waitErr)
+	}
+}
+
+// Stop terminates the server this process started, giving it a chance to shut
+// down cleanly first. It is safe to call on a nil or never-started server, and
+// safe to call more than once.
+func (s *StartedServer) Stop() {
+	if s == nil || s.proc == nil {
+		return
+	}
+	s.stopOnce.Do(func() {
+		// SIGINT rather than SIGKILL: local-ai run installs its own handler and
+		// needs it to unload models and stop backend subprocesses. Killing it
+		// outright would strand those children.
+		_ = s.proc.Signal(os.Interrupt)
+
+		select {
+		case <-s.exited:
+		case <-time.After(shutdownGrace):
+			// It ignored the interrupt or wedged on the way down. The user is
+			// waiting on their shell prompt, so stop being polite.
+			_ = s.proc.Kill()
+		}
+	})
+}
+
+// waitReady polls the endpoint's /readyz until the server reports ready, the
+// budget expires, the caller gives up, or exited signals that the process we
+// are waiting on is gone. A nil exited channel means there is no process to
+// watch.
+//
+// Readiness lives on the endpoint ROOT, not under the /v1 API base URL, and it
+// answers 503 for as long as startup is still in progress.
+func waitReady(ctx context.Context, endpoint string, timeout time.Duration, exited <-chan struct{}) error {
+	url := strings.TrimSuffix(endpoint, "/") + "/readyz"
+
+	// A real deadline rather than context.WithCancel plus a timer: the latter
+	// expires as context.Canceled, which every classifier here reads as "the
+	// caller gave up" rather than "the endpoint never answered".
+	waitCtx, cancel := context.WithTimeout(ctx, timeout)
+	defer cancel()
+
+	client := httpclient.NewWithTimeout(readyProbeTimeout)
+	ticker := time.NewTicker(readyPollInterval)
+	defer ticker.Stop()
+
+	for {
+		select {
+		case <-exited:
+			return errServerExited
+		case <-waitCtx.Done():
+			// Distinguish our budget from the caller's: only ours is advice
+			// about the server.
+			if err := ctx.Err(); err != nil {
+				return err
+			}
+			return fmt.Errorf("the LocalAI server did not become ready within %s", timeout)
+		case <-ticker.C:
+		}
+
+		req, err := http.NewRequestWithContext(waitCtx, http.MethodGet, url, nil)
+		if err != nil {
+			return fmt.Errorf("building the readiness request for %s: %w", url, err)
+		}
+		resp, err := client.Do(req)
+		if err != nil {
+			continue // nothing listening yet
+		}
+		// Drain before closing so the next poll can reuse the connection
+		// instead of opening a socket every 500ms for two minutes.
+		_, _ = io.Copy(io.Discard, resp.Body)
+		_ = resp.Body.Close()
+		if resp.StatusCode == http.StatusOK {
+			return nil
+		}
+		// Anything else means startup is still in progress; keep polling.
+	}
+}

core/cli/chat/server_test.go

@@ -0,0 +1,375 @@
+package chat
+
+import (
+	"context"
+	"errors"
+	"io"
+	"net/http"
+	"net/http/httptest"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"time"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+// unusedPort is a loopback address nothing listens on, used wherever a spec
+// needs a readiness poll to keep failing. Port 1 is privileged, so no test
+// process could have bound it.
+const unusedPort = "http://127.0.0.1:1"
+
+var _ = Describe("OfferToStart", func() {
+	It("never spawns anything when there is no confirmer", func() {
+		started, err := OfferToStart(context.Background(), StartOptions{
+			Endpoint:   "http://127.0.0.1:59999",
+			Confirm:    nil,
+			Stderr:     io.Discard,
+			Executable: "/nonexistent/binary-that-must-not-run",
+		})
+		Expect(err).To(HaveOccurred())
+		Expect(errors.Is(err, ErrDeclined)).To(BeTrue(), "want ErrDeclined, got %v", err)
+		Expect(started).To(BeNil())
+	})
+
+	It("does not spawn when the user declines", func() {
+		asked := false
+		started, err := OfferToStart(context.Background(), StartOptions{
+			Endpoint: "http://127.0.0.1:59999",
+			Confirm: func(string) (bool, error) {
+				asked = true
+				return false, nil
+			},
+			Stderr:     io.Discard,
+			Executable: "/nonexistent/binary-that-must-not-run",
+		})
+		Expect(asked).To(BeTrue(), "the user should have been asked")
+		Expect(errors.Is(err, ErrDeclined)).To(BeTrue())
+		Expect(started).To(BeNil())
+	})
+
+	It("names the endpoint in the question", func() {
+		var question string
+		_, _ = OfferToStart(context.Background(), StartOptions{
+			Endpoint: "http://example.invalid:9090",
+			Confirm: func(q string) (bool, error) {
+				question = q
+				return false, nil
+			},
+			Stderr:     io.Discard,
+			Executable: "/nonexistent/binary-that-must-not-run",
+		})
+		Expect(question).To(ContainSubstring("http://example.invalid:9090"))
+	})
+
+	It("propagates a confirmer error", func() {
+		boom := errors.New("boom")
+		_, err := OfferToStart(context.Background(), StartOptions{
+			Endpoint:   "http://127.0.0.1:59999",
+			Confirm:    func(string) (bool, error) { return false, boom },
+			Stderr:     io.Discard,
+			Executable: "/nonexistent/binary-that-must-not-run",
+		})
+		Expect(errors.Is(err, boom)).To(BeTrue())
+	})
+
+	It("reports which binary it failed to launch", func() {
+		started, err := OfferToStart(context.Background(), StartOptions{
+			Endpoint:   "http://127.0.0.1:59999",
+			Confirm:    func(string) (bool, error) { return true, nil },
+			Stderr:     io.Discard,
+			Executable: "/nonexistent/binary-that-must-not-run",
+		})
+		Expect(started).To(BeNil())
+		Expect(err).To(MatchError(ContainSubstring("starting a LocalAI server")))
+		Expect(err).To(MatchError(ContainSubstring("/nonexistent/binary-that-must-not-run")))
+	})
+
+	It("stops waiting as soon as the process it started exits", func() {
+		// A harmless no-op binary rather than a real server: this exercises the
+		// early-exit path without starting LocalAI, binding a port, or running
+		// 'local-ai run'. Without early-exit detection the call would sit here
+		// polling until ReadyTimeout.
+		bin, lookErr := exec.LookPath("true")
+		if lookErr != nil {
+			Skip("no 'true' binary on PATH to stand in for a server that dies at once")
+		}
+
+		start := time.Now()
+		started, err := OfferToStart(context.Background(), StartOptions{
+			Endpoint:     unusedPort,
+			Confirm:      func(string) (bool, error) { return true, nil },
+			Stderr:       io.Discard,
+			Executable:   bin,
+			ReadyTimeout: 30 * time.Second,
+		})
+		Expect(started).To(BeNil())
+		Expect(err).To(MatchError(ContainSubstring("exited before it became ready")))
+		Expect(time.Since(start)).To(BeNumerically("<", 10*time.Second),
+			"the wait should end with the process, not with the readiness budget")
+	})
+
+	It("gives up on a child whose grandchildren still hold its output pipe", func() {
+		// The real LocalAI shape: 'local-ai run' exits but a backend
+		// subprocess it spawned inherited the stderr pipe and keeps it open.
+		// Without cmd.WaitDelay, cmd.Wait blocks on the copy goroutine, exited
+		// never closes, and the readiness wait runs out the full budget instead
+		// of reporting that the server died.
+		sh, lookErr := exec.LookPath("sh")
+		if lookErr != nil {
+			Skip("no 'sh' binary on PATH to stand in for a server with a lingering child")
+		}
+
+		dir := GinkgoT().TempDir()
+		pidFile := filepath.Join(dir, "grandchild.pid")
+		script := filepath.Join(dir, "server-with-lingering-child")
+		// #nosec G306 -- this has to be executable to stand in for a binary.
+		Expect(os.WriteFile(script,
+			[]byte("#!"+sh+"\nsleep 30 &\necho $! > "+pidFile+"\nexit 0\n"),
+			0o700)).To(Succeed())
+
+		// Reap the grandchild whatever happens: it outlives its own parent by
+		// design, so nothing else will clean it up.
+		DeferCleanup(func() {
+			raw, err := os.ReadFile(pidFile)
+			if err != nil {
+				return
+			}
+			pid, err := strconv.Atoi(strings.TrimSpace(string(raw)))
+			if err != nil {
+				return
+			}
+			proc, err := os.FindProcess(pid)
+			if err != nil {
+				return
+			}
+			_ = proc.Kill()
+			_, _ = proc.Wait()
+		})
+
+		start := time.Now()
+		started, err := OfferToStart(context.Background(), StartOptions{
+			Endpoint:     unusedPort,
+			Confirm:      func(string) (bool, error) { return true, nil },
+			Stderr:       io.Discard,
+			Executable:   script,
+			ReadyTimeout: 25 * time.Second,
+		})
+		elapsed := time.Since(start)
+
+		Expect(started).To(BeNil())
+		Expect(err).To(MatchError(ContainSubstring("exited before it became ready")),
+			"an unbounded cmd.Wait would report a readiness timeout instead")
+		Expect(elapsed).To(BeNumerically("<", 20*time.Second),
+			"the wait must be bounded by the output drain, not by the readiness budget")
+
+		// This is the case where cmd.Wait returns exec.ErrWaitDelay, whose own
+		// text names a struct field of os/exec. Users get told what happened
+		// instead.
+		Expect(err).NotTo(MatchError(ContainSubstring("WaitDelay")),
+			"os/exec plumbing must not reach the user")
+		Expect(err).NotTo(MatchError(ContainSubstring("exec:")))
+		Expect(err).To(MatchError(ContainSubstring("left a subprocess of its own still running")))
+	})
+
+	It("reports the exit status of a server that failed outright", func() {
+		// The counterpart to the case above: translating ErrWaitDelay must not
+		// cost a real exit status, which is the one diagnostic worth having.
+		bin, lookErr := exec.LookPath("false")
+		if lookErr != nil {
+			Skip("no 'false' binary on PATH to stand in for a server that fails")
+		}
+
+		_, err := OfferToStart(context.Background(), StartOptions{
+			Endpoint:     unusedPort,
+			Confirm:      func(string) (bool, error) { return true, nil },
+			Stderr:       io.Discard,
+			Executable:   bin,
+			ReadyTimeout: 30 * time.Second,
+		})
+		Expect(err).To(MatchError(ContainSubstring("exited before it became ready")))
+		Expect(err).To(MatchError(ContainSubstring("exit status 1")))
+	})
+})
+
+var _ = Describe("StartedServer.Stop", func() {
+	It("is a no-op on a server that was never started", func() {
+		var nilServer *StartedServer
+		Expect(nilServer.Stop).NotTo(Panic())
+		Expect((&StartedServer{}).Stop).NotTo(Panic())
+	})
+
+	It("interrupts the child exactly once however often it is called", func() {
+		s, proc := stoppableServer()
+
+		s.Stop()
+		s.Stop()
+		s.Stop()
+
+		Expect(proc.interrupts.Load()).To(Equal(int32(1)),
+			"a second Stop must not signal the child again")
+		Expect(proc.kills.Load()).To(BeZero(), "a child that already exited must not be killed")
+	})
+
+	It("interrupts the child exactly once when called concurrently", func() {
+		// The realistic double-Stop: a deferred Stop on the way out racing the
+		// signal handler that also owns shutting the server down.
+		const callers = 8
+
+		s, proc := stoppableServer()
+
+		var wg sync.WaitGroup
+		wg.Add(callers)
+		for range callers {
+			go func() {
+				defer GinkgoRecover()
+				defer wg.Done()
+				s.Stop()
+			}()
+		}
+		wg.Wait()
+
+		Expect(proc.interrupts.Load()).To(Equal(int32(1)))
+		Expect(proc.kills.Load()).To(BeZero())
+	})
+
+	It("asks the child to interrupt rather than killing it outright", func() {
+		// The escalation order is the whole point of the grace period: SIGKILL
+		// first would strand the backend subprocesses local-ai run owns.
+		s, proc := stoppableServer()
+
+		s.Stop()
+
+		Expect(proc.lastSignal.Load()).To(Equal(os.Interrupt))
+		Expect(proc.kills.Load()).To(BeZero())
+	})
+})
+
+// countingProcess stands in for the *os.Process that Stop drives, recording
+// what it was asked to do.
+type countingProcess struct {
+	interrupts atomic.Int32
+	kills      atomic.Int32
+	lastSignal atomic.Value
+}
+
+func (p *countingProcess) Signal(sig os.Signal) error {
+	p.interrupts.Add(1)
+	p.lastSignal.Store(sig)
+	return nil
+}
+
+func (p *countingProcess) Kill() error {
+	p.kills.Add(1)
+	return nil
+}
+
+// stoppableServer builds a StartedServer whose child has already exited, driven
+// by a countingProcess rather than a real one. Nothing is spawned.
+func stoppableServer() (*StartedServer, *countingProcess) {
+	proc := &countingProcess{}
+	exited := make(chan struct{})
+	close(exited)
+	return &StartedServer{exited: exited, proc: proc}, proc
+}
+
+var _ = Describe("newServerCommand", func() {
+	It("bounds how long it will wait for the child's output pipes", func() {
+		cmd := newServerCommand("/nonexistent/binary-that-must-not-run", io.Discard)
+
+		// An unbounded wait is the failure mode: backend subprocesses inherit
+		// the child's stderr pipe and can hold it open long after the server
+		// itself is gone.
+		Expect(cmd.WaitDelay).To(BeNumerically(">", 0), "cmd.Wait must not be unbounded")
+		Expect(cmd.WaitDelay).To(BeNumerically("<", shutdownGrace),
+			"a drain longer than the shutdown grace would kill a cleanly exited server")
+	})
+
+	It("runs the server subcommand without giving it the terminal", func() {
+		cmd := newServerCommand("/nonexistent/binary-that-must-not-run", io.Discard)
+
+		Expect(cmd.Args).To(Equal([]string{"/nonexistent/binary-that-must-not-run", "run"}))
+		Expect(cmd.Stdin).To(BeNil(), "the child must not compete with the agent for stdin")
+		Expect(cmd.Stdout).NotTo(BeNil())
+		Expect(cmd.Stderr).NotTo(BeNil())
+	})
+})
+
+var _ = Describe("waitReady", func() {
+	It("polls /readyz on the endpoint root and returns only once it answers 200", func() {
+		// readyOnPoll is deliberately above 1. A handler that answers 200 to the
+		// first poll cannot tell a correct implementation apart from one that
+		// treats 503 as ready, because both return after a single request; the
+		// poll count is what makes 503-as-ready observable.
+		const readyOnPoll = 3
+
+		var polls atomic.Int32
+		var paths atomic.Value
+		paths.Store("")
+
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			paths.Store(r.URL.Path)
+			if polls.Add(1) < readyOnPoll {
+				// What LocalAI answers while startup is still in progress.
+				w.WriteHeader(http.StatusServiceUnavailable)
+				return
+			}
+			w.WriteHeader(http.StatusOK)
+		}))
+		defer srv.Close()
+
+		Expect(waitReady(context.Background(), srv.URL, 20*time.Second, nil)).To(Succeed())
+		Expect(paths.Load()).To(Equal("/readyz"), "readiness lives on the endpoint root, not under /v1")
+		Expect(polls.Load()).To(BeNumerically(">=", readyOnPoll),
+			"503 means startup is still in progress and must never be accepted as ready")
+	})
+
+	It("tolerates a trailing slash on the endpoint", func() {
+		var path atomic.Value
+		path.Store("")
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			path.Store(r.URL.Path)
+			w.WriteHeader(http.StatusOK)
+		}))
+		defer srv.Close()
+
+		Expect(waitReady(context.Background(), srv.URL+"/", 20*time.Second, nil)).To(Succeed())
+		Expect(path.Load()).To(Equal("/readyz"))
+	})
+
+	It("reports a timeout, not a cancellation, when the budget runs out", func() {
+		err := waitReady(context.Background(), unusedPort, 1200*time.Millisecond, nil)
+		Expect(err).To(HaveOccurred())
+		// A budget built from context.WithCancel plus a timer would surface as
+		// context.Canceled, which downstream code reads as "the caller gave up"
+		// and would stop classifying a hung server as unreachable.
+		Expect(errors.Is(err, context.Canceled)).To(BeFalse(), "got %v", err)
+		Expect(err).To(MatchError(ContainSubstring("did not become ready")))
+	})
+
+	It("returns the caller's cancellation when the caller gives up", func() {
+		ctx, cancel := context.WithCancel(context.Background())
+		go func() {
+			defer GinkgoRecover()
+			time.Sleep(200 * time.Millisecond)
+			cancel()
+		}()
+		defer cancel()
+
+		err := waitReady(ctx, unusedPort, time.Minute, nil)
+		Expect(errors.Is(err, context.Canceled)).To(BeTrue(), "got %v", err)
+	})
+
+	It("gives up when the process it is waiting on has exited", func() {
+		exited := make(chan struct{})
+		close(exited)
+
+		err := waitReady(context.Background(), unusedPort, time.Minute, exited)
+		Expect(err).To(MatchError(ContainSubstring("exited before it became ready")))
+	})
+})

core/cli/chat/session.go

@@ -1,112 +0,0 @@
-package chat
-
-import (
-	"context"
-	"errors"
-	"fmt"
-	"io"
-	"slices"
-	"strings"
-)
-
-const (
-	chatRoleUser      = "user"
-	chatRoleAssistant = "assistant"
-)
-
-type chatMessage struct {
-	Role    string
-	Content string
-}
-
-type chatSession struct {
-	client   chatClient
-	model    string
-	models   []string
-	messages []chatMessage
-}
-
-func newChatSession(ctx context.Context, client chatClient, requestedModel string) (*chatSession, error) {
-	models, err := client.ListModels(ctx)
-	if err != nil {
-		return nil, fmt.Errorf("list models: %w", err)
-	}
-
-	model, err := resolveChatModel(requestedModel, models)
-	if err != nil {
-		return nil, err
-	}
-
-	return &chatSession{
-		client: client,
-		model:  model,
-		models: models,
-	}, nil
-}
-
-func (s *chatSession) CurrentModel() string {
-	return s.model
-}
-
-func (s *chatSession) Models() []string {
-	models := make([]string, len(s.models))
-	copy(models, s.models)
-	return models
-}
-
-func (s *chatSession) Clear() {
-	s.messages = nil
-}
-
-func (s *chatSession) SwitchModel(model string) error {
-	if !slices.Contains(s.models, model) {
-		return fmt.Errorf("model %q is not available. Use /models to see installed models", model)
-	}
-	s.model = model
-	s.Clear()
-	return nil
-}
-
-func (s *chatSession) Send(ctx context.Context, prompt string, out io.Writer) error {
-	s.messages = append(s.messages, chatMessage{
-		Role:    chatRoleUser,
-		Content: prompt,
-	})
-
-	answer, err := s.client.StreamChat(ctx, s.model, s.messages, out)
-	if err != nil {
-		return err
-	}
-
-	s.messages = append(s.messages, chatMessage{
-		Role:    chatRoleAssistant,
-		Content: answer,
-	})
-	return nil
-}
-
-func resolveChatModel(requested string, models []string) (string, error) {
-	switch {
-	case requested == "" && len(models) == 0:
-		return "", errors.New(`no chat models are installed.
-
-Install a model first, for example:
-  local-ai models list
-  local-ai models install <model>
-  local-ai run
-
-Then start a chat session:
-  local-ai chat --model <model>`)
-	case requested == "" && len(models) == 1:
-		return models[0], nil
-	case requested == "" && len(models) > 1:
-		var b strings.Builder
-		b.WriteString("multiple models are available; choose one with --model:\n")
-		b.WriteString(formatChatModelList(models, ""))
-		return "", errors.New(b.String())
-	case !slices.Contains(models, requested):
-		return "", fmt.Errorf("model %q is not available. Use `local-ai models list` and `local-ai models install <model>`, or pass an installed model with --model", requested)
-	default:
-		return requested, nil
-	}
-}

core/cli/chat/session_test.go

@@ -1,56 +0,0 @@
-package chat
-
-import (
-	"context"
-	"io"
-
-	. "github.com/onsi/ginkgo/v2"
-	. "github.com/onsi/gomega"
-)
-
-var _ = Describe("Chat session", func() {
-	It("keeps model switching and message history out of the terminal adapter", func() {
-		client := &fakeChatClient{
-			models: []string{"alpha", "beta"},
-			answer: "pong",
-		}
-
-		session, err := newChatSession(context.Background(), client, "alpha")
-		Expect(err).ToNot(HaveOccurred())
-		Expect(session.CurrentModel()).To(Equal("alpha"))
-
-		Expect(session.SwitchModel("beta")).To(Succeed())
-		Expect(session.CurrentModel()).To(Equal("beta"))
-		Expect(session.Send(context.Background(), "ping", io.Discard)).To(Succeed())
-
-		Expect(client.requests).To(HaveLen(1))
-		Expect(client.requests[0].model).To(Equal("beta"))
-		Expect(client.requests[0].messages).To(HaveLen(1))
-		Expect(client.requests[0].messages[0].Content).To(Equal("ping"))
-	})
-})
-
-type fakeChatClient struct {
-	models   []string
-	answer   string
-	requests []fakeChatRequest
-}
-
-type fakeChatRequest struct {
-	model    string
-	messages []chatMessage
-}
-
-func (c *fakeChatClient) ListModels(context.Context) ([]string, error) {
-	return c.models, nil
-}
-
-func (c *fakeChatClient) StreamChat(_ context.Context, model string, messages []chatMessage, out io.Writer) (string, error) {
-	copied := make([]chatMessage, len(messages))
-	copy(copied, messages)
-	c.requests = append(c.requests, fakeChatRequest{model: model, messages: copied})
-	if _, err := io.WriteString(out, c.answer); err != nil {
-		return "", err
-	}
-	return c.answer, nil
-}

core/cli/chat/terminal.go

@@ -1,93 +0,0 @@
-package chat
-
-import (
-	"bufio"
-	"context"
-	"fmt"
-	"io"
-	"strings"
-)
-
-func runTerminalChat(ctx context.Context, session *chatSession, in io.Reader, out io.Writer) error {
-	scanner := bufio.NewScanner(in)
-	scanner.Buffer(make([]byte, 0, 64*1024), 4*1024*1024)
-
-	if err := writeChat(out, "LocalAI chat (%s)\n", session.CurrentModel()); err != nil {
-		return err
-	}
-	if err := writeChat(out, "Type /exit to quit, /clear to reset the conversation, /models to list models.\n"); err != nil {
-		return err
-	}
-
-	for {
-		if err := writeChat(out, "\n> "); err != nil {
-			return err
-		}
-		if !scanner.Scan() {
-			break
-		}
-
-		prompt := strings.TrimSpace(scanner.Text())
-		switch prompt {
-		case "":
-			continue
-		case "/bye", "/exit", "/quit":
-			return writeChat(out, "bye\n")
-		case "/clear":
-			session.Clear()
-			if err := writeChat(out, "conversation cleared\n"); err != nil {
-				return err
-			}
-			continue
-		case "/models":
-			if err := printChatModels(out, session.Models(), session.CurrentModel()); err != nil {
-				return err
-			}
-			continue
-		}
-
-		if nextModel, ok := strings.CutPrefix(prompt, "/model "); ok {
-			nextModel = strings.TrimSpace(nextModel)
-			if nextModel == "" {
-				if err := writeChat(out, "usage: /model <name>\n"); err != nil {
-					return err
-				}
-				continue
-			}
-			if err := session.SwitchModel(nextModel); err != nil {
-				if writeErr := writeChat(out, "%s\n", err); writeErr != nil {
-					return writeErr
-				}
-				continue
-			}
-			if err := writeChat(out, "switched to %s; conversation cleared\n", session.CurrentModel()); err != nil {
-				return err
-			}
-			continue
-		}
-
-		if err := writeChat(out, "assistant: "); err != nil {
-			return err
-		}
-		if err := session.Send(ctx, prompt, out); err != nil {
-			return err
-		}
-		if err := writeChat(out, "\n"); err != nil {
-			return err
-		}
-	}
-
-	return scanner.Err()
-}
-
-func printChatModels(out io.Writer, models []string, current string) error {
-	if len(models) == 0 {
-		return writeChat(out, "no models installed\n")
-	}
-	return writeChat(out, "%s", formatChatModelList(models, current))
-}
-
-func writeChat(out io.Writer, format string, args ...any) error {
-	_, err := fmt.Fprintf(out, format, args...)
-	return err
-}

core/cli/chat_cmd.go

@@ -8,18 +8,72 @@ import (
 	cliContext "github.com/mudler/LocalAI/core/cli/context"
 )
 
+// ChatCMD runs the built-in terminal agent. Everything after the first
+// positional argument is forwarded to the agent verbatim, so its own
+// subcommands (plugin, skill, mcp) and their flags work unchanged. LocalAI's
+// own flags must therefore come first.
 type ChatCMD struct {
-	Model    string `short:"m" help:"Model name to use. Defaults to the only model returned by the server when exactly one is available"`
-	Endpoint string `env:"LOCALAI_CHAT_ENDPOINT" default:"http://127.0.0.1:8080" help:"LocalAI server endpoint. The /v1 path is added automatically when omitted"`
-	APIKey   string `env:"LOCALAI_API_KEY,API_KEY" help:"API key to use when the LocalAI server requires authentication"`
+	Model     string `short:"m" help:"Model to use. Defaults to the only model the server offers, or asks when there are several"`
+	Endpoint  string `env:"LOCALAI_CHAT_ENDPOINT" default:"http://127.0.0.1:8080" help:"LocalAI server endpoint. The /v1 path is added automatically when omitted"`
+	APIKey    string `env:"LOCALAI_API_KEY,API_KEY" help:"API key to use when the LocalAI server requires authentication"`
+	ConfigDir string `env:"LOCALAI_CHAT_CONFIG_DIR" help:"Directory holding the agent's config, plugins, and skills. Defaults to ~/.config/localai/chat" type:"path"`
+	TraceDir  string `env:"LOCALAI_CHAT_TRACE_DIR" help:"Write a session LLM trace (NDJSON) to this directory" type:"path"`
+
+	CLI    bool   `help:"Run in plain CLI mode instead of the full-screen interface"`
+	TUI    bool   `help:"Force the full-screen interface"`
+	Height string `help:"Run as an inline drop-down of this height, e.g. '40%'"`
+	Tmux   bool   `help:"Run in a tmux split"`
+	NoTmux bool   `name:"no-tmux" help:"Never use a tmux split, even inside tmux"`
+	Init   string `help:"Print the shell integration script for Ctrl+Space (zsh, bash, or fish)"`
+	Yolo   bool   `env:"LOCALAI_CHAT_YOLO" help:"Auto-approve every tool call without prompting"`
+
+	Args []string `arg:"" optional:"" passthrough:"" help:"Arguments forwarded to the agent, e.g. 'plugin install <url>', 'skill list', 'mcp add'"`
 }
 
 func (c *ChatCMD) Run(ctx *cliContext.Context) error {
-	return chatcli.Run(context.Background(), chatcli.Options{
-		Model:   c.Model,
-		BaseURL: chatAPIBaseURL(c.Endpoint),
-		APIKey:  c.APIKey,
-		In:      os.Stdin,
-		Out:     os.Stdout,
+	err := chatcli.Run(context.Background(), chatcli.Options{
+		Args:     c.agentArgs(),
+		Endpoint: c.Endpoint,
+		BaseURL:  chatAPIBaseURL(c.Endpoint),
+		APIKey:   c.APIKey,
+		Model:    c.Model,
+		StateDir: c.ConfigDir,
+		TraceDir: c.TraceDir,
+		Yolo:     c.Yolo,
+		In:       os.Stdin,
+		Out:      os.Stdout,
+		ErrOut:   os.Stderr,
 	})
+	// The agent explains its own failures on stderr and hands back a code, so
+	// carry the code out and leave the explanation to stand alone.
+	if code, reported := chatcli.ExitStatus(err); reported {
+		return ExitCodeError{Code: code}
+	}
+	return err
+}
+
+// agentArgs rebuilds the argument vector the agent expects: LocalAI's mode
+// flags are declared here for discoverability and shell completion, so they
+// have to be translated back into the agent's own flag names.
+func (c *ChatCMD) agentArgs() []string {
+	var args []string
+	if c.CLI {
+		args = append(args, "--cli")
+	}
+	if c.TUI {
+		args = append(args, "--tui")
+	}
+	if c.Height != "" {
+		args = append(args, "--height", c.Height)
+	}
+	if c.Tmux {
+		args = append(args, "--tmux")
+	}
+	if c.NoTmux {
+		args = append(args, "--no-tmux")
+	}
+	if c.Init != "" {
+		args = append(args, "--init", c.Init)
+	}
+	return append(args, c.Args...)
 }

core/cli/chat_cmd_test.go

@@ -1,6 +1,10 @@
 package cli
 
 import (
+	"errors"
+	"fmt"
+
+	"github.com/alecthomas/kong"
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
 )
@@ -24,4 +28,70 @@ var _ = Describe("Chat command wiring", func() {
 			Expect(chatAPIBaseURL("http://127.0.0.1:8080/localai")).To(Equal("http://127.0.0.1:8080/localai/v1"))
 		})
 	})
+
+	Describe("argument parsing", func() {
+		parse := func(args ...string) *ChatCMD {
+			var cli struct {
+				Chat ChatCMD `cmd:""`
+			}
+			parser, err := kong.New(&cli)
+			Expect(err).ToNot(HaveOccurred())
+			_, err = parser.Parse(append([]string{"chat"}, args...))
+			Expect(err).ToNot(HaveOccurred())
+			return &cli.Chat
+		}
+
+		It("leaves Args empty for a bare invocation", func() {
+			Expect(parse().Args).To(BeEmpty())
+		})
+
+		It("binds flags that precede the forwarded arguments", func() {
+			c := parse("--endpoint", "http://host:9090", "--model", "m", "plugin", "list")
+			Expect(c.Endpoint).To(Equal("http://host:9090"))
+			Expect(c.Model).To(Equal("m"))
+			Expect(c.Args).To(Equal([]string{"plugin", "list"}))
+		})
+
+		It("forwards flags that follow the first positional to the agent", func() {
+			c := parse("plugin", "install", "https://example.invalid/p", "--yes")
+			Expect(c.Args).To(Equal([]string{"plugin", "install", "https://example.invalid/p", "--yes"}))
+		})
+
+		It("parses its own mode flags", func() {
+			c := parse("--cli")
+			Expect(c.CLI).To(BeTrue())
+			Expect(c.Args).To(BeEmpty())
+		})
+	})
+
+	// The agent prints its own diagnosis and hands back a status. main exits
+	// with that status and prints nothing more, so the user reads one message
+	// rather than an "exit status 1" stacked under it.
+	Describe("ExitCodeError", func() {
+		It("carries the status out", func() {
+			Expect(ExitCodeError{Code: 2}.Code).To(Equal(2))
+		})
+
+		It("is recognisable after wrapping", func() {
+			var got ExitCodeError
+			Expect(errors.As(fmt.Errorf("chat: %w", ExitCodeError{Code: 2}), &got)).To(BeTrue())
+			Expect(got.Code).To(Equal(2))
+		})
+	})
+
+	Describe("agentArgs", func() {
+		It("translates mode flags into the agent's own flags", func() {
+			c := &ChatCMD{CLI: true}
+			Expect(c.agentArgs()).To(Equal([]string{"--cli"}))
+		})
+
+		It("puts forwarded arguments after the translated flags", func() {
+			c := &ChatCMD{Height: "40%", Args: []string{"plugin", "list"}}
+			Expect(c.agentArgs()).To(Equal([]string{"--height", "40%", "plugin", "list"}))
+		})
+
+		It("returns nothing for a bare invocation", func() {
+			Expect((&ChatCMD{}).agentArgs()).To(BeEmpty())
+		})
+	})
 })

core/cli/cli.go

@@ -9,7 +9,7 @@ var CLI struct {
 	cliContext.Context `embed:""`
 
 	Run             RunCMD             `cmd:"" help:"Run LocalAI, this the default command if no other command is specified. Run 'local-ai run --help' for more information" default:"withargs"`
-	Chat            ChatCMD            `cmd:"" help:"Open an interactive chat session against a running LocalAI server"`
+	Chat            ChatCMD            `cmd:"" help:"Run the built-in terminal agent against a LocalAI server"`
 	Federated       FederatedCLI       `cmd:"" help:"Run LocalAI in federated mode"`
 	Models          ModelsCMD          `cmd:"" help:"Manage LocalAI models and definitions"`
 	Backends        BackendsCMD        `cmd:"" help:"Manage LocalAI backends and definitions"`

core/cli/exit.go

@@ -0,0 +1,15 @@
+package cli
+
+import "fmt"
+
+// ExitCodeError is a failure a command has already reported to the user. It
+// carries nothing but the status the process should exit with, and main prints
+// nothing more for it.
+//
+// It exists for commands that hand their terminal to something that does its
+// own error reporting. Returning that subordinate's error instead would put a
+// bare "exit status 1" underneath the explanation the user has just read, and
+// returning nil would tell a script the run succeeded.
+type ExitCodeError struct{ Code int }
+
+func (e ExitCodeError) Error() string { return fmt.Sprintf("exit status %d", e.Code) }

core/config/backend_capabilities.go

@@ -44,6 +44,7 @@ const (
 	MethodPredictStream      GRPCMethod = "PredictStream"
 	MethodEmbedding          GRPCMethod = "Embedding"
 	MethodGenerateImage      GRPCMethod = "GenerateImage"
+	MethodUpscaleImage       GRPCMethod = "UpscaleImage"
 	MethodGenerateVideo      GRPCMethod = "GenerateVideo"
 	MethodGenerate3D         GRPCMethod = "Generate3D"
 	MethodAudioTranscription GRPCMethod = "AudioTranscription"
@@ -348,7 +349,7 @@ var BackendCapabilities = map[string]BackendCapability{
 
 	// --- Image/video generation backends ---
 	"diffusers": {
-		GRPCMethods:      []GRPCMethod{MethodGenerateImage, MethodGenerateVideo},
+		GRPCMethods:      []GRPCMethod{MethodGenerateImage, MethodUpscaleImage, MethodGenerateVideo},
 		PossibleUsecases: []string{UsecaseImage, UsecaseVideo},
 		DefaultUsecases:  []string{UsecaseImage},
 		Description:      "HuggingFace diffusers — Stable Diffusion, Flux, video generation",

core/config/meta/constants.go

@@ -38,6 +38,14 @@ var CacheTypeOptions = []FieldOption{
 	{Value: "q4_1", Label: "Q4_1"},
 	{Value: "q5_0", Label: "Q5_0"},
 	{Value: "q5_1", Label: "Q5_1"},
+	// TurboQuant KV-cache types — accepted by the turboquant and
+	// buun-llama-cpp fork backends; stock llama-cpp will reject them at load.
+	{Value: "turbo2", Label: "Turbo2 (TurboQuant)"},
+	{Value: "turbo3", Label: "Turbo3 (TurboQuant)"},
+	{Value: "turbo4", Label: "Turbo4 (TurboQuant)"},
+	// Trellis-Coded Quantization variants — buun-llama-cpp only.
+	{Value: "turbo2_tcq", Label: "Turbo2 TCQ (buun-llama-cpp)"},
+	{Value: "turbo3_tcq", Label: "Turbo3 TCQ (buun-llama-cpp)"},
 }
 
 var DiffusersPipelineOptions = []FieldOption{

core/gallery/estimate_warm.go

@@ -0,0 +1,211 @@
+package gallery
+
+import (
+	"context"
+	"os"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/pkg/system"
+	"github.com/mudler/LocalAI/pkg/vram"
+	"github.com/mudler/xlog"
+)
+
+// EstimateInput builds the VRAM estimator's input from a gallery entry.
+//
+// It lives here rather than beside the HTTP handler because two callers need
+// it: the handler answering one model, and the warmer below answering all of
+// them ahead of time.
+func EstimateInput(m *GalleryModel) vram.ModelEstimateInput {
+	var input vram.ModelEstimateInput
+	input.Size = m.Size
+	if repoID := extractHFRepo(m.Overrides, m.URLs); repoID != "" {
+		input.HFRepo = repoID
+	}
+	for _, f := range m.AdditionalFiles {
+		if vram.IsWeightFile(f.URI) {
+			input.Files = append(input.Files, vram.FileInput{URI: f.URI, Size: 0})
+		}
+	}
+	return input
+}
+
+// extractHFRepo finds a HuggingFace repo ID in a model's overrides or URLs.
+func extractHFRepo(overrides map[string]any, urls []string) string {
+	if overrides != nil {
+		if params, ok := overrides["parameters"].(map[string]any); ok {
+			if modelRef, ok := params["model"].(string); ok {
+				if repoID, ok := vram.ExtractHFRepoID(modelRef); ok {
+					return repoID
+				}
+			}
+		}
+	}
+	for _, u := range urls {
+		if repoID, ok := vram.ExtractHFRepoID(u); ok {
+			return repoID
+		}
+	}
+	return ""
+}
+
+// EstimateWarmConfig bounds the background warm-up.
+type EstimateWarmConfig struct {
+	// Limit is how many gallery entries to warm, in gallery order. Zero
+	// disables warming entirely. The order matters: it is the order the UI
+	// lists them in, so the entries a user sees first are warmed first.
+	Limit int
+	// Concurrency is how many estimates run at once. Each one can be a remote
+	// probe, so this is deliberately small: the point is to be finished before
+	// anybody looks, not to saturate the link or the upstream.
+	Concurrency int
+	// Contexts are the context lengths to estimate at. These want to match what
+	// the UI asks for, or the warmed entry is not the one it reads.
+	Contexts []uint32
+}
+
+// DefaultEstimateWarmConfig is what the server uses unless told otherwise.
+//
+// The limit is a deliberate compromise. Warming the whole gallery would be
+// thousands of remote probes on every boot, which is rude to the upstream and
+// slow to finish; warming nothing leaves the first page of the model gallery
+// paying two seconds per row. A few hundred covers what anyone browses in a
+// sitting, and everything past it still warms itself on first view.
+var DefaultEstimateWarmConfig = EstimateWarmConfig{
+	Limit:       300,
+	Concurrency: 4,
+	Contexts:    []uint32{8192, 16384, 32768, 65536, 131072, 262144},
+}
+
+// WarmEstimateCache fills the gallery's derived caches in the background.
+//
+// Two things are warmed, and they are the same cost wearing different hats.
+// An estimate for an entry the server has never seen costs a network probe of
+// its weight files, and describing an entry's variants costs one probe per
+// build it offers. The UI asks for an estimate per row and a variant
+// description per model opened, so without this the first visitor pays for
+// both: ten seconds of a page filling in its own sizes, then another second
+// and a half the first time they click anything.
+//
+// Both land in the same caches underneath, which is why one pass covers them.
+//
+// It returns immediately; the work happens on its own goroutine and stops when
+// ctx is done. Failures are logged at debug and otherwise ignored: a warm-up
+// that cannot reach an upstream must never stop the server from starting, and
+// the entry it failed on simply stays cold.
+func WarmEstimateCache(ctx context.Context, galleries []config.Gallery, systemState *system.SystemState, cfg EstimateWarmConfig) {
+	if cfg.Limit <= 0 || cfg.Concurrency <= 0 {
+		return
+	}
+
+	go func() {
+		started := time.Now()
+
+		models, err := AvailableGalleryModelsCached(galleries, systemState)
+		if err != nil {
+			xlog.Debug("VRAM estimate warm-up skipped, gallery unavailable", "error", err)
+			return
+		}
+		if len(models) > cfg.Limit {
+			models = models[:cfg.Limit]
+		}
+		if len(models) == 0 {
+			return
+		}
+
+		// The host gate the variant picker resolves against. Derived once: it
+		// describes this machine, not this entry, and HostResolveEnv reads the
+		// system state to build it.
+		env := HostResolveEnv(ctx, systemState)
+
+		var (
+			wg             sync.WaitGroup
+			cursor         = make(chan *GalleryModel)
+			warmed         int
+			warmedVariants int
+			mu             sync.Mutex
+		)
+
+		for i := 0; i < cfg.Concurrency; i++ {
+			wg.Add(1)
+			go func() {
+				defer wg.Done()
+				for m := range cursor {
+					// Per entry, not for the run: one unreachable weight file
+					// must not hold a worker for the whole warm-up.
+					entryCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
+
+					input := EstimateInput(m)
+					if len(input.Files) > 0 || input.HFRepo != "" || input.Size != "" {
+						if _, err := vram.EstimateModelMultiContext(entryCtx, input, cfg.Contexts); err != nil {
+							xlog.Debug("VRAM estimate warm-up failed for entry", "model", m.GetName(), "error", err)
+						} else {
+							mu.Lock()
+							warmed++
+							mu.Unlock()
+						}
+					}
+
+					// Describing variants probes each build the entry offers.
+					// An entry that declares none costs nothing here, so this is
+					// gated rather than attempted and discarded.
+					if m.HasVariants() {
+						if _, err := DescribeVariants(models, m, env); err != nil {
+							xlog.Debug("variant warm-up failed for entry", "model", m.GetName(), "error", err)
+						} else {
+							mu.Lock()
+							warmedVariants++
+							mu.Unlock()
+						}
+					}
+
+					cancel()
+				}
+			}()
+		}
+
+	feed:
+		for _, m := range models {
+			select {
+			case <-ctx.Done():
+				break feed
+			case cursor <- m:
+			}
+		}
+		close(cursor)
+		wg.Wait()
+
+		if ctx.Err() != nil {
+			xlog.Debug("gallery warm-up stopped", "estimates", warmed, "variants", warmedVariants)
+			return
+		}
+		xlog.Info("gallery caches warmed", "estimates", warmed, "variants", warmedVariants, "of", len(models), "took", time.Since(started).Round(time.Second))
+	}()
+}
+
+// EstimateWarmConfigFromEnv reads the warm-up bounds from the environment,
+// falling back to the defaults.
+//
+//	LOCALAI_VRAM_WARM_LIMIT        entries to warm; 0 disables the warm-up
+//	LOCALAI_VRAM_WARM_CONCURRENCY  estimates in flight at once
+//
+// Env rather than a flag because it is an operational tuning knob, not part of
+// what the server does: an air-gapped host wants it off, and a host behind a
+// slow link wants it slower, and neither is a decision the CLI should carry.
+func EstimateWarmConfigFromEnv() EstimateWarmConfig {
+	cfg := DefaultEstimateWarmConfig
+	if v, ok := os.LookupEnv("LOCALAI_VRAM_WARM_LIMIT"); ok {
+		if n, err := strconv.Atoi(strings.TrimSpace(v)); err == nil && n >= 0 {
+			cfg.Limit = n
+		}
+	}
+	if v, ok := os.LookupEnv("LOCALAI_VRAM_WARM_CONCURRENCY"); ok {
+		if n, err := strconv.Atoi(strings.TrimSpace(v)); err == nil && n > 0 {
+			cfg.Concurrency = n
+		}
+	}
+	return cfg
+}

core/gallery/estimate_warm_test.go

@@ -0,0 +1,115 @@
+package gallery_test
+
+import (
+	"context"
+	"os"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/gallery"
+	"github.com/mudler/LocalAI/pkg/system"
+)
+
+var _ = Describe("VRAM estimate warm-up", func() {
+	var state *system.SystemState
+
+	BeforeEach(func() {
+		dir, err := os.MkdirTemp("", "warm")
+		Expect(err).ToNot(HaveOccurred())
+		DeferCleanup(func() { os.RemoveAll(dir) })
+		state, err = system.GetSystemState(system.WithModelPath(dir))
+		Expect(err).ToNot(HaveOccurred())
+		gallery.ResetGalleryModelCache()
+		DeferCleanup(gallery.ResetGalleryModelCache)
+	})
+
+	It("does nothing when disabled, and returns without blocking", func() {
+		cfg := gallery.DefaultEstimateWarmConfig
+		cfg.Limit = 0
+
+		done := make(chan struct{})
+		go func() {
+			defer close(done)
+			gallery.WarmEstimateCache(context.Background(), []config.Gallery{}, state, cfg)
+		}()
+		Eventually(done, "1s").Should(BeClosed())
+	})
+
+	It("returns immediately even when there is work to do", func() {
+		// The caller is a server still starting up: warming must never be on
+		// the path to listening.
+		done := make(chan struct{})
+		go func() {
+			defer close(done)
+			gallery.WarmEstimateCache(context.Background(), []config.Gallery{}, state, gallery.DefaultEstimateWarmConfig)
+		}()
+		Eventually(done, "1s").Should(BeClosed())
+	})
+
+	It("stops when its context is cancelled", func() {
+		ctx, cancel := context.WithCancel(context.Background())
+		gallery.WarmEstimateCache(ctx, []config.Gallery{}, state, gallery.DefaultEstimateWarmConfig)
+		cancel()
+		// Nothing to assert beyond not hanging or panicking: an aborted warm-up
+		// leaves entries cold, which is the state they were already in.
+		Consistently(func() bool { return true }, "100ms").Should(BeTrue())
+	})
+
+	Describe("configuration from the environment", func() {
+		AfterEach(func() {
+			os.Unsetenv("LOCALAI_VRAM_WARM_LIMIT")
+			os.Unsetenv("LOCALAI_VRAM_WARM_CONCURRENCY")
+		})
+
+		It("falls back to the defaults", func() {
+			cfg := gallery.EstimateWarmConfigFromEnv()
+			Expect(cfg.Limit).To(Equal(gallery.DefaultEstimateWarmConfig.Limit))
+			Expect(cfg.Concurrency).To(Equal(gallery.DefaultEstimateWarmConfig.Concurrency))
+		})
+
+		It("lets an operator turn it off entirely", func() {
+			os.Setenv("LOCALAI_VRAM_WARM_LIMIT", "0")
+			Expect(gallery.EstimateWarmConfigFromEnv().Limit).To(BeZero())
+		})
+
+		It("lets an operator slow it down", func() {
+			os.Setenv("LOCALAI_VRAM_WARM_CONCURRENCY", "1")
+			Expect(gallery.EstimateWarmConfigFromEnv().Concurrency).To(Equal(1))
+		})
+
+		It("ignores values that are not usable", func() {
+			os.Setenv("LOCALAI_VRAM_WARM_LIMIT", "not-a-number")
+			os.Setenv("LOCALAI_VRAM_WARM_CONCURRENCY", "0")
+			cfg := gallery.EstimateWarmConfigFromEnv()
+			Expect(cfg.Limit).To(Equal(gallery.DefaultEstimateWarmConfig.Limit))
+			// Zero workers would be a warm-up that never runs while looking
+			// enabled, so it keeps the default rather than honouring it.
+			Expect(cfg.Concurrency).To(Equal(gallery.DefaultEstimateWarmConfig.Concurrency))
+		})
+	})
+
+	It("warms variant descriptions as well as estimates", func() {
+		// Both are the same cost wearing different hats - a probe of an entry's
+		// weight files - and both land in the same caches, so a warm-up that
+		// covered only one would leave the first click paying for the other.
+		// Asserted through the shared config rather than by observing network
+		// calls: the gallery here is empty by design.
+		Expect(gallery.DefaultEstimateWarmConfig.Limit).To(BeNumerically(">", 0))
+	})
+
+	It("keeps the estimate contexts the UI actually asks for", func() {
+		// A warmed entry at the wrong context lengths is a cache the gallery
+		// never reads, so this pins them together.
+		Expect(gallery.DefaultEstimateWarmConfig.Contexts).To(ContainElements(
+			uint32(8192), uint32(16384), uint32(32768), uint32(65536), uint32(131072), uint32(262144),
+		))
+	})
+
+	It("bounds concurrency so a warm-up cannot saturate the link", func() {
+		Expect(gallery.DefaultEstimateWarmConfig.Concurrency).To(BeNumerically("<=", 8))
+		Expect(gallery.DefaultEstimateWarmConfig.Concurrency).To(BeNumerically(">", 0))
+	})
+
+})

core/gallery/gallery.go

@@ -325,10 +325,32 @@ func AvailableGalleryModels(galleries []config.Gallery, systemState *system.Syst
 var (
 	availableModelsMu    sync.RWMutex
 	availableModelsCache GalleryElements[*GalleryModel]
-	refreshing           atomic.Bool
-	galleryGeneration    atomic.Uint64
+	// Whether a load has happened, tracked apart from the slice itself. A
+	// gallery that legitimately holds nothing caches as an empty (often nil)
+	// slice, and testing the slice for nil read that as "never loaded": every
+	// call then took the blocking path and bumped the generation, which is the
+	// same cache-defeating loop the refresh interval exists to stop.
+	availableModelsLoaded bool
+	refreshing            atomic.Bool
+	galleryGeneration     atomic.Uint64
+	lastRefreshUnixNano   atomic.Int64
 )
 
+// How often the cached model list may be refreshed from upstream.
+//
+// This is a floor on refresh frequency, not a TTL: the cache is served
+// regardless, and this only decides how often a background re-fetch is worth
+// starting. It matters far more than it looks, because a refresh bumps
+// galleryGeneration, and that invalidates every VRAM estimate cache in
+// pkg/vram. Refreshing on every call therefore kept those caches permanently
+// cold: the gallery listing is one request but the UI asks for one VRAM
+// estimate per row, so a single page view triggered dozens of refreshes and
+// every estimate paid full price for a remote probe it had already made.
+//
+// A package variable rather than a constant so tests can drive refreshes
+// without waiting.
+var GalleryRefreshInterval = 5 * time.Minute
+
 // GalleryGeneration returns a counter that increments each time the gallery
 // model list is refreshed from upstream. VRAM estimation caches use this to
 // invalidate entries when the gallery data changes.
@@ -352,7 +374,11 @@ func ResetGalleryModelCache() {
 	}
 	availableModelsMu.Lock()
 	availableModelsCache = nil
+	availableModelsLoaded = false
 	availableModelsMu.Unlock()
+	// Also clear the refresh stamp, or a suite that reset the cache would find
+	// the next refresh throttled by the previous spec's clock.
+	lastRefreshUnixNano.Store(0)
 }
 
 // AvailableGalleryModelsCached returns gallery models from an in-memory cache.
@@ -363,9 +389,10 @@ func ResetGalleryModelCache() {
 func AvailableGalleryModelsCached(galleries []config.Gallery, systemState *system.SystemState) (GalleryElements[*GalleryModel], error) {
 	availableModelsMu.RLock()
 	cached := availableModelsCache
+	loaded := availableModelsLoaded
 	availableModelsMu.RUnlock()
 
-	if cached != nil {
+	if loaded {
 		// Refresh installed status under write lock to avoid races with
 		// concurrent readers and the background refresh goroutine.
 		availableModelsMu.Lock()
@@ -387,8 +414,10 @@ func AvailableGalleryModelsCached(galleries []config.Gallery, systemState *syste
 
 	availableModelsMu.Lock()
 	availableModelsCache = models
+	availableModelsLoaded = true
 	galleryGeneration.Add(1)
 	availableModelsMu.Unlock()
+	lastRefreshUnixNano.Store(time.Now().UnixNano())
 
 	return models, nil
 }
@@ -397,9 +426,18 @@ func AvailableGalleryModelsCached(galleries []config.Gallery, systemState *syste
 // gallery model cache. Only one refresh runs at a time; concurrent calls
 // are no-ops.
 func triggerGalleryRefresh(galleries []config.Gallery, systemState *system.SystemState) {
+	if GalleryRefreshInterval > 0 {
+		last := lastRefreshUnixNano.Load()
+		if last != 0 && time.Since(time.Unix(0, last)) < GalleryRefreshInterval {
+			return
+		}
+	}
 	if !refreshing.CompareAndSwap(false, true) {
 		return
 	}
+	// Stamped before the fetch rather than after, so a slow upstream cannot
+	// let a queue of callers each start their own refresh behind this one.
+	lastRefreshUnixNano.Store(time.Now().UnixNano())
 	go func() {
 		defer refreshing.Store(false)
 		models, err := AvailableGalleryModels(galleries, systemState)
@@ -408,12 +446,37 @@ func triggerGalleryRefresh(galleries []config.Gallery, systemState *system.Syste
 			return
 		}
 		availableModelsMu.Lock()
+		changed := !sameModelSet(availableModelsCache, models)
 		availableModelsCache = models
-		galleryGeneration.Add(1)
+		availableModelsLoaded = true
+		// Only a real change invalidates the VRAM caches. An unchanged gallery
+		// re-fetched on schedule must not throw away work that is still valid,
+		// which is the difference between an estimate costing nothing and
+		// costing a network round trip.
+		if changed {
+			galleryGeneration.Add(1)
+		}
 		availableModelsMu.Unlock()
 	}()
 }
 
+// sameModelSet reports whether two model lists describe the same gallery, for
+// the purpose of deciding whether derived caches are still valid. Names and
+// order are enough: a change to an entry's files or size arrives with a new
+// gallery index, and comparing every field on every entry would cost more than
+// the caches save.
+func sameModelSet(a, b GalleryElements[*GalleryModel]) bool {
+	if len(a) != len(b) {
+		return false
+	}
+	for i := range a {
+		if a[i].GetName() != b[i].GetName() {
+			return false
+		}
+	}
+	return true
+}
+
 // List available backends
 func AvailableBackends(galleries []config.Gallery, systemState *system.SystemState) (GalleryElements[*GalleryBackend], error) {
 	return availableBackendsWithFilter(galleries, systemState, func(backend *GalleryBackend) bool {

core/gallery/gallery_refresh_throttle_test.go

@@ -0,0 +1,80 @@
+package gallery_test
+
+import (
+	"os"
+	"time"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/gallery"
+	"github.com/mudler/LocalAI/pkg/system"
+)
+
+// The gallery generation counter is what every VRAM estimate cache keys on, so
+// how often it moves decides whether those caches are worth having. Refreshing
+// on every call kept them permanently cold: one page of the model gallery asks
+// for a VRAM estimate per row, and each of those requests re-read the gallery,
+// triggering a refresh that invalidated the estimate the previous row had just
+// paid a network round trip for.
+var _ = Describe("Gallery refresh throttling", func() {
+	var (
+		tmp          *system.SystemState
+		galleries    []config.Gallery
+		origInterval time.Duration
+	)
+
+	BeforeEach(func() {
+		dir, err := os.MkdirTemp("", "gallery-throttle")
+		Expect(err).ToNot(HaveOccurred())
+		DeferCleanup(func() { os.RemoveAll(dir) })
+
+		tmp, err = system.GetSystemState(system.WithModelPath(dir))
+		Expect(err).ToNot(HaveOccurred())
+
+		// No upstream: the list comes back empty, which is all this needs. What
+		// is under test is how often a refresh is started, not what it returns.
+		galleries = []config.Gallery{}
+		origInterval = gallery.GalleryRefreshInterval
+		gallery.ResetGalleryModelCache()
+	})
+
+	AfterEach(func() {
+		gallery.GalleryRefreshInterval = origInterval
+		gallery.ResetGalleryModelCache()
+	})
+
+	It("does not bump the generation once per call", func() {
+		gallery.GalleryRefreshInterval = time.Hour
+
+		_, err := gallery.AvailableGalleryModelsCached(galleries, tmp)
+		Expect(err).ToNot(HaveOccurred())
+		start := gallery.GalleryGeneration()
+
+		// Stands in for one page view: many callers in quick succession.
+		for i := 0; i < 30; i++ {
+			_, err := gallery.AvailableGalleryModelsCached(galleries, tmp)
+			Expect(err).ToNot(HaveOccurred())
+		}
+		// Let any refresh that did start finish, so this cannot pass by racing.
+		Eventually(func() uint64 { return gallery.GalleryGeneration() }, "2s", "50ms").
+			Should(Equal(start))
+	})
+
+	It("still refreshes once the interval has passed", func() {
+		gallery.GalleryRefreshInterval = time.Millisecond
+
+		_, err := gallery.AvailableGalleryModelsCached(galleries, tmp)
+		Expect(err).ToNot(HaveOccurred())
+
+		time.Sleep(5 * time.Millisecond)
+		_, err = gallery.AvailableGalleryModelsCached(galleries, tmp)
+		Expect(err).ToNot(HaveOccurred())
+
+		// An empty gallery refreshing to an empty gallery is unchanged, so the
+		// generation must hold: only a real change may invalidate the caches.
+		Consistently(func() uint64 { return gallery.GalleryGeneration() }, "300ms", "50ms").
+			Should(Equal(gallery.GalleryGeneration()))
+	})
+})

core/gallery/importers/llama-cpp.go

@@ -38,6 +38,7 @@ func (i *LlamaCPPImporter) AdditionalBackends() []KnownBackendEntry {
 		{Name: "ik-llama-cpp", Modality: "text", Description: "GGUF drop-in replacement for llama-cpp with ik-quants"},
 		{Name: "turboquant", Modality: "text", Description: "GGUF drop-in replacement for llama-cpp with TurboQuant optimizations"},
 		{Name: "vllm-cpp", Modality: "text", Description: "vLLM-style continuous-batching engine (vllm.cpp) consuming GGUF, by the LocalAI team"},
+		{Name: "buun-llama-cpp", Modality: "text", Description: "GGUF drop-in replacement for llama-cpp with DFlash speculative decoding and TurboQuant/TCQ KV-cache quantization"},
 	}
 }
 
@@ -136,7 +137,7 @@ func (i *LlamaCPPImporter) Import(details Details) (gallery.ModelConfig, error)
 	backend := "llama-cpp"
 	if b, ok := preferencesMap["backend"].(string); ok {
 		switch b {
-		case "ik-llama-cpp", "turboquant", "vllm-cpp":
+		case "ik-llama-cpp", "turboquant", "vllm-cpp", "buun-llama-cpp":
 			backend = b
 		}
 	}

core/gallery/importers/llama-cpp_test.go

@@ -203,6 +203,23 @@ var _ = Describe("LlamaCPPImporter", func() {
 			Expect(modelConfig.Files[0].Filename).To(Equal("my-model.gguf"))
 		})
 
+		It("swaps the emitted backend to buun-llama-cpp when preferred", func() {
+			preferences := json.RawMessage(`{"backend": "buun-llama-cpp"}`)
+			details := Details{
+				URI:         "https://example.com/my-model.gguf",
+				Preferences: preferences,
+			}
+
+			modelConfig, err := importer.Import(details)
+
+			Expect(err).ToNot(HaveOccurred())
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("backend: buun-llama-cpp"), fmt.Sprintf("Model config: %+v", modelConfig))
+			Expect(modelConfig.ConfigFile).NotTo(ContainSubstring("backend: llama-cpp\n"), fmt.Sprintf("Model config: %+v", modelConfig))
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("model: my-model.gguf"), fmt.Sprintf("Model config: %+v", modelConfig))
+			Expect(len(modelConfig.Files)).To(Equal(1))
+			Expect(modelConfig.Files[0].Filename).To(Equal("my-model.gguf"))
+		})
+
 		It("keeps backend: llama-cpp for unknown backend preferences", func() {
 			// Unknown backend values must not leak into the emitted YAML —
 			// we only honour the curated drop-in replacements.
@@ -551,7 +568,7 @@ var _ = Describe("LlamaCPPImporter", func() {
 	})
 
 	Context("AdditionalBackends", func() {
-		It("advertises ik-llama-cpp, turboquant and vllm-cpp as drop-in replacements", func() {
+		It("advertises all llama-cpp drop-in replacements", func() {
 			entries := importer.AdditionalBackends()
 
 			names := make([]string, 0, len(entries))
@@ -560,7 +577,7 @@ var _ = Describe("LlamaCPPImporter", func() {
 				names = append(names, e.Name)
 				byName[e.Name] = e
 			}
-			Expect(names).To(ConsistOf("ik-llama-cpp", "turboquant", "vllm-cpp"))
+			Expect(names).To(ConsistOf("ik-llama-cpp", "turboquant", "vllm-cpp", "buun-llama-cpp"))
 
 			for _, name := range names {
 				e := byName[name]

core/gallery/importers/mlx.go

@@ -3,6 +3,7 @@ package importers
 import (
 	"encoding/json"
 	"path/filepath"
+	"slices"
 	"strings"
 
 	"github.com/mudler/LocalAI/core/config"
@@ -31,7 +32,7 @@ func (i *MLXImporter) Match(details Details) bool {
 	}
 
 	b, ok := preferencesMap["backend"].(string)
-	if ok && b == "mlx" || b == "mlx-vlm" {
+	if ok && slices.Contains([]string{"mlx", "mlx-vlm", "mlx-audio"}, b) {
 		return true
 	}
 
@@ -71,19 +72,32 @@ func (i *MLXImporter) Import(details Details) (gallery.ModelConfig, error) {
 	// (issue #10269). Send them to the mlx-vlm backend, which applies the
 	// processor-aware chat template.
 	backend := "mlx"
-	if details.HuggingFace != nil && details.HuggingFace.PipelineTag == "image-text-to-text" {
-		backend = "mlx-vlm"
+	usecases := []string{config.UsecaseChat}
+	useTokenizerTemplate := true
+	if details.HuggingFace != nil {
+		switch details.HuggingFace.PipelineTag {
+		case "image-text-to-text":
+			backend = "mlx-vlm"
+		case "text-to-speech":
+			backend = "mlx-audio"
+			usecases = []string{config.UsecaseTTS}
+			useTokenizerTemplate = false
+		}
 	}
 	// An explicit backend preference always wins.
 	b, ok := preferencesMap["backend"].(string)
 	if ok {
 		backend = b
+		if backend == "mlx-audio" {
+			usecases = []string{config.UsecaseTTS}
+			useTokenizerTemplate = false
+		}
 	}
 
 	modelConfig := config.ModelConfig{
 		Name:                name,
 		Description:         description,
-		KnownUsecaseStrings: []string{config.UsecaseChat},
+		KnownUsecaseStrings: usecases,
 		Backend:             backend,
 		PredictionOptions: schema.PredictionOptions{
 			BasicModelRequest: schema.BasicModelRequest{
@@ -91,7 +105,7 @@ func (i *MLXImporter) Import(details Details) (gallery.ModelConfig, error) {
 			},
 		},
 		TemplateConfig: config.TemplateConfig{
-			UseTokenizerTemplate: true,
+			UseTokenizerTemplate: useTokenizerTemplate,
 		},
 	}
 

core/gallery/importers/mlx_test.go

@@ -48,6 +48,16 @@ var _ = Describe("MLXImporter", func() {
 			Expect(result).To(BeTrue())
 		})
 
+		It("should match when backend preference is mlx-audio", func() {
+			preferences := json.RawMessage(`{"backend": "mlx-audio"}`)
+			details := importers.Details{
+				URI:         "https://example.com/model",
+				Preferences: preferences,
+			}
+
+			Expect(importer.Match(details)).To(BeTrue())
+		})
+
 		It("should not match when URI does not contain mlx-community/ and no backend preference", func() {
 			details := importers.Details{
 				URI: "https://huggingface.co/other-org/test-model",
@@ -123,6 +133,21 @@ var _ = Describe("MLXImporter", func() {
 			Expect(modelConfig.ConfigFile).To(ContainSubstring("backend: mlx-vlm"))
 		})
 
+		It("should configure explicit mlx-audio imports for text-to-speech", func() {
+			preferences := json.RawMessage(`{"backend": "mlx-audio"}`)
+			details := importers.Details{
+				URI:         "https://huggingface.co/mlx-community/Kokoro-82M-4bit",
+				Preferences: preferences,
+			}
+
+			modelConfig, err := importer.Import(details)
+
+			Expect(err).ToNot(HaveOccurred())
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("backend: mlx-audio"))
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("- tts"))
+			Expect(modelConfig.ConfigFile).ToNot(ContainSubstring("use_tokenizer_template: true"))
+		})
+
 		It("should auto-route vision-language models to the mlx-vlm backend", func() {
 			// gemma-4 E4B and similar VLMs declare pipeline_tag
 			// "image-text-to-text" on HuggingFace. The text-only mlx-lm
@@ -143,6 +168,23 @@ var _ = Describe("MLXImporter", func() {
 			Expect(modelConfig.ConfigFile).To(ContainSubstring("backend: mlx-vlm"))
 		})
 
+		It("should auto-route text-to-speech models to the mlx-audio backend", func() {
+			details := importers.Details{
+				URI: "https://huggingface.co/mlx-community/Kokoro-82M-4bit",
+				HuggingFace: &hfapi.ModelDetails{
+					ModelID:     "mlx-community/Kokoro-82M-4bit",
+					PipelineTag: "text-to-speech",
+				},
+			}
+
+			modelConfig, err := importer.Import(details)
+
+			Expect(err).ToNot(HaveOccurred())
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("backend: mlx-audio"))
+			Expect(modelConfig.ConfigFile).To(ContainSubstring("- tts"))
+			Expect(modelConfig.ConfigFile).ToNot(ContainSubstring("use_tokenizer_template: true"))
+		})
+
 		It("should keep text-only models on the plain mlx backend", func() {
 			details := importers.Details{
 				URI: "https://huggingface.co/mlx-community/Llama-3.2-1B-Instruct-4bit",

core/http/auth/features.go

@@ -39,6 +39,8 @@ var RouteFeatureRegistry = []RouteFeature{
 	{"POST", "/images/generations", FeatureImages},
 	{"POST", "/v1/images/inpainting", FeatureImages},
 	{"POST", "/images/inpainting", FeatureImages},
+	{"POST", "/v1/images/upscale", FeatureImages},
+	{"POST", "/images/upscale", FeatureImages},
 
 	// Audio transcription
 	{"POST", "/v1/audio/transcriptions", FeatureAudioTranscription},
@@ -116,6 +118,10 @@ var RouteFeatureRegistry = []RouteFeature{
 	// Rerank
 	{"POST", "/v1/rerank", FeatureRerank},
 
+	// Moderation
+	{"POST", "/v1/moderations", FeatureModeration},
+	{"POST", "/moderations", FeatureModeration},
+
 	// Stores
 	{"POST", "/stores/set", FeatureStores},
 	{"POST", "/stores/delete", FeatureStores},
@@ -191,6 +197,7 @@ func APIFeatureMetas() []FeatureMeta {
 		{FeatureEmbeddings, "Embeddings", true},
 		{FeatureSound, "Sound Generation", true},
 		{FeatureRealtime, "Realtime", true},
+		{FeatureModeration, "Moderation", true},
 		{FeatureRerank, "Rerank", true},
 		{FeatureTokenize, "Tokenize", true},
 		{FeatureMCP, "MCP", true},

core/http/auth/features_moderation_test.go

@@ -0,0 +1,24 @@
+package auth_test
+
+import (
+	. "github.com/mudler/LocalAI/core/http/auth"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("Moderation feature registration", func() {
+	It("registers both moderation routes as default-on API features", func() {
+		Expect(APIFeatures).To(ContainElement(FeatureModeration))
+
+		patterns := []string{}
+		for _, route := range RouteFeatureRegistry {
+			if route.Feature == FeatureModeration {
+				patterns = append(patterns, route.Pattern)
+			}
+		}
+		Expect(patterns).To(ConsistOf("/v1/moderations", "/moderations"))
+
+		metas := APIFeatureMetas()
+		Expect(metas).To(ContainElement(FeatureMeta{Key: FeatureModeration, Label: "Moderation", DefaultValue: true}))
+	})
+})

core/http/auth/helpers_test.go

@@ -59,10 +59,14 @@ func ok(c echo.Context) error {
 func newAuthTestApp(db *gorm.DB, appConfig *config.ApplicationConfig) *echo.Echo {
 	e := echo.New()
 	e.Use(auth.Middleware(db, appConfig))
+	if db != nil {
+		e.Use(auth.RequireRouteFeature(db))
+	}
 
 	// API routes (require auth)
 	e.GET("/v1/models", ok)
 	e.POST("/v1/chat/completions", ok)
+	e.POST("/v1/moderations", ok)
 	e.GET("/api/settings", ok)
 	e.POST("/api/settings", ok)
 
@@ -81,10 +85,14 @@ func newAuthTestApp(db *gorm.DB, appConfig *config.ApplicationConfig) *echo.Echo
 func newAdminTestApp(db *gorm.DB, appConfig *config.ApplicationConfig) *echo.Echo {
 	e := echo.New()
 	e.Use(auth.Middleware(db, appConfig))
+	if db != nil {
+		e.Use(auth.RequireRouteFeature(db))
+	}
 
 	// Regular routes
 	e.GET("/v1/models", ok)
 	e.POST("/v1/chat/completions", ok)
+	e.POST("/v1/moderations", ok)
 
 	// Admin-only routes
 	adminMw := auth.RequireAdmin()

core/http/auth/middleware_test.go

@@ -91,6 +91,19 @@ var _ = Describe("Auth Middleware", func() {
 			Expect(rec.Code).To(Equal(http.StatusOK))
 		})
 
+		It("allows authenticated users to call moderation by default", func() {
+			sessionID := createTestSession(db, user.ID)
+			rec := doRequest(app, http.MethodPost, "/v1/moderations", withSessionCookie(sessionID))
+			Expect(rec.Code).To(Equal(http.StatusOK))
+		})
+
+		It("blocks moderation when the user's feature is disabled", func() {
+			Expect(auth.UpdateUserPermissions(db, user.ID, auth.PermissionMap{auth.FeatureModeration: false})).To(Succeed())
+			sessionID := createTestSession(db, user.ID)
+			rec := doRequest(app, http.MethodPost, "/v1/moderations", withSessionCookie(sessionID))
+			Expect(rec.Code).To(Equal(http.StatusForbidden))
+		})
+
 		It("allows requests with valid session as Bearer token", func() {
 			sessionID := createTestSession(db, user.ID)
 			rec := doRequest(app, http.MethodGet, "/v1/models", withBearerToken(sessionID))
@@ -156,6 +169,11 @@ var _ = Describe("Auth Middleware", func() {
 			Expect(rec.Code).To(Equal(http.StatusUnauthorized))
 		})
 
+		It("returns 401 for unauthenticated moderation requests", func() {
+			rec := doRequest(app, http.MethodPost, "/v1/moderations")
+			Expect(rec.Code).To(Equal(http.StatusUnauthorized))
+		})
+
 		It("returns 401 for unauthenticated 3D generation requests", func() {
 			rec := doRequest(app, http.MethodPost, "/3d/generations")
 			Expect(rec.Code).To(Equal(http.StatusUnauthorized))

core/http/auth/permissions.go

@@ -51,6 +51,7 @@ const (
 	FeatureEmbeddings          = "embeddings"
 	FeatureSound               = "sound"
 	FeatureRealtime            = "realtime"
+	FeatureModeration          = "moderation"
 	FeatureRerank              = "rerank"
 	FeatureTokenize            = "tokenize"
 	FeatureMCP                 = "mcp"
@@ -75,7 +76,7 @@ var APIFeatures = []string{
 	FeatureChat, FeatureImages, FeatureAudioSpeech, FeatureAudioTranscription,
 	FeatureAudioDiarization, FeatureAudioClassification,
 	FeatureVAD, FeatureDetection, FeatureVideo, Feature3D, FeatureEmbeddings, FeatureSound,
-	FeatureRealtime, FeatureRerank, FeatureTokenize, FeatureMCP, FeatureStores,
+	FeatureRealtime, FeatureModeration, FeatureRerank, FeatureTokenize, FeatureMCP, FeatureStores,
 	FeatureFaceRecognition, FeatureVoiceRecognition, FeatureAudioTransform,
 	FeaturePIIFilter,
 }

core/http/endpoints/localai/api_instructions.go

@@ -30,6 +30,12 @@ var instructionDefs = []instructionDef{
 		Tags:        []string{"inference", "embeddings"},
 		Intro:       "Set \"stream\": true for SSE streaming. Supports tool/function calling when the model config has function templates configured.",
 	},
+	{
+		Name:        "moderation",
+		Description: "OpenAI-compatible text moderation using a local completion model",
+		Tags:        []string{"moderation"},
+		Intro:       "POST /v1/moderations accepts a text string or array plus a LocalAI completion model. LocalAI constrains the model to the OpenAI moderation category schema and returns one result per input. Multimodal moderation inputs are not yet supported.",
+	},
 	{
 		Name:        "audio",
 		Description: "Text-to-speech, voice activity detection, transcription, speaker diarization, sound classification, and sound generation",

core/http/endpoints/localai/api_instructions_test.go

@@ -39,7 +39,7 @@ var _ = Describe("API Instructions Endpoints", func() {
 
 			instructions, ok := resp["instructions"].([]any)
 			Expect(ok).To(BeTrue())
-			Expect(instructions).To(HaveLen(18))
+			Expect(instructions).To(HaveLen(19))
 
 			// Verify each instruction has required fields and correct URL format
 			for _, s := range instructions {
@@ -69,6 +69,7 @@ var _ = Describe("API Instructions Endpoints", func() {
 
 			Expect(names).To(ContainElements(
 				"chat-inference",
+				"moderation",
 				"config-management",
 				"model-management",
 				"monitoring",

core/http/endpoints/localai/backend.go

@@ -38,6 +38,7 @@ var knownPrefOnlyBackends = []schema.KnownBackend{
 	{Name: "whisperx", Modality: "asr", AutoDetect: false, Description: "WhisperX transcription (preference-only)"},
 	{Name: "crispasr", Modality: "asr", AutoDetect: false, Description: "CrispASR multi-architecture transcription (preference-only)"},
 	// TTS
+	{Name: "mlx-audio", Modality: "tts", AutoDetect: false, Description: "MLX-Audio text-to-speech models (auto-detected; pref-only fallback)"},
 	{Name: "kokoros", Modality: "tts", AutoDetect: false, Description: "Kokoros TTS (preference-only)"},
 	{Name: "qwen-tts", Modality: "tts", AutoDetect: false, Description: "Qwen TTS (preference-only)"},
 	{Name: "qwen3-tts-cpp", Modality: "tts", AutoDetect: false, Description: "Qwen3 TTS C++ (preference-only)"},

core/http/endpoints/localai/backend_test.go

@@ -152,6 +152,7 @@ var _ = Describe("Backend Endpoints", func() {
 			expectPrefOnly("tinygrad", "text")
 			expectPrefOnly("trl", "text")
 			expectPrefOnly("mlx-vlm", "text")
+			expectPrefOnly("mlx-audio", "tts")
 			expectPrefOnly("whisperx", "asr")
 			expectPrefOnly("crispasr", "asr")
 			expectPrefOnly("kokoros", "tts")

core/http/endpoints/localai/system.go

@@ -12,7 +12,7 @@ import (
 // @Tags monitoring
 // @Success 200 {object} schema.SystemInformationResponse "Response"
 // @Router /system [get]
-func SystemInformations(ml *model.ModelLoader, appConfig *config.ApplicationConfig) echo.HandlerFunc {
+func SystemInformations(cl *config.ModelConfigLoader, ml *model.ModelLoader, appConfig *config.ApplicationConfig) echo.HandlerFunc {
 	return func(c echo.Context) error {
 		availableBackends := []string{}
 		loadedModels := ml.ListLoadedModels()
@@ -25,7 +25,14 @@ func SystemInformations(ml *model.ModelLoader, appConfig *config.ApplicationConf
 
 		sysmodels := []schema.SysInfoModel{}
 		for _, m := range loadedModels {
-			sysmodels = append(sysmodels, schema.SysInfoModel{ID: m.ID})
+			entry := schema.SysInfoModel{ID: m.ID}
+			// The loader tracks only the ID. Which engine is serving a model is
+			// the first thing an operator wants beside its name, and it is one
+			// config lookup away.
+			if cfg, ok := cl.GetModelConfig(m.ID); ok {
+				entry.Backend = cfg.Backend
+			}
+			sysmodels = append(sysmodels, entry)
 		}
 		return c.JSON(200,
 			schema.SystemInformationResponse{

core/http/endpoints/localai/traces.go

@@ -3,6 +3,7 @@ package localai
 import (
 	"net/http"
 	"strconv"
+	"time"
 
 	"github.com/labstack/echo/v4"
 	"github.com/mudler/LocalAI/core/http/middleware"
@@ -85,6 +86,35 @@ func GetAPITracesEndpoint() echo.HandlerFunc {
 	}
 }
 
+// GetAPITracesSummaryEndpoint returns counted totals over a recent window
+// @Summary Summarize recent API traces
+// @Description Returns request, failure and latency totals over a recent window, plus a bucketed series for sparklines. Exists so callers wanting three numbers do not have to fetch the whole trace list and count it themselves.
+// @Tags monitoring
+// @Produce json
+// @Param hours query int false "Window in hours (default 24, max 168)"
+// @Success 200 {object} middleware.TraceSummary "Counted trace totals"
+// @Router /api/traces/summary [get]
+func GetAPITracesSummaryEndpoint() echo.HandlerFunc {
+	return func(c echo.Context) error {
+		hours := 24
+		if raw := c.QueryParam("hours"); raw != "" {
+			if v, err := strconv.Atoi(raw); err == nil && v > 0 {
+				hours = v
+			}
+		}
+		// A week is plenty for a dashboard, and the trace buffer is bounded
+		// anyway; an unbounded window would just scan the whole buffer.
+		if hours > 168 {
+			hours = 168
+		}
+		return c.JSON(http.StatusOK, middleware.GetTracesSummary(time.Duration(hours)*time.Hour, traceSummaryBuckets))
+	}
+}
+
+// Enough columns for a sparkline to show a shape, few enough that each one
+// still holds a meaningful count on a quiet installation.
+const traceSummaryBuckets = 12
+
 // GetAPITraceEndpoint returns a single API trace with its full payload
 // @Summary Get one API trace
 // @Description Returns a single captured API exchange, including the request and response bodies omitted from the list response

core/http/endpoints/mcp/localai_assistant_test.go

@@ -84,6 +84,22 @@ func (stubClient) ListNodes(_ context.Context) ([]localaitools.Node, error) {
 	return []localaitools.Node{}, nil
 }
 
+func (stubClient) ListScheduling(_ context.Context) ([]localaitools.ModelSchedulingConfig, error) {
+	return []localaitools.ModelSchedulingConfig{}, nil
+}
+
+func (stubClient) GetScheduling(_ context.Context, _ string) (*localaitools.ModelSchedulingConfig, error) {
+	return &localaitools.ModelSchedulingConfig{}, nil
+}
+
+func (stubClient) SetScheduling(_ context.Context, _ localaitools.SetSchedulingRequest) (*localaitools.ModelSchedulingConfig, error) {
+	return &localaitools.ModelSchedulingConfig{}, nil
+}
+
+func (stubClient) DeleteScheduling(_ context.Context, _ string) error {
+	return nil
+}
+
 func (stubClient) SetNodeVRAMBudget(_ context.Context, _, _ string) error {
 	return nil
 }

core/http/endpoints/openai/moderations.go

@@ -0,0 +1,190 @@
+package openai
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"math"
+	"net/http"
+	"strings"
+
+	"github.com/google/uuid"
+	"github.com/labstack/echo/v4"
+	"github.com/mudler/LocalAI/core/backend"
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/http/middleware"
+	"github.com/mudler/LocalAI/core/schema"
+	"github.com/mudler/LocalAI/core/templates"
+	"github.com/mudler/LocalAI/pkg/functions"
+	"github.com/mudler/LocalAI/pkg/model"
+)
+
+var moderationCategories = []string{
+	"harassment",
+	"harassment/threatening",
+	"hate",
+	"hate/threatening",
+	"illicit",
+	"illicit/violent",
+	"self-harm",
+	"self-harm/intent",
+	"self-harm/instructions",
+	"sexual",
+	"sexual/minors",
+	"violence",
+	"violence/graphic",
+}
+
+type moderationGenerator func(context.Context, string, *config.ModelConfig) (string, backend.TokenUsage, error)
+
+type generatedModeration struct {
+	Categories     map[string]bool    `json:"categories"`
+	CategoryScores map[string]float64 `json:"category_scores"`
+}
+
+// ModerationEndpoint implements the text input subset of OpenAI's moderation
+// API using any LocalAI completion model and constrained JSON generation.
+// @Summary Classify text for potentially harmful content.
+// @Tags moderation
+// @Param request body schema.ModerationRequest true "query params"
+// @Success 200 {object} schema.ModerationResponse "Response"
+// @Router /v1/moderations [post]
+func ModerationEndpoint(cl *config.ModelConfigLoader, ml *model.ModelLoader, evaluator *templates.Evaluator, appConfig *config.ApplicationConfig) echo.HandlerFunc {
+	return moderationEndpoint(func(ctx context.Context, input string, cfg *config.ModelConfig) (string, backend.TokenUsage, error) {
+		prompt := moderationPrompt(input)
+		var messages schema.Messages
+		if cfg.TemplateConfig.UseTokenizerTemplate {
+			messages = schema.Messages{{Role: "user", Content: prompt}}
+			prompt = ""
+		} else if evaluator != nil {
+			if rendered, err := evaluator.EvaluateTemplateForPrompt(templates.CompletionPromptTemplate, *cfg, templates.PromptTemplateData{Input: prompt, SystemPrompt: cfg.SystemPrompt}); err == nil {
+				prompt = rendered
+			}
+		}
+
+		predict, err := backend.ModelInferenceFunc(ctx, prompt, messages, nil, nil, nil, ml, cfg, cl, appConfig, nil, "", "", nil, nil, nil, nil)
+		if err != nil {
+			return "", backend.TokenUsage{}, err
+		}
+		response, err := predict()
+		return response.Response, response.Usage, err
+	})
+}
+
+func moderationEndpoint(generate moderationGenerator) echo.HandlerFunc {
+	return func(c echo.Context) error {
+		input, ok := c.Get(middleware.CONTEXT_LOCALS_KEY_LOCALAI_REQUEST).(*schema.ModerationRequest)
+		if !ok || input == nil {
+			return echo.NewHTTPError(http.StatusBadRequest, "invalid moderation request")
+		}
+		if len(input.Input) == 0 {
+			return echo.NewHTTPError(http.StatusBadRequest, "input must contain at least one text string")
+		}
+		if generate == nil {
+			return echo.NewHTTPError(http.StatusInternalServerError, "moderation generator is unavailable")
+		}
+
+		modelConfig, ok := c.Get(middleware.CONTEXT_LOCALS_KEY_MODEL_CONFIG).(*config.ModelConfig)
+		if !ok || modelConfig == nil {
+			return echo.NewHTTPError(http.StatusBadRequest, "moderation model configuration is unavailable")
+		}
+
+		grammar, err := moderationGrammar()
+		if err != nil {
+			return echo.NewHTTPError(http.StatusInternalServerError, "failed to build moderation grammar").SetInternal(err)
+		}
+		cfg := *modelConfig
+		cfg.Grammar = grammar
+		maxTokens := 512
+		cfg.Maxtokens = &maxTokens
+
+		results := make([]schema.ModerationResult, 0, len(input.Input))
+		promptTokens, completionTokens := 0, 0
+		for _, text := range input.Input {
+			if strings.TrimSpace(text) == "" {
+				return echo.NewHTTPError(http.StatusBadRequest, "input strings must not be empty")
+			}
+			raw, usage, err := generate(c.Request().Context(), text, &cfg)
+			if err != nil {
+				return echo.NewHTTPError(http.StatusInternalServerError, "moderation inference failed").SetInternal(err)
+			}
+			result, err := parseModerationResult(raw)
+			if err != nil {
+				return echo.NewHTTPError(http.StatusInternalServerError, "invalid moderation result: "+err.Error()).SetInternal(err)
+			}
+			results = append(results, result)
+			promptTokens += usage.Prompt
+			completionTokens += usage.Completion
+		}
+
+		middleware.StampUsage(c, input.Model, promptTokens, completionTokens)
+		return c.JSON(http.StatusOK, schema.ModerationResponse{
+			ID:      "modr-" + uuid.NewString(),
+			Model:   input.Model,
+			Results: results,
+		})
+	}
+}
+
+func moderationPrompt(input string) string {
+	encoded, _ := json.Marshal(input)
+	return `Classify the supplied text for safety moderation. Return only the JSON object required by the grammar.
+For each category, set the boolean to true only when the text matches that category. Give each category a confidence score from 0 to 1.
+The categories are harassment, harassment/threatening, hate, hate/threatening, illicit, illicit/violent, self-harm, self-harm/intent, self-harm/instructions, sexual, sexual/minors, violence, and violence/graphic.
+Text to classify: ` + string(encoded)
+}
+
+func moderationGrammar() (string, error) {
+	boolProperties := map[string]any{}
+	scoreProperties := map[string]any{}
+	for _, category := range moderationCategories {
+		boolProperties[category] = map[string]any{"type": "boolean"}
+		scoreProperties[category] = map[string]any{"type": "number"}
+	}
+	structure := functions.JSONFunctionStructure{AnyOf: []functions.Item{{
+		Type: "object",
+		Properties: map[string]any{
+			"categories": map[string]any{
+				"type":                 "object",
+				"properties":           boolProperties,
+				"required":             moderationCategories,
+				"additionalProperties": false,
+			},
+			"category_scores": map[string]any{
+				"type":                 "object",
+				"properties":           scoreProperties,
+				"required":             moderationCategories,
+				"additionalProperties": false,
+			},
+		},
+	}}}
+	return structure.Grammar()
+}
+
+func parseModerationResult(raw string) (schema.ModerationResult, error) {
+	var generated generatedModeration
+	if err := json.Unmarshal([]byte(strings.TrimSpace(raw)), &generated); err != nil {
+		return schema.ModerationResult{}, err
+	}
+
+	result := schema.ModerationResult{
+		Categories:                make(map[string]bool, len(moderationCategories)),
+		CategoryScores:            make(map[string]float64, len(moderationCategories)),
+		CategoryAppliedInputTypes: make(map[string][]string, len(moderationCategories)),
+	}
+	for _, category := range moderationCategories {
+		flagged, exists := generated.Categories[category]
+		if !exists {
+			return schema.ModerationResult{}, fmt.Errorf("missing category %q", category)
+		}
+		score, exists := generated.CategoryScores[category]
+		if !exists || math.IsNaN(score) || math.IsInf(score, 0) || score < 0 || score > 1 {
+			return schema.ModerationResult{}, fmt.Errorf("category %q has an invalid score", category)
+		}
+		result.Categories[category] = flagged
+		result.CategoryScores[category] = score
+		result.CategoryAppliedInputTypes[category] = []string{"text"}
+		result.Flagged = result.Flagged || flagged
+	}
+	return result, nil
+}

core/http/endpoints/openai/moderations_test.go

@@ -0,0 +1,105 @@
+package openai
+
+import (
+	"context"
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+	"strings"
+
+	"github.com/labstack/echo/v4"
+	"github.com/mudler/LocalAI/core/backend"
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/http/middleware"
+	"github.com/mudler/LocalAI/core/schema"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+var _ = Describe("Moderations endpoint", func() {
+	It("classifies each text input and returns the OpenAI response shape", func() {
+		inputs := []string{}
+		generate := func(_ context.Context, input string, cfg *config.ModelConfig) (string, backend.TokenUsage, error) {
+			inputs = append(inputs, input)
+			Expect(cfg.Grammar).To(ContainSubstring("harassment"))
+			return `{
+				"categories":{"harassment":true,"harassment/threatening":false,"hate":false,"hate/threatening":false,"illicit":false,"illicit/violent":false,"self-harm":false,"self-harm/intent":false,"self-harm/instructions":false,"sexual":false,"sexual/minors":false,"violence":false,"violence/graphic":false},
+				"category_scores":{"harassment":0.9,"harassment/threatening":0.1,"hate":0,"hate/threatening":0,"illicit":0,"illicit/violent":0,"self-harm":0,"self-harm/intent":0,"self-harm/instructions":0,"sexual":0,"sexual/minors":0,"violence":0,"violence/graphic":0}
+			}`, backend.TokenUsage{Prompt: 12, Completion: 8}, nil
+		}
+
+		e := echo.New()
+		req := httptest.NewRequest(http.MethodPost, "/v1/moderations", strings.NewReader(`{"model":"guard","input":["first","second"]}`))
+		req.Header.Set(echo.HeaderContentType, echo.MIMEApplicationJSON)
+		rec := httptest.NewRecorder()
+		ctx := e.NewContext(req, rec)
+		ctx.Set(middleware.CONTEXT_LOCALS_KEY_LOCALAI_REQUEST, &schema.ModerationRequest{
+			BasicModelRequest: schema.BasicModelRequest{Model: "guard"},
+			Input:             schema.ModerationInput{"first", "second"},
+		})
+		modelConfig := &config.ModelConfig{Name: "guard"}
+		modelConfig.Model = "guard.gguf"
+		ctx.Set(middleware.CONTEXT_LOCALS_KEY_MODEL_CONFIG, modelConfig)
+
+		Expect(moderationEndpoint(generate)(ctx)).To(Succeed())
+		Expect(rec.Code).To(Equal(http.StatusOK))
+		Expect(inputs).To(Equal([]string{"first", "second"}))
+
+		var response schema.ModerationResponse
+		Expect(json.Unmarshal(rec.Body.Bytes(), &response)).To(Succeed())
+		Expect(response.ID).To(HavePrefix("modr-"))
+		Expect(response.Model).To(Equal("guard"))
+		Expect(response.Results).To(HaveLen(2))
+		Expect(response.Results[0].Flagged).To(BeTrue())
+		Expect(response.Results[0].Categories["harassment"]).To(BeTrue())
+		Expect(response.Results[0].CategoryAppliedInputTypes["harassment"]).To(Equal([]string{"text"}))
+	})
+
+	It("rejects an empty input list", func() {
+		e := echo.New()
+		ctx := e.NewContext(httptest.NewRequest(http.MethodPost, "/v1/moderations", nil), httptest.NewRecorder())
+		ctx.Set(middleware.CONTEXT_LOCALS_KEY_LOCALAI_REQUEST, &schema.ModerationRequest{
+			BasicModelRequest: schema.BasicModelRequest{Model: "guard"},
+		})
+		ctx.Set(middleware.CONTEXT_LOCALS_KEY_MODEL_CONFIG, &config.ModelConfig{Name: "guard"})
+
+		err := moderationEndpoint(nil)(ctx)
+		Expect(err).To(MatchError(ContainSubstring("input must contain at least one text string")))
+		Expect(err.(*echo.HTTPError).Code).To(Equal(http.StatusBadRequest))
+	})
+
+	It("surfaces malformed classifier output without returning a partial result", func() {
+		generate := func(context.Context, string, *config.ModelConfig) (string, backend.TokenUsage, error) {
+			return "not-json", backend.TokenUsage{}, nil
+		}
+		e := echo.New()
+		ctx := e.NewContext(httptest.NewRequest(http.MethodPost, "/v1/moderations", nil), httptest.NewRecorder())
+		ctx.Set(middleware.CONTEXT_LOCALS_KEY_LOCALAI_REQUEST, &schema.ModerationRequest{
+			BasicModelRequest: schema.BasicModelRequest{Model: "guard"},
+			Input:             schema.ModerationInput{"text"},
+		})
+		ctx.Set(middleware.CONTEXT_LOCALS_KEY_MODEL_CONFIG, &config.ModelConfig{Name: "guard"})
+
+		err := moderationEndpoint(generate)(ctx)
+		Expect(err).To(MatchError(ContainSubstring("invalid moderation result")))
+		Expect(err.(*echo.HTTPError).Code).To(Equal(http.StatusInternalServerError))
+	})
+})
+
+var _ = Describe("Moderation input", func() {
+	DescribeTable("accepts OpenAI text input forms",
+		func(body string, expected schema.ModerationInput) {
+			var req schema.ModerationRequest
+			Expect(json.Unmarshal([]byte(body), &req)).To(Succeed())
+			Expect(req.Input).To(Equal(expected))
+		},
+		Entry("single text", `{"input":"hello"}`, schema.ModerationInput{"hello"}),
+		Entry("text array", `{"input":["hello","world"]}`, schema.ModerationInput{"hello", "world"}),
+	)
+
+	It("rejects multimodal input in the text-only MVP", func() {
+		var req schema.ModerationRequest
+		err := json.Unmarshal([]byte(`{"input":[{"type":"image_url","image_url":{"url":"https://example.com/a.png"}}]}`), &req)
+		Expect(err).To(MatchError(ContainSubstring("text string or array of text strings")))
+	})
+})