Prefer an explicitly configured Triton assembler, otherwise use the executable ptxas from CUDA_HOME so torch.compile can target GPU architectures newer than Triton bundled tooling.
Assisted-by: Codex:gpt-5
Replace copied HauhauCS base-model text with metadata for the actual Genesis Hermes V6 artifact and link its upstream base model.
Assisted-by: Codex:gpt-5 [Hugging Face]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
No -gpu-rocm-hipblas-llama-cpp image has been published since 2026-08-01.
Every build since has been killed by GitHub at exactly its 6h job limit:
job 91830652349 cancelled 6h00m (2026-08-04)
job 91763226161 cancelled 6h00m (2026-08-03)
job 91466626154 cancelled 6h00m (2026-08-02 full matrix)
The registry shows the damage: master-gpu-rocm-hipblas-llama-cpp last
built 2026-08-01 05:53, latest-gpu-rocm-hipblas-llama-cpp 2026-07-15,
against master-cpu-llama-cpp which is current.
Same cause as #11321, different mechanism. Since #11255 every x86 GPU
image also builds ggml's CPU_ALL_VARIANTS matrix. SYCL died because icpx
stalls on one translation unit; ROCm dies on volume. hipcc compiles the
HIP kernels once per entry in AMDGPU_TARGETS, and that list is eleven
architectures (gfx908, gfx90a, gfx942, gfx950, gfx1030, gfx1100, gfx1101,
gfx1102, gfx1151, gfx1200, gfx1201). The CPU matrix lands on top of that.
The numbers are unambiguous. The same job took 2h27m in the 2026-07-26
full matrix, before #11255. #11255 merged 2026-08-01 07:26, an hour and a
half after the last image was published, and it has been 6h00m ever since.
The tail of the last run shows it 61% through ggml-hip at the 83 minute
mark, still building HIP template instances.
Route hipblas to the portable fallback, exactly as #11321 did for SYCL and
for the same practical reason: it is what these images shipped before
#11255, and run.sh already prefers *-cpu-all when present and falls back
otherwise. Expected to restore the 2h27m build with room to spare.
Not fixed here: the CPU variant matrix is genuinely wanted on ROCm for
partial offload. Getting it needs the build to fit in 6h, which means
trimming AMDGPU_TARGETS or splitting the job per architecture. Both are
larger changes than unbreaking the image, and neither should ride along
with a build that is currently not shipping at all.
Verified: make test-build-scripts passes, including the extended
llama-cpp-build-target_test.sh. bonsai is unaffected (own compile script,
ROCm builds in 1h52m) and turboquant has no hipblas variant.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): enable and vendor the MLX GEMM provider on darwin/metal
The darwin vllm-cpp image built the Metal backend with vllm.cpp's native MSL
GEMM only. vllm.cpp also ships an optional MLX provider for the dense GEMM,
kept OFF upstream because it costs a ~19 MB libmlx.dylib plus a ~105 MB
mlx.metallib, on the stated position that it must earn that cost by
measurement.
Measured on an Apple M4 (16 GiB, macOS 26.5.2) it does. One binary, arms
toggled with VT_OP_PROVIDER_DISABLE=mlx so there is no build-difference
confound, Qwen3-1.7B-bf16 p=512 g=128, 2 reps, arm order alternated per rep:
B=1 5.79 vs 3.08 agg tok/s (1.88x) TTFT 3.32 s vs 7.68 s
B=8 25.70 vs 13.69 (1.88x) TTFT 13.95 s vs 34.38 s
B=16 38.65 vs 17.69 (2.19x) TTFT 18.33 s vs 54.48 s
Peak RSS is unchanged (6.65 to 7.50 GB in both arms) and the output is
bit-identical: vllm.cpp's three-way parity test measures mlx-vs-msl NMSE of 0
on all six shapes, and mlx-vs-cpu equal to msl-vs-cpu, against a 5e-4 bar. MLX
serves the dense GEMM alone; paged attention stays vllm.cpp's own kernel
because MLX has no paged-KV primitive. Full disposition, including the
INDICATIVE status and the isolation actually achieved, is in vllm.cpp
docs/BENCHMARKS.md "MLX GEMM provider A/B on Apple M4".
Build: MLX comes from the pinned prebuilt pip wheel (MLX_VERSION, default
0.29.3) into a venv under the backend dir. Building MLX from source needs
`xcrun metal`, i.e. a full Xcode the macOS runners do not have, while the wheel
ships include/, lib/libmlx.dylib and the compiled metallib ready to link. The
install is a stamp FILE rather than a phony target, because a phony
prerequisite is always newer than libvllm and would re-link it every
invocation. VLLM_CPP_MLX=off restores the previous Metal build.
Packaging vendors libmlx.dylib, mlx.metallib and MLX's MIT license into
package/lib/. Three things this had to get right, each verified on the M4
before it was written rather than after:
1. libvllm.dylib links @rpath/libmlx.dylib and its build-time LC_RPATH points
inside the build venv, a path no user has. Every build rpath is deleted
and replaced with @loader_path/lib.
2. MLX loads its metallib from beside its OWN dylib, so both files must land
in the same directory or every Metal op fails with "Failed to load the
default metallib".
3. install_name_tool invalidates the code signature and macOS refuses to load
an arm64 image with a stale one, so the patched library is re-signed
ad-hoc.
Verified end to end on the M4 by building through this Makefile and running the
packaged artifact: `DYLD_PRINT_LIBRARIES` resolves libmlx from package/lib/,
`codesign -v` passes, no build-venv path survives in the load commands, and a
real generation runs with the provider selected (op=65 selected=mlx) and zero
metallib failures. A missing rpath now fails the build instead of the user's
first inference.
Cost: the darwin vllm-cpp image grows by about 124 MB.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
* fix(vllm-cpp): default the MLX GEMM provider OFF on darwin
This branch opened with VLLM_CPP_MLX=on, justified by an A/B that measured the
MLX provider at 1.88x to 2.19x against the native MSL GEMM. That measurement was
correct when taken and is now stale: vllm.cpp's own Metal kernels have improved
several-fold since, through mma prefill attention, a vectorised decode V
accumulation, vectorised attention staging, a fused qk-norm-RoPE preamble and a
simdgroup-per-row softmax. The native path MLX was compared against no longer
exists.
Re-measured on the same Apple M4, in the same binary, with the arms toggled by
VT_OP_PROVIDER_DISABLE=mlx, on Qwen3-1.7B-bf16 warm at p=512 g=128:
MLX provider ON prefill TTFT 1370 ms warm throughput 11.98 tok/s
MLX provider OFF prefill TTFT 1400 ms warm throughput 22.06 tok/s
Shipping the previous default would have halved Apple Silicon throughput.
MLX's steel GEMM is still about 20% faster than ours in isolation, but the
provider pays a per-op mx::eval synchronisation plus an output memcpy, because it
cannot write into our buffer. Across prefill's roughly 112 GEMMs that overhead
leaves a 2% gain; on decode, where the same synchronisation is paid once per
matmul per token, it costs 46%. The option is kept for prefill-dominated
workloads, where the margin is small but real.
The README section is rewritten rather than patched: it previously presented the
stale table as the reason for the default, so leaving it in place would have made
the new default look arbitrary.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): bump vllm.cpp and default MLX ON, gated to prefill
Bumps VLLM_CPP_VERSION from 9e1c9025 to eec09bed and turns VLLM_CPP_MLX back on.
These two must move together, which is why they are one commit.
Upstream now shape-gates the MLX provider to prefill: it declines m < 2, which is
exactly the decode GEMV. MLX's steel GEMM wins prefill, 524.5 ms of TTFT against
602 for the native path, but loses decode badly because the provider pays an
mx::eval synchronisation and an output memcpy on every call while decode makes
about 112 calls per token. Ungated it does both; gated it does only the good half.
Measured on an Apple M4 with Qwen3-1.7B-bf16 warm at p=512 g=128:
MLX gated to prefill (pin >= 89c46aeb) TTFT 524.5 ms 24.40 tok/s, 99.1% of MLX-LM
MLX ungated (older pins) TTFT 537 ms 12.7 tok/s
MLX off TTFT 602 ms 23.9 tok/s
This branch briefly defaulted the provider off, which was the correct call for an
ungated provider at the old pin. The gate is what makes on correct again, so the
pin and the flag are coupled: rolling VLLM_CPP_VERSION back before 89c46aeb while
leaving MLX on would select the middle row and roughly halve throughput. Both the
Makefile comment and the README state that dependency explicitly.
The bump also brings six Metal kernels landed upstream since the old pin — mma
prefill attention, a vectorised decode V accumulation, vectorised attention
staging, a fused qk-norm-RoPE preamble, a simdgroup-per-row softmax and a
simdgroup-per-head preamble — which take the non-MLX Metal path from 89.4% to
96.4% of MLX-LM on their own.
One caveat, recorded in the README: MLX's GEMM is not bit-identical to the native
kernel, so an MLX build produces a different greedy sequence than a non-MLX build.
That is a property of the provider rather than of the gate and predates this
packaging.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(vllm-cpp): correct the MLX-gated figure to 97.6%, from 99.1%
The previous commit quoted 99.1% of MLX-LM for the prefill-gated MLX build. That
figure divided by a two-run MLX-LM baseline, 27.135 and 27.744 generation tok/s
averaged to 27.44. Re-measured interleaved with ours over four ABBA blocks,
MLX-LM's decode is 27.848 with a 0.34% spread across six runs, so the 27.135 was
an outlier and averaging it in overstated us by roughly 1.5 points.
Corrected: the gated configuration is 24.37 tok/s, or 97.6% of MLX-LM, and the
MLX-off build is 23.9 tok/s or 95.9%. Prefill TTFT is unchanged at 524.5 ms
against MLX-LM's 532.6, so we remain about 1.5% faster there.
Nothing else changes. MLX still wins prefill and loses decode, the shape gate is
still the right disposition, and the pin and the flag are still coupled. The gate
is worth about 1.7 points over the MLX-off build rather than 2.7.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): pin MLX gate from mainline
The previous pin was a merge commit from the experimental C ABI v9 branch. Pin the same MLX prefill gate on upstream main so the backend build does not pull unrelated ABI v9 work into every platform variant.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): restore backend build portability
Keep the current master pin when enabling MLX so every backend variant builds against the known-good vllm.cpp revision. Suppress Apple clang’s GNU constant-folding diagnostic for Objective-C++ Metal compilation only, since upstream treats warnings as errors.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): demote MLX header VLA warning
MLX 0.29.3 headers trigger Apple clang's gnu-folding-constant diagnostic in the Objective-C++ provider. Keep the diagnostic visible while exempting only it from vllm.cpp's global warnings-as-errors policy.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): suppress MLX header VLA warning
Target-level Objective-C++ -Werror is appended after the directory flags, so a no-error demotion is re-promoted. Disable this single warning for the MLX header while keeping every other warning fatal.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): pin source-scoped MLX warning fix
Move the AppleClang warning exception into vllm.cpp where its target warning policy is defined, and pin LocalAI to that source-scoped fix.
Assisted-by: Codex:gpt-5
* fix(vllm-cpp): pin effective MLX warning suppression
The source-scoped no-error flag was overridden by the target warning policy. Pin the companion vllm.cpp change that disables only the MLX header diagnostic for its Objective-C++ translation unit.
Assisted-by: Codex:gpt-5
* fix(vllm-cpp): pin diagnostic pragma fix
Pin the companion vllm.cpp correction that scopes the AppleClang folding warning suppression inside the MLX translation unit, after command-line warning policy.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): pin remaining Darwin build fixes
Advance the MLX-enabled backend to the vllm.cpp revision already validated by the dependency update branch. This includes the feature guards and AppleClang pragma boundary needed by the Darwin build.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): pin MLX system dependency boundary
Pin the companion vllm.cpp change that models MLX as an imported system dependency, keeping third-party header diagnostics out of the project's warnings-as-errors policy while retaining fatal warnings for project sources.
Assisted-by: Codex:gpt-5 [Codex]
* fix(vllm-cpp): pin scoped MLX warning guard
Advance vllm.cpp to the companion fix that keeps MLX headers on a SYSTEM dependency and scopes AppleClang folding-constant suppression to the external includes.
Assisted-by: Codex:gpt-5 [systematic-debugging] [test-driven-development]
* fix(vllm-cpp): use available MLX wheel
MLX 0.29.3 is no longer available to the Darwin runner, so the backend build stopped before CMake. Pin the first available compatible wheel and keep the documented default in sync.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update antirez/ds4
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(ds4): link upstream CUDA MMQ objects
The updated ds4 CUDA object now calls into the vendored MMQ implementation. Build and link those objects into both the gRPC server and distributed worker.
Assisted-by: Codex:gpt-5 [Codex]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Wait for observable loop events instead of budgeting hundreds of milliseconds for scheduler timing. Keep a short bounded overlap observation for the two-leader exclusion check.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The blog has a deep post for 4.8 and a history post that covers the earlier
releases at summary altitude, but nothing in between. These five fill that
gap in the same shape as what-landed-in-localai-4-8: what the release was
for, runnable examples, and the limits that apply.
Every endpoint, CLI flag, env var and gallery entry is verified against the
matching release tag rather than taken from the release notes. That caught
two paths the published 3.10.0 notes got wrong: tracing is /api/traces, not
/api/v1/trace, and a stored response is fetched from /v1/responses/:id, not
/api/v1/responses/{response_id}.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* blog: rewrite the engines post without the AI tells
The HN thread on this post (item 49125065) spent most of its comments on the
writing rather than the engines. Readers quoted specific lines back as tells.
This is the same post with the same numbers, edited against the updated
no-ai-slop skill.
Every figure, table and link is unchanged, except that "27% of the memory"
is now the underlying 363 MB against 1328 MB from the table.
Two substantive framing fixes, both from the reply draft in
hn-reply-engines-post.md:
- vllm.cpp is no longer implied to be a speed win. The table is a tie, the
result is the install size, and the post now says so before a reader has to
work it out and post about it.
- Added one line on the language mix. Readers took the C++/Python/Go tree as
incoherence rather than as a Go core with per-ecosystem backends.
Cut throughout: the ledger metaphor ("what those ports buy", "not paid for in
throughput"), unearned framing ("the honest reading is", "has nothing to do
with"), the shape summary ("that is the general shape of these wins"),
confident deference ("people who are better at those models than we are"),
self-grading numbers ("a good result for a 66 MiB binary"), verbless
comparisons, three of the four exactness idioms, and the aphoristic headings
and verdicts. The double-tricolon summary is one plain clause now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* blog, website: same anti-slop sweep over the rest of the site
One-by-one pass over the other four posts and the site templates, with the
same rules used on the engines post. All figures, tables, links and PR
numbers are unchanged everywhere; the edits are to prose only.
apex-moe-quantization: ledger metaphors were the main issue, eight uses of
buy/cost/pay/spend for things that are not money. Also "the honest reading
is", "that is the comparison that matters", and two section-ending aphorisms
("Size is a speed knob as much as a memory knob", "Q6_K is the ceiling worth
paying for").
localai-since-march-2023: light touch, this one already reads like a person.
Removed "the curve is not the point", a "not the feature list, but the four
decisions" contrast, and two "X is what made / is the piece that" forms.
parakeet-cpp-asr-on-cpu: six exactness idioms across one post, "byte for
byte" twice, "character for character" twice, "byte-identical" twice and
"bit-identical" once, including in the title. Down to one, kept where the
precision is load-bearing. Also the "what end-of-utterance detection buys
you" heading and the "we say so rather than averaging it away" flex.
what-landed-in-localai-4-8: no changes. It is dense, flat and ends every
section on a PR number or a plain fact, which is the shape the other posts
should look like.
Site templates: "Most backends wrap somebody else's engine. These do not."
was the same contrast the engines post opened with. Also "Not a degraded mode
that technically runs", "A port only ships once it matches the original",
"Speed is the part we then go and win ... not a marketing run", and the last
"byte for byte" on the landing page.
Hugo builds clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* website: it is eighteen engines, not nineteen
Three places said nineteen: the /engines/ page description, the JUL 2026
timeline entry on the landing page, and the header comment in
data/engines.yaml.
Eighteen is right, confirmed two ways. The "Backends built by us" table in
the README has exactly 18 rows, and data/engines.yaml has 19 entries of which
one is apex-quant, which is a quantization recipe rather than an engine. The
two lists otherwise match name for name.
The yaml comment is the likely origin: it read "the nineteen native engines
the LocalAI team wrote, and the one quantization recipe that feeds them",
which counts apex-quant twice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
master does not compile:
vet: core/http/endpoints/mcp/localai_assistant_test.go:157:19:
method stubClient.ListScheduling already declared at
core/http/endpoints/mcp/localai_assistant_test.go:87:19
Two fixes for the same breakage landed. The four Scheduling methods were
already present at lines 87-99, in interface order after ListNodes, by
the time #11318 merged; #11318 appended its own copy after
GetRouterDecisions. The two blocks sit in different parts of the file, so
git merged both without a conflict and nothing flagged it.
Remove the appended copy and keep the one in interface order. Pure
deletion, no behaviour change.
Verified: go vet clean on ./core/http/endpoints/mcp/, and
go test ./core/http/endpoints/mcp/ passes.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The News page kept a hand-maintained "Highlights" list that had drifted:
it was missing all of 2025, duplicated the README's own news list, and
linked /features/middleware/ for a page that lives at operations/.
Both of its jobs already have owners. website/content/blog/ carries the
release write-ups and engineering notes, and GitHub Releases carries the
full changelog. Replace the list with a pointer at those two, so there is
one place to update instead of three.
The page keeps its url and front matter, so /docs/basics/news/ and the
root /basics/news/ redirect that .github/ci/gen-redirects.sh generates
both keep resolving.
Also drop the two contributor instructions in .agents that told authors
to add a whats-new.md bullet per feature: announcing a capability is the
release blog post's job, per .agents/preparing-a-release.md.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Write] [Bash]
Deploy site to GitHub Pages failed on five of the last eight master
pushes, always in the build job before Hugo runs:
Setup go version spec 1.22
...
go: downloading go1.26.0 (linux/amd64)
go: download go1.26.0: golang.org/toolchain@v0.0.1-go1.26.0.linux-amd64:
Get "https://proxy.golang.org/...": connect: network is unreachable
##[error]Command failed: go env GOPATH
The workflow pinned setup-go to 1.22 while go.mod declares go 1.26.0, so
the `go run ./.github/ci/modelslist.go` step that generates the gallery
page had to fetch the real toolchain from proxy.golang.org first. That
fetch is not reliably reachable from the runner, which is why the deploy
alternated between passing and failing rather than failing outright.
Track go.mod instead of a literal. The version the module needs is then
installed directly and there is no toolchain download to fail.
This matters beyond CI noise: the docs and the site, including the
release blog post, ship through this workflow.
Scoped deliberately to gh-pages, the workflow with the observed failure.
test-extra.yml pins 1.25.4 in a dozen places and is below go.mod for the
same reason, so those jobs also download a toolchain, but they are
currently green and rewriting twelve pins on a hunch risks more than it
fixes. Worth a follow-up.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
#11228 added ListScheduling, GetScheduling, SetScheduling and
DeleteScheduling to localaitools.LocalAIClient but did not update
stubClient, the hand-written test double in the mcp endpoints package.
The package therefore fails to typecheck, which takes out both lint and
tests on master:
cannot use stubClient{} as localaitools.LocalAIClient value in
argument to h.Initialize: stubClient does not implement
localaitools.LocalAIClient (missing method DeleteScheduling)
Red on 8f74f74b, fd4ec083 and 8a68f357; green on cd62e8ff, the commit
before.
Add the four methods with the same inert bodies the rest of the stub
uses. The real implementations are covered in the localaitools suites;
this double only exists so the holder can be constructed.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
chrome-audit.spec.js walks 25 routes in a single test, and has been the
UI E2E suite's failure on 5 of the last 6 master runs. It always dies the
same way, at the 30s per-test default:
Test timeout of 30000ms exceeded.
Error: page.waitForTimeout: Test timeout of 30000ms exceeded.
19 | await page.goto(route)
> 20 | await page.waitForTimeout(400)
The spec is new in 5cb0c1a8; the commit before it was green, and every
run since has been red on this file.
The failure is cumulative rather than one bad route. Across those runs
the clock runs out at line 19, 20 or 21 depending on where the loop
happens to be, and the timeout lands on waitForTimeout rather than on
goto, which is what running out of budget looks like as opposed to a
navigation that hangs. 30s over 25 routes is ~1.2s each, including a
deliberate 400ms settle, so there is very little headroom to begin with.
Give the test a budget proportional to its work: six seconds a route.
That absorbs a slow runner and still fails promptly if a route genuinely
hangs.
Verified: the spec passes on the current UI in 12.2s solo, and the full
suite passes 418 at 8 workers locally. What I could NOT do is reproduce
the CI timeout on this machine, which has 20 cores against the runner's
2 to 4; under synthetic CPU load it still finished in 13.5s. So the fix
is argued from the CI signature and the arithmetic, not from a local
repro, and the proof is this spec going green on the hosted runner.
Note test.setTimeout() has to be called inside the test body. At module
scope Playwright rejects it with "test.setTimeout() can only be called
from a test".
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Every CUDA sglang image failed in the 2026-08-02 full-matrix rebuild:
-gpu-nvidia-cuda-12-sglang, -gpu-nvidia-cuda-13-sglang and
-nvidia-l4t-cuda-13-arm64-sglang, all with the same build error.
Building cuda-tile==1.6.0rc3
x Failed to build `cuda-tile==1.6.0rc3`
ModuleNotFoundError: No module named 'wheel_stub'
hint: `cuda-tile` (v1.6.0rc3) was included because `sglang` (v0.5.16)
depends on `flashinfer-python` (v0.6.14) which depends on `cuda-tile`
This is the failure mode requirements-cublas1{2,3}-after.txt already
carries an nvidia-modelopt bound for, arriving through a different
package. install.sh passes a global --prerelease=allow, which is
load-bearing for flash-attn-4, so an unbounded dependency resolves to a
prerelease; cuda-tile 1.6.0rc3's build backend imports wheel_stub without
declaring it in build-system.requires; --no-build-isolation means nothing
provides it, and the build dies.
Nothing in this repo changed. cuda-tile published 1.6.0rc1 and rc3 and
the weekly cron picked them up, which is the drift that job exists to
catch.
Bound the one package rather than dropping the global flag, matching the
existing precedent. 1.5.0 is the newest stable release, so <1.6 takes the
last good one. l4t13 gets the same bound: it installs plain sglang rather
than sglang[all], but flashinfer-python is a dependency of both.
NOT VERIFIED LOCALLY: reproducing this needs a CUDA docker build, which
this machine cannot run. The diagnosis is from the CI log and the
resolver's own hint, and the change follows a fix already proven in these
same files. CI on this PR is the check that matters.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Since #11255 and #11276 every GPU image also builds ggml's CPU_ALL_VARIANTS
matrix, so a partial offload uses the host's SIMD kernels. That works
everywhere except SYCL, where the Makefile compiles the whole tree with
icpx -fsycl: icpx never finishes ggml-cpu/arch/x86/repack.cpp at
-march=sapphirerapids. In run 30765516644 both sycl_f16 and sycl_f32 stopped
at that translation unit and sat there for 5h30m with a single compile in
flight until GitHub killed the job at its 6h limit, and turboquant's f16 job
lost its runner outright. gcc compiles the same file in seconds in the vulkan
and CPU jobs of the same run, so the CPU variant matrix is only unbuildable
under icpx.
Route SYCL back to the portable fallback binary, which is what these images
shipped before #11255. run.sh already prefers *-cpu-all when present and falls
back otherwise, so nothing else has to change.
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(api): add text moderation endpoint
Add an OpenAI-compatible /v1/moderations endpoint backed by constrained local text generation. Register its auth and discovery surfaces, document the text-only MVP, and cover response shaping and access control.
Assisted-by: Codex:gpt-5
* test(mcp): update assistant client stub
Keep the LocalAI Assistant holder test stub aligned with the scheduling methods added to LocalAIClient so repository-wide type checking succeeds.\n\nAssisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(api): add POST /v1/images/upscale endpoint
Add a new image upscaling endpoint that accepts a source image and
returns an upscaled version. Supports selectable upscaler models
(e.g. realesrgan) and a configurable scale factor (2x or 4x).
- backend.proto: add UpscaleImage RPC and UpscaleImageRequest message
- pkg/grpc: implement UpscaleImage in Backend interface, client, server
and embed shim
- core/backend/upscale.go: new backend helper (mirrors ImageGeneration)
- core/http/endpoints/openai/upscale.go: new multipart/form-data handler
- core/http/routes/openai.go: register POST /v1/images/upscale
- core/http/auth/features.go: gate upscale routes under FeatureImages
- backend/python/diffusers/backend.py: implement UpscaleImage — uses
diffusers upscale pipeline when loaded, falls back to Lanczos resize
* fix(grpc): add UpscaleImage stub to Base backend
All Go backends embedding Base now satisfy the AIModel interface
without needing to implement UpscaleImage explicitly.
* fix(images): complete upscale endpoint integration
Store generated upscales under the served images directory, validate scale factors, document and advertise the endpoint, and add a functional Stable Diffusion x4 gallery model.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Give gallery operations distinct pause and cancel paths. Pause preserves partial download data so reinstalling the same model or backend resumes through HTTP Range, while cancel keeps its destructive semantics. Surface the action in the Activity UI and document the API behavior.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add multilingual 1B and 8B Q4_K_M GGUF embedding entries and link them as variants for automatic memory-aware selection.
Assisted-by: Codex:gpt-5 [web]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
fix(gallery): repair DeepSeek V4 0731 entry
Use the official single-file ggml-org MXFP4 artifact with its verified SHA256 and route it through llama.cpp instead of treating an unsloth repository page as a ds4 model file.
Assisted-by: Codex:gpt-5 [Hugging Face API]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add Q4_K_M and Q8_0 GGUF builds for the trending Instella-MoE-16B-A3B-Think model, with host-selectable variant metadata and verified Hugging Face LFS hashes.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(ui): give the Operate overview real numbers and traces a latency shape
First two items from a component-by-component comparison against the mocks.
The pattern that audit found: everything newly built matched, everything
pre-existing got the palette but not the layout, and an "absent rather than
empty" rule hid most of the overview exactly when someone was looking at an
idle installation.
**The headline grid is always rendered**, including at zero, with a fourth cell
for host memory. Hiding it removed the page's structure precisely when it was
most likely to be read, and "0 failed" is information — an absent panel is not.
The quiet case is now said in a line underneath instead of by showing nothing.
**The sections state counts** rather than listing their destinations: backends,
models, updates and running operations instead of the words "Usage and traces".
That needed installed backend and model counts in the summary context, which
are two more cheap reads on the poll that was already running.
**Traces rows carry latency as a bar as well as a figure**, scaled against the
slowest request currently in view and turning amber past two seconds. The table
had no latency column at all — the number was buried in the expanded detail, so
the shape of the tail was invisible while scanning. Scaling against the view
rather than an absolute ceiling is deliberate: what matters when reading a page
of traces is which of these are the outliers, and an absolute scale flattens
every row on a fast installation into nothing.
Full e2e suite: 409 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): name the engine on Home's resident models, and add jump-back-in
Third item from the mock comparison.
The mock showed each resident model with the engine serving it. /system carried
only the id, so the audit recorded this as blocked on a server field — but the
config loader is already in scope where that response is built, so it is one
lookup. SysInfoModel gains an optional `backend`, resolved from the model's
config and omitted rather than guessed when there is none (a loose file, or a
config since removed). Home renders the column blank in that case; the test
pins both halves of that.
Memory per model stays out. It is not one lookup — it would mean asking each
backend process — and inventing a number beside a real one is worse than
leaving the column off.
"Jump back in" is the block the mock had and Home did not. The quick-links row
above it is a set of first-run actions; these are the three places someone
returns to, each stated with what it currently holds rather than as a bare
label.
Go: routes suite passes. Full e2e suite: 412 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): rank the recommended models as lanes instead of equal cards
The hardware recommendations were a grid of equally-weighted cards. The list is
already sorted by fit, and a grid throws that order away: three cards side by
side say "pick one", when the page has actually formed an opinion about which
one.
They are lanes now, read top to bottom in fit order, with the leader carrying
the single amber "Best fit" label and the rest marked "Also fits". One opinion
per page — the alternatives are alternatives, not runners-up each worth their
own colour, which is how a strip of coloured badges ends up meaning nothing.
Below 720px the size and VRAM columns drop and the lane keeps the name and the
install action, which are the two things a narrow screen needs.
The existing panel spec moves off .rec-models-item onto .lane rather than being
deleted; dismissal, collapse, keyboard operation and install all still pass
unchanged, and there is a new assertion that exactly one row is called out.
Full e2e suite: 413 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): drop capsule chips app-wide, and un-break the empty voice library
**Pills are gone.** A capsule radius reads as a tag floating on the surface,
which fights a system whose structure is hairlines and square corners — and
with chips on Discover, Host, Activity and the biometrics pages, "some pages
have pills" was the real inconsistency rather than any one page.
Sixteen selectors move to the small radius: filter buttons, tab pills, activity
and biometrics chips, file and count badges, the jump-to-latest control, the
nav badge. Round *buttons* keep their circle — .lightbox__nav and
.home-send-btn are circles, not capsules — as do every progress track, status
dot and avatar, which are round because they are round, not because they are
tags.
**The empty voice library was unusable.** `.voice-library-empty` sets
min-height: 430px, border: 0 and background: transparent — a description of the
empty PANEL — and it had been attached to the action instead. The create button
was therefore a 430px transparent box that pushed itself out of the panel and
could not be seen. Moved onto the container it describes, which now centres its
action rather than letting it fall off the bottom. Same class-mangling shape as
the Agents header fixed earlier.
Full e2e suite: 416 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): put Host's headline figures on the shared hairline strip
Host had shadowed, clickable StatCards above a page that already has a rail, a
pane and a tab bar — a second dashboard language on one screen, and a different
one again from the figures inside its own detail pane.
The Operate overview's figure grid is generalised into a shared `.stat-strip`
and Host adopts it, so the two pages read as one system: same cell, same figure
scale, same tone vocabulary, and the same hairline grid the split-view StatGrid
already uses. The cells stay clickable and still route into the tab and filter
they describe, because a count is worth more when it is also the way to the
thing counted.
Tone is spent only where the number means something — running and updates when
non-zero — since a strip where every cell is coloured has no emphasis left.
Two bugs made on the way, both now covered:
- The first version put `<button>` elements inside a `<dl>` with `<dt>`/`<dd>`
inside the buttons. Neither is valid, the browser re-parents both, and the
cells collapsed. These cells are a set of controls, so a plain container of
buttons is also the honest markup.
- Even correct, the strip rendered 2px tall: `.page--app` is a flex column
whose split view takes flex:1, so a child with no intrinsic minimum is shrunk
away. The old cards survived only because `.stat-card` carried
min-height:96px. The strip now declines to shrink, with a test pinning it.
The stat-card specs are retargeted rather than deleted: they were written to
guard a class collision on a page that no longer uses cards, so they now guard
the strip's labels and its height.
Full e2e suite: 417 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): make Backends notices an edge rather than a filled card
The install and upgrade banners were tinted cards with a full border. A filled
panel makes every notice shout at the weight of an error, which is how notices
stop being read — and Backends shows one on most visits, so it was shouting
routinely.
They are now a hairline with a coloured left edge, the same treatment the
Operate overview gives rows that want a decision, so "this needs you" looks the
same wherever it appears. Counts in the notice take the monospace tabular
figures the rest of the console uses.
Also drops the last inline style on the page, and refreshes the inline-style
baseline, which has read 624 against a real count since #11288 landed. The gate
exits 0 either way, so nothing was failing — but a baseline 86 above the truth
would have let that many inline styles back in unnoticed. Now at 538, which
tightens the ratchet rather than loosening it.
The spec creates the upgrade it asserts on rather than skipping when the mock
has no notice: a test that skips is a test that proves nothing.
Full e2e suite: 418 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): finish the mock parity list, and stop hiding the recommendations
The last two items from the audit, plus a correction.
**Discover's use-case shelf is lanes.** These are a list of ways in, read in
order; a grid of equal cards asks the reader to compare them, which is not the
choice on offer.
**The request panel reaches every generator.** Video, 3D, Sound and Audio FX
join Images and Speech, so each one teaches its own endpoint rather than two of
six doing it. Audio FX records the fields that shape the request rather than
the bytes, since its payload is multipart.
**Recommendations no longer collapse themselves.** They were folded away by
default once anything was installed. That is the page's one opinion about this
host, and an opinion hidden by default is one the reader never gets. Someone
who disagrees can still collapse it and that choice is remembered — the
difference is that we no longer make it for them. Three specs asserted the old
default and now assert the new one.
The use-case heading also sat a line's width from the text it introduces, so
the two read as one paragraph. It has air under it now, and the shelf is
separated from the recommendations above it.
Two tests removed rather than kept: a generator loop whose only real assertion
was `expect(endpoint.length).toBeGreaterThan(0)`, and an earlier card-gap guard
that could only skip. A test that cannot fail is worse than no test, because it
reads as coverage.
Full e2e suite: 418 passed, 4 skipped. Inline styles at baseline.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): make the Host figures legible and give the strip its spacing back
Three defects introduced by the Host redesign, all found by looking at the
running app rather than by the suite.
**The figures were invisible.** "Running now" and "Updates available" rendered
pure black on the dark ground. Two causes compounding: the `--muted` tone alias
never landed, because the source rule has extra spaces before its brace and the
exact-match edit missed it silently; and a `<button>` does not inherit colour,
so with no tone rule the value fell back to the user agent's `buttontext`.
Both fixed, and a test now fails on any figure computing to pure black.
**The strip sat flush against the resources panel.** `.stat-strip` declares
`margin: 0 0 ...` and is declared later in the file than `.manage-summary`, so
the shorthand quietly won and the top margin became zero. Raised to
`.stat-strip.manage-summary` so it beats the shorthand on specificity rather
than on declaration order, which is the kind of thing that breaks again the
next time a rule moves.
**Discover's use-case heading had a doubled gap.** `.zero-pane` is a flex
column that already separates its children; adding a margin on top of the gap
stacked the two. The margin is gone and the heading keeps only its own breathing
room.
Full e2e suite: 420 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): make Studio's tabs path segments rather than a query parameter
`/app/studio?tab=images` reads like a filter applied to a page. It is
navigation: a different generator, with its own state and its own deep link. It
is now `/app/studio/images`, with the overview at `/app/studio`.
Legacy `?tab=` links are redirected once to the path form, replacing the
history entry so Back does not bounce between two spellings of the same place.
Bookmarks and older links keep working and land on the canonical URL rather
than a second version of it, which is the part worth having a test for.
The nine `?tab=` references were all in specs, none in docs, so the migration
is contained. They move to paths, and a new spec pins the redirect.
Full e2e suite: 421 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): make the hardware recommendation a section, not a dismissable card
It was a bordered card with a collapse control and a close button, sitting
inside a pane that is otherwise hairline sections. Two problems: it read as
something bolted onto the page rather than part of it, and treating it as an
interruption to be shut is the wrong frame for the one thing the page has to
say about the machine it is running on.
It is now a plain section with the same heading treatment as the shelves below
it. The collapse state, the dismissal, their storage keys and the legacy key
read for backwards compatibility all go with it, along with the installedCount
prop that existed only to pick a default collapse.
Five specs described behaviour that no longer exists and are removed rather
than adjusted — collapsing, dismissing, persistence of both, and the toggle's
keyboard handling. One new spec asserts the replacement contract: no control
with aria-expanded, no dismiss, and no card border.
Full e2e suite: 416 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): restore every stripped icon and every default-chrome control
You reported two broken icons. They were not two: an earlier automated edit had
stripped the `fa-*` class from twenty `<i>` elements across eleven pages, and an
`<i>` with no icon class renders nothing at all. Settings' save button, the
voice-profile back link, and eighteen others — agent row actions, task and job
buttons, import and create actions — were all drawing empty space.
Each is restored from its own context rather than a blanket icon: the agent row
gets pause/play, pen, comments, file-export and trash; the fine-tune toggle
swaps plus for xmark as it opens; the P2P documentation link gets the
external-link glyph.
The same edit left controls without their classes. Fine-tune's "Import config"
was rendering in the browser's own chrome, and `.p2p-cmd__copy` set a border
but no background, so it fell back to `buttonface` — a pale grey chip on a dark
command block. FineTune's "New job" also had its icon classes folded into the
button's className, the same mangling already fixed on the Agents header.
Rather than fix the reported two and wait for the next report, this adds a
standing audit: twenty-five routes are walked and the test fails on any visible
control rendering with user-agent chrome, or any `<i>` without an `fa-*` class.
It found the three remaining cases after the first sweep, and it is the reason
the next one cannot ship quietly.
Full e2e suite: 417 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): give Operate a front door and fold six rail groups into four
Opening Operate ran firstVisiblePath() and landed on Backends, because
Backends happens to be written first in operateConsole.groups. The section
that should answer "is anything wrong" opened on a package manager, and
nothing was reported until you visited it.
Adds /app/operate. Its one irreplaceable block is "Needs attention", which
is empty when nothing is wrong and says so in a line rather than rendering a
reassuring green panel. It collects stale backends, failed operations and
unhealthy nodes. Everything else on the page is a summary you could already
assemble by visiting four others.
The rail regroups from six headings to four: Inference and Activity were both
"the runtime right now", Access and System were both administration. No
destination is removed and no gate changes, so isConsoleItemVisible and
consolePaths are untouched. Overview leads the first group, which is what
makes firstVisiblePath() return it without knowing it exists.
Rail entries now carry a signal beside the label. This does not replace the
sidebar badge and is not built as if it does: the badge stays on the
always-visible sidebar entry for the reason recorded in Sidebar.jsx, that the
rail exists only on Operate routes and can be collapsed. The signals are
orientation while inside Operate, so they are aria-hidden and nothing urgent
depends on them alone.
OperateSummaryContext polls once for the whole console, following
OperationsContext, which exists because per-consumer setInterval against one
endpoint was the defect it fixed. It is mounted by ConsoleLayout for the
Operate console only, so "poll only while in Operate" needs no route check.
Built on usePolling, so it pauses on a hidden tab. Operations are read from
OperationsContext rather than polled a second time, and each source degrades
to no-signal on its own so one dead endpoint cannot blank the rest. It reads
the cached GET /api/backends/upgrades and never the POST that forces a real
registry check.
Traces and Usage get no signal yet: /api/traces returns the list, so a count
would mean fetching every trace to render one number. A counts endpoint is
the honest fix and is scoped separately.
Full e2e suite green (369 passed, 4 skipped), including a render-smoke entry
for the new route.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): open Studio on what this machine can actually make
Studio was a tab strip over six generators that opened on Images, which was
never a decision, only the first entry in BASE_TABS. Nothing said which
modalities this installation could run, so the way to learn that video had no
model was to pick the tab and find an empty select.
Adds an overview tab and makes it the fallback. Explicit tabs still win, so
existing deep links keep working; anything unrecognised or gated now lands on
the overview rather than Images.
Each tab carries a dot: filled when an installed model advertises that
modality, hollow when nothing serves it. That is the feature in one detail,
turning the strip from navigation into a report of what the machine can do
before anything is clicked. The dot is aria-hidden because the overview states
the same facts in words and the dots change as models load.
Two kinds of unavailable, which had to stop looking alike:
- switched off, via a permission: no tab and no lane, unchanged
- available with no model: a lane, and a route to installing one
Studio now owns one MODALITIES table so the tab strip and the overview cannot
disagree about what exists, and calls useModels() once, unfiltered, grouping in
the browser. useModels(capability) fetches the whole list and filters locally,
so a hook per modality would have been six identical requests to
/api/models/capabilities on every mount. There is a test for that.
Recent outputs read every localStorage store through a new
readAllMediaHistory(), which avoids mounting five hooks that carry save timers
the overview has no use for. 3D is read separately through use3DHistory rather
than folded in: its entries are GLB blobs in IndexedDB, so they cannot come
from the same synchronous read.
Typical cost is the median of this machine's own history, not a guess, and
renders as a dash when there is nothing to go on.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): stop the stat cards and the console rail breaking on small screens
Two unrelated causes behind one report that /app/manage looks wrong when the
window is narrow.
The stat cards were being laid out by the wrong rule at every width. Two
different components both claimed `.stat-grid`: the dashboard card strip that
holds .stat-card children, and the detail-pane StatGrid the split views
introduced further down App.css. Being later, the second won every shared
property, so the cards got its 120px columns and its 1px hairline gap in place
of their own 180px columns and spacing-md. Four cards were packed onto a row
that fits two, labels wrapped to three lines and clipped, and the icon crowded
the value. Renamed the strip to `.stat-cards`, after the children it actually
holds, which also removes the mismatch of a `.stat-grid` container full of
`.stat-card`s. The split-view component keeps `.stat-grid` and its BEM parts.
The expanded console rail had no bounded height. Thirteen destinations stacked
in one column is taller than a phone, so opening the menu pushed the page's own
heading past the fold: the menu replaced the page rather than annotating it.
Capped at 55vh with internal scrolling below 768px, so the content behind stays
reachable.
Both are asserted on behaviour rather than markup: no stat-card label may be
clipped, the card gap must not be the detail pane's hairline, and expanding the
rail must leave the page heading on screen.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): retemper the palette to localai.io and add the lane primitive
The token half of the style transfer, plus the shared list idiom the two
overviews had each grown their own copy of.
theme.css moves from Nord to the website's palette, variable names preserved so
every consumer moves with it: ground #13171f -> #0d1117, accent frost cyan
#88c0d0 -> action blue #4f8cff, success sage -> mint #56d6a4, warning -> the
amber #f1b95d the site spends only on the thing asking for a decision. Eyebrows
go mint. Dividers become an opaque #29384a hairline rather than alpha over a
varying surface, which is what makes stacked surfaces read crisply on the site.
Light is derived, not inverted. The site ships one theme and never had to
answer this, but the app does: blue darkens to #2f62d8, mint to #0d8b60 and
amber to #8a5d0b, all clearing 4.5:1 on a cool paper ground, where the
dark-mode values sit near 2:1. Same three roles, different values.
Three files restate the palette because CSS variables cannot reach them:
cmTheme.js (the whole CodeMirror theme), VoiceVisualizer and WaveformPlayer
(canvas). Left alone they would have quietly kept the app half-Nord.
The `.lane` primitive replaces the near-identical row CSS that OperateOverview
and StudioOverview had each written: a full-bleed row on a hairline that insets
on hover, with no card and no shadow. Callers supply only the column template.
Both pages now use it, along with `.lane-head` for section rhythm and a
`.page-pad` container for top-level pages outside a console shell — without
which Studio sat flush against the sidebar with its eyebrow clipped.
Studio's tab strip wraps rather than running off the edge at narrow widths.
Full e2e suite: 386 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): put Home's resident models on lanes and give the footer one line
Home's status line was three chips saying a thing was true. It now reports
figures: how many models are resident, how many nodes are healthy, what share
of memory is in use, set in tabular monospace so the digits line up. A chip
answers whether; a figure answers how much, which is what someone opening the
page at a glance is after.
Resident models move from status chips to lanes, with the id set in a new
`.lane__name--id` because an id is something you might type or paste and the UI
face makes it read as a label. /api/system-information carries only the id, so
there is deliberately no backend or memory column: inventing one would mean a
server change this does not make.
The footer was three centred rows and cost the bottom sixth of every page for
chrome. It is one line now, version left and links right, wrapping to centred
when the viewport is too narrow to hold both. Every link it had, it keeps.
Full e2e suite: 392 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): correct three contrast failures and stop a guaranteed-404 poll
A contrast audit of the new palette found three values below WCAG AA, one of
which the previous commit message claimed was fine:
- White on the #4f8cff button is 3.22:1, which is large-text only. The website
does exactly this, but a button label in an app is not large text, so the
label goes to dark ink at 5.88:1. Light mode keeps white, which is 5.44:1 on
its darker blue.
- Light-mode success was 4.08:1 on paper, not the 4.5 claimed. Darkened to
#0a734f, 5.56:1.
- Nord red was already 4.28:1 on raised surfaces, a pre-existing miss carried
over unexamined. Lifted to #c96f78, 5.02:1.
Lanes gain the two states they were missing: a 44px target on coarse pointers,
matching what EntityRail already does so the two list idioms feel the same
under a thumb, and a reduced-motion variant that keeps the background feedback
while dropping the hover inset, which is a position change.
The Operate summary no longer asks for /api/nodes on a single-node install. The
cluster API answers 503 when distributed mode is off, so it was a guaranteed
miss every fifteen seconds; it is now gated on useDistributedMode, the same
condition the rail already uses for the Nodes entry.
Full e2e suite: 392 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): restore the gap between overview blocks, and stop claiming zero nodes
Two defects a design review surfaced.
`.lane-head:first-child { margin-top: 0 }` was meant to stop the first block on
a page carrying a top margin. But every <section> makes its lane-head a first
child, so the reset applied to all of them and the gap between blocks vanished:
"Sections" sat flush against the attention row above it. The header supplies its
own bottom margin, so a uniform top margin is correct everywhere.
The Cluster summary read "0 nodes" on a single-node install, which looks like a
fault when the cluster API is simply switched off. It now says "Single node".
Full e2e suite: 392 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): open dark by default, and stop clipping the collapsed sidebar footer
Dark is the identity rather than a preference: localai.io ships one theme and
it is this one, so an install should look like LocalAI before anyone has chosen
anything. The OS setting no longer selects light on first load. The toggle
still does, and a stored choice wins forever after, which the tests assert
both ways.
The collapsed sidebar footer stacked its controls but kept the expanded row's
inline padding, so their edges were clipped against the 51px rail.
Full e2e suite: 394 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(api): count traces server-side and give the Operate overview real totals
The overview's headline block had no source. /api/traces returns the trace
list, so "37 errors in 24h" meant fetching every buffered exchange to count it
in the browser — waste that grows with the buffer, to produce three integers.
Adds GET /api/traces/summary: totals, failures, p95 and a bucketed series for
sparklines, over a window that defaults to 24 hours and is capped at a week.
Deliberate calls, each with a spec:
- A 4xx is the caller getting it wrong, not the installation being unhealthy,
so only 5xx and transport errors count as failures.
- p95 is a nearest-rank percentile rather than the slowest request, which is
what a max would report and what makes latency panels lie.
- Buckets are oldest-first so a sparkline reads left to right, and the slice is
never nil: nil serialises as null and breaks .map() on the other side, which
is a silent runtime error rather than an empty chart.
- Exchanges outside the window are not counted at all.
The route is registered before /api/traces/:id so "summary" is not captured as
a trace ID.
On the client, Traces and Usage gain the rail signals they were shipped
without, the Observability section summary now states counts instead of listing
its destinations, and an installation that has served nothing says so rather
than showing three zeroes dressed as telemetry.
Sparkline is a bare stroke with an emphasised endpoint and no axes: the figure
above it already states the value, so its only job is the shape.
Go: 185 middleware specs pass. Full e2e suite: 396 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): stop the memory chart calling a trade-off an error
The VRAM-by-context chart rendered any build over the limit in error red, and
escalated the verdict to the error tone as soon as two context sizes crossed
it. But an over-limit build still installs — #11288 keeps a test on exactly
that — so red overstates what is happening. A model that fits at 32k and not
64k is a trade-off, not a fault.
Over-limit bars and the limit line now use the warning tone, which is the
constraint colour used everywhere else in this branch: know what you are doing,
not you may not. The error tone is reserved for "fits nowhere", where the model
genuinely cannot run on this host.
Full e2e suite: 397 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): give the new surfaces orchestrated motion
Uses the reveal system already in the codebase rather than adding a library:
pageReveal, .reveal-stagger and staggerStyle() were built for exactly this, and
anime.js would be ~17KB duplicating four lines of CSS for list reveals.
The overview's headline figures, attention rows and section lanes stagger in,
as do Studio's modality lanes and recent outputs, so a page assembles in the
order it is read instead of appearing all at once.
Two additions beyond stagger. Rail signals transition on opacity when a poll
lands, so a number changing reads as an update rather than a jump cut, and it
stays on the compositor so it cannot reflow the rail. The attention block
animates its left edge in — the one thing on the page that should announce
itself, and on the border rather than the text so nothing moves under a reader.
Both are dropped entirely under prefers-reduced-motion, alongside the lane
hover inset already handled.
Full e2e suite: 397 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): put the generators on a hairline field stack and record the request
The workbench treatment from the mocks, applied where it costs least: both
changes land on shared surfaces, so all six generators get them at once rather
than drifting apart page by page.
The control column stops being a shadowed card of boxed groups and becomes a
hairline field stack — the panel is the page's left half, not an object
floating on it — with uppercase micro-labels matching the eyebrow treatment
used elsewhere. Because .media-controls is shared, Images, Video, 3D, Speech,
Sound and Audio FX all move together.
RequestPanel shows the request the form actually built, with a copy-as-curl.
LocalAI is API-first and Studio is the best place in the app to teach its own
endpoints: the form stops being a black box, and a result worth keeping can be
reproduced from a shell without reverse-engineering which fields the page sent.
It records what was sent rather than what the form currently holds, and renders
nothing until a request has been made — a panel describing a request nobody
made is a tutorial, not a record. Wired into Images and Speech.
Full e2e suite: 401 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): make Chat a transcript instead of a bubble thread
Rounded, filled, asymmetric bubbles fight a system built on hairlines, and they
carry the speaker in shape and side rather than in words. The assistant side
had already given up its bubble; this finishes the job.
Both roles now run full width down one column, separated by a rule, each with a
mono role label. The user turn keeps a left edge in the action tone so the two
are still told apart at a glance, without a fill or a corner radius. The
avatars go: the accent and the label carry the speaker, so the glyph was
decoration once neither side had a bubble.
Saying who is speaking in words rather than in geometry is also what survives
being read aloud, printed, or looked at by someone who cannot pick the sides
apart by colour.
Full e2e suite: 404 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): dress the API reference in LocalAI's palette
The Swagger page was the last surface still shipping in someone else's colours,
which is conspicuous now that everything it links from is dark.
Swagger UI has no theming hook, so rather than fork it we serve our own index
ahead of the library's wildcard and restate the palette over its stylesheet.
The library's own bundle and assets are still what load, so a swagger-ui
upgrade cannot silently break the page — this is a skin, not a fork.
Two things needed real care. Swagger tints the entire operation row per method
via .opblock.opblock-post and friends, so the palette had to match that
specificity rather than reach for !important; the method now lives on one edge
instead of washing across the row, because a page where every row is a status
colour has no status colour left. And the filled method chip put white on pale
green, which was the least readable thing on the page — it is an outlined mono
chip now, carrying the method in its border and text.
Palette values are copied from theme.css rather than referenced: this page is
served by Go and never sees the app's CSS. The comment says so, and says to
keep them in step.
Go: routes and middleware suites pass. Full e2e suite: 405 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): make tall split-view pages reachable, repair the Agents header, scale titles
Three things found by actually using the app rather than measuring it.
**Host was unusable.** The shell above a split view is overflow:hidden so the
document cannot grow, which left anything taller than the viewport simply
unreachable — and Host stacks a resources card, four stat cards and a tab bar
above its split, so the bottom of the pane fell off at every window height with
nothing to scroll. Every sweep I ran for this was horizontal, which is why it
kept coming back clean.
The page now scrolls inside the pinned shell. The pane keeps its own scroller:
letting it grow instead pushes the document taller and stretches the rail to
match, which is the regression e2e/discover-height.spec.js exists to catch, and
which the first version of this fix duly caused.
**The Agents header controls were unstyled** — "Create Agent" was rendering
with the browser's default chrome. The markup had been mangled at some point:
six unrelated classes merged into one string on the link, and the label and
button left with none at all and empty icons. Repaired, with the inline flex
replaced by a shared .header-actions class.
**Page titles take the editorial scale from the site**: larger, tracked at
-0.04em, on a line height near 1, so a two-word title reads as a statement
rather than a label. The typeface is unchanged — DESIGN.md keeps the existing
type system — so the whole difference is scale, tracking and leading, which is
where the site gets its voice from. This was the biggest reason the running app
still did not look like the mocks.
Full e2e suite: 404 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Follow-up to #11288, which warmed the VRAM estimate caches at startup and left
the variant picker paying its own way.
Describing an entry's variants probes the weight files of every build it
offers, so the first time a model is opened costs 1.2-1.9s against a cold
cache. That is the same cost as an estimate wearing a different hat, and it
lands in the same caches underneath, so it belongs in the same pass rather than
in a second mechanism.
The warm-up now describes variants for the entries it walks. Entries that
declare none cost nothing: the call is gated on HasVariants rather than
attempted and discarded. The host resolve env is derived once for the run,
since it describes the machine rather than the entry.
Failure handling matches the estimate half. An entry whose variants cannot be
described is logged at debug and skipped, and the estimate for that same entry
is unaffected, because neither half is allowed to fail the other.
Measured against a live instance with 1,595 models, first ever call to
/api/models/variants/:id after a cold boot:
before 1.2-1.9s
after 2ms
The warm-up's own cost barely moves: 3m0s to 3m19s for 300 entries, of which
40 declared variants. It stays bounded by the same knobs, and
LOCALAI_VRAM_WARM_LIMIT=0 still turns the whole thing off.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): rename the Install Models nav entry to Discover
"Install Models" named the action rather than the destination, and it was
the only multi-word entry in a rail of one-word ones (Home, Chat, Studio,
Talk, Build, Operate). A bare "Models" was the obvious fix but it collides
with the installed-models view under Host, which is a different page for a
different job.
"Discover" keeps the rhythm and says what the page is for. The icon moves
from a download arrow to a compass for the same reason: the page is browsed
before it is installed from.
Translated in all seven locales rather than left to fall back, so a locale
switch does not leave the entry in English.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(ui): replace the gallery table with a rail and a detail pane
The eight-column table was not the real problem; the click-to-expand row
underneath it was. Variants, files and a VRAM estimate never fitted inside a
<tr>, so they were pushed into a drawer that could hold one model at a time,
could not be linked to, and had no room to say anything useful.
The gallery is now a rail to scan and a pane that answers. The pane has two
states and no third: with nothing selected it is the discovery page, and with
a model selected it is that model's detail. Selection lives in the URL, so a
model is linkable and Back steps out of the detail instead of off the page.
The rail groups by capability while browsing and flattens to results the
moment a term is typed. That is a rule rather than a toggle: once someone has
said what they are looking for, the buckets are between them and the answer,
and making the user choose would be handing them our problem.
The detail pane plots VRAM against context length with the host's own limit
drawn across it. This is new information, not a restyle. A single number
invites "so will it run?", and the honest answer is usually "yes, up to a 32k
context", which is a shape rather than a number. The estimates were already
fetched for every context size, so it costs no new request. Backends that
take no context length say so instead of being given a meaningless chart, and
a host with no GPU gets no chart at all rather than bars with nothing to
compare against.
The split-button variant menu goes with the actions column. The pane lists
every build with its backend, quantization, size, fit and a details
disclosure, each installable, which is what the dropdown was a cramped
substitute for. Its tests move onto that list; the three contracts it alone
carried (fetch-once caching, the loading state, an unfit build staying
installable) are backfilled against the pane.
RecommendedModels moves inside the pane, where it has the width to argue for
a model instead of listing one, and keeps its own dismissal and collapse.
Rail entries carry no description. Two lines is the budget and the second is
better spent on whether the thing will run; the stripped-Markdown contract
moves to the pane's lede, tooltip included.
e2e: 123 passing across models-gallery, navigation, recommended-panel,
model-artifact-operation, operations-strip and page-render-smoke. Inline
styles in Models.jsx drop from 82 to 41.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* refactor(ui): extract the split view into shared components
Discover shipped its rail, pane and detail header as private functions inside
Models.jsx. Backends and Host have the same defect and want the same shape, so
leaving them there guarantees three rails that drift.
SplitView, EntityRail, DetailHeader and StatGrid now live under
components/split/. EntityRail is deliberately data-driven: a surface maps its
own entity onto { id, name, icon, meta, stripe, groupId } and keeps its
vocabulary to itself, which is what stops the rail learning about models,
backends and loaded state all at once.
The CSS moves with it. What was .discover__rail is .entity-rail, .discover__
pane is .split-view__pane and so on, because a class named after one page is a
lie on the next two. Only what is genuinely Discover's stays behind the old
prefix: the shelves, the hero and the VRAM-by-context chart.
Two additions the shared rail needs and Discover did not: a state stripe, for
surfaces read by condition before they are read by name, and an empty label.
Discover passes neither.
No behaviour change. e2e 100 passing across models-gallery, navigation and
models-recommended-panel.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(ui): put the backend gallery on the split view
Same defect as the model gallery, so the same shape: a seven-column table over
a click-to-expand row that was the only place the repository, licence, tags and
links could go.
The rail groups backends by the use case they serve, sharing Discover's
taxonomy on purpose: a backend is the runtime a use case needs, so "vision"
ought to mean the same thing one level down. It flattens on a query for the
same reason it does on Discover.
The zero state is the one real departure. A backend's fitness is not free
memory, it is the accelerator and platform it was built for, so the pane leads
with what this host is, then what is not installed yet, then whether anything
installed has gone stale. The table listed 37 runtimes and left "which of these
can even run here" entirely to the reader.
Distribution moves into the pane, which is the one thing a row could never
carry: which nodes hold a copy and which do not, with the install-on-more
control next to it rather than squeezed against a chip.
The distributed and target-node action logic is unchanged, including the guard
that keeps a hardware-specific build off the fan-out path. The split-button
popover loses its per-row anchoring because there are no rows; one pane, one
anchor.
Selection lives in ?backend=, preserving the ?target= scope rather than
clobbering it.
e2e: 139 passing across models-gallery, navigation, backends-management,
models-recommended-panel, nodes-per-node-backend-actions, page-render-smoke,
operations-strip and model-artifact-operation. The backends spec gains six
split-view tests; its three description-cell tests move onto the pane lede.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(ui): put the Host inventory on the split view
The last of the three surfaces, and the one that is not a catalog. Both tabs
had the same click-to-expand row, so the shell transfers; what does not
transfer is the zero state, because there is nothing to discover in your own
inventory.
With nothing selected the pane reports what is happening: how many models are
loaded, what failed, what has an update, and which models are holding VRAM
right now. Every number was already on the page. None of them had been
assembled into one statement, so "what is going on" was a question the tabs
could not answer however long you looked at them.
The rail buckets by state rather than capability - Running, Idle, Disabled for
models; Update available, Installed for backends - which is the opposite of the
galleries and deliberately so: nobody opens Host wondering which of their
models does vision. Entries carry a state stripe for the same reason.
Load and Stop are promoted out of the kebab, because that is what an operator
came for; the rest stays behind the menu rather than diluting it. Adopted,
pinned and alias badges follow the model into the pane: they are facts about
the thing, not about its state, and the rail line is spent on state.
Deliberately NOT done: folding the two tabs into one rail, as the mock had it.
It costs five URL parameters, the manage-tab localStorage key and the
stat-card shortcuts, all of which are live deep-links today. The tabs stay as
the group selector; merging them is a follow-up with its own migration.
e2e: full suite 355 passing. New host-split-view spec; alias-template,
manage-logs-link, manage-action-menu-position and model-editor-back-nav move
off `.table` and the row kebab onto the rail and the pane.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* polish(ui): accessibility and consistency pass over the three split views
Findings from a pass over what the previous four commits actually shipped,
rather than what they were supposed to.
The rail was not a listbox. ARIA lets a listbox contain options and groups,
and nothing else, but each group's collapse control is a button that has to
sit inside the scroller with the entries it folds. It is now a labelled group
of buttons, which is the honest description; selection is announced with
aria-current and the arrow keys are unaffected.
Every entry was its own tab stop, so tabbing past a forty-entry rail to reach
the pane took forty keystrokes. Roving tabindex makes the rail one stop, and
arrowing now moves focus with the selection instead of leaving it behind on an
entry Tab can no longer reach.
The rail rounds its corners with overflow:hidden, which was clipping the focus
ring off the first and last entries entirely. Inset outlines fix it.
A 30px row is fine under a mouse and too small under a thumb, so coarse
pointers get a 44px target without costing density on a desktop.
One slot said three different things: "9 models loaded" on Discover, "12
loaded" on Backends, "3 of 9" on Host. All three lists are a page of a larger
set, so all three now say so the same way.
Also removed: an emptyLabel prop on EntityRail that nothing passed, its dead
CSS rule, and MODELS_COLSPAN and ResourceRowDesc, which died with the tables.
e2e: full suite 355 passing.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(ui): correct three defects only a real gallery exposed
Running the branch against a live instance with 1,595 models and 1,017
backends, rather than against mocked fixtures, surfaced three things the e2e
suite could not.
Grouping did nothing. The rails matched on the use-case keys the filter chips
send (`chat`, `tts`, `transcript`), but those are a server-side vocabulary the
handler maps onto entries. What entries actually carry is free-form and
inconsistent: models come back tagged `llm`, `gguf`, `vision`, `coding`, and
backends `LLM`, `text-to-text`, `audio-transcription`. Nothing matched, so
every model landed in "Everything else" and the feature was decorative.
Grouping now lives in utils/entityGroups.js, shared by both galleries, matching
case-insensitively against the vocabulary the API really uses, with the entry's
backend as a fallback signal - a backend named `whisper` is a speech backend
whatever its tags say. Order is specific before general and that is
load-bearing: a vision model is tagged `llm` too, so testing text first would
swallow it.
The zero state claimed GPU memory on a machine with no GPU. The resources
endpoint reports system RAM in the same field when gpu_count is 0, so the hero
read "84.4 GB of GPU memory" next to the recommendations panel correctly
saying "No GPU detected". The number was never wrong, only its label; it now
says system memory unless a GPU is actually present.
The page title still said "Install Models" under a nav entry saying Discover.
Also: the keyboard test named the model it expected to arrive at, which made it
a hostage of the grouping table and broke the moment the buckets were fixed. It
now asserts that the selection moves and returns.
e2e: full suite 355 passing.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(ui): the filters and the rail were fighting over the same job
Four things you find odd on Discover, and they turn out to be one mistake seen
from four sides.
The rail grouped the current page. The listing is paginated at nine rows, so
those bucket headers described nine entries out of 1,595, and turning a page
reshuffled the sections under the reader. The structure was never stable
because it was computed over the wrong set.
The chips were redundant for the same reason, seen from the other side. They
send tag= and filter all 1,595 server-side. The rail grouped nine of them
client-side by the same axis. Two controls for one job, and the weaker one was
the one this branch added, so it goes. Grouping stays only on Host, where the
list is complete, local, and bucketed by state rather than capability.
The search bar felt odd because it sat in a full-width band while the thing it
narrowed was a 290px rail below and to the left. The whole band now lives in
the rail column: search, backend, use cases, refinements, then the list it
narrows. One column to say what you want, one to show what you got. Nineteen
chips do not fit at that width, so they fold into a disclosure that states the
selection. A disclosure and not a popover, deliberately: picking use cases is
multi-select and interleaves with the backend select and the toggles below,
and a popover dismisses itself the moment you touch either.
The header held two counts and two buttons at arm's length from all of it. The
counts were the third statement of the same number on one screen, after the
rail's "9 of 1,247" and the pane's own headline, so they go. The buttons move
into the pane's zero state, which is the surface that answers "what do I do
here".
Also: the two first-run empty states wore .loading-center, which is
display:flex in the default row direction because it exists to centre one
spinner. With four children that put the icon, the heading, the sentence and
the buttons on a single line with no gap. They are now a proper full-height
empty state.
e2e: full suite 353 passing. Grouping tests are replaced by ones asserting the
rail stays flat; chip tests open the disclosure first; two filter-layout tests
that asserted the old three-band arrangement now assert the column.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* polish(ui): make Discover a full-height view, group the chips, name the refinements
Four things, all of them the same complaint: the page read as a document with
controls scattered on it rather than as one view.
The header is fused. A title block with its own padding, a subtitle and two
counts made the split view look like an attachment to a document that happened
to sit below it. It is now a slim bar carrying the title, the count and the two
page-level actions, and the split fills the rest of the window. Rail and pane
scroll independently, so the filters and the pane's headline stay put while a
long list moves under them.
The chips group. Nineteen in a flat row is a lot to scan even behind a
disclosure, and they already belong to the four families the rest of the UI
speaks, so they are bucketed by those. "All" sits on its own above them without
a heading, because it is a reset rather than a use case.
The refinements stop looking dumped. When the band became a column they were
three controls left where they landed; they now read as a named section with
one control per row.
The zero state suggests again. It had decayed into a "Browsing / 9 of 1,247 /
select a model" line that restated the count for the third time on one screen.
It now offers the four use cases as tiles that set the filter, which is the
shelf idea from the mock without inventing curation or paying for a second
fetch.
Two bugs found by looking at it rather than at the tests: the disclosure was
clamped to 190px, which cut it off partway through its third section so two of
the five never appeared at all; and the creation actions rendered twice, once
in the new bar and once in the pane hero a few pixels away.
e2e: full suite 353 passing. The chip-row test now holds its contract across
the per-family rows rather than a single one, and additionally asserts every
family is present and non-empty.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(ui): pin the split view's height so a long detail scrolls the pane
Selecting a model with a long description grew the whole page and dragged the
rail down with it, which is the opposite of what "full height" was supposed to
buy.
The flex chain was right and the ceiling was missing. .app-layout and
.main-content are min-height:100dvh, which is a floor: flex distributes free
space but nothing caps growth, so a pane taller than the viewport expanded the
column, the document scrolled, and the rail stretched to match. height:100% on
the pane then resolved against an auto-height parent and did nothing.
The chat route already solves this by pinning .main-content to 100dvh. The
same treatment now applies to any route containing a .page--app, selected with
:has() so the shell does not have to learn which pages happen to be split
views. Below the stacking breakpoint the pin is lifted, because two stacked
halves in two short scrollers is worse than a page that scrolls.
Measured on a live instance: document height stays at the viewport across
selection (950px either side) and the pane overflows internally instead.
Adds discover-height.spec.js, which asserts the page height and the rail height
are unchanged by selection and that the pane is the thing that scrolls. The
existing specs could not have caught this: they mock short descriptions, and
the bug only appears when the pane has more content than the viewport holds.
e2e: full suite 355 passing.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(ui): give Backends and Host the full-height view, and fix the Update button
Backends now matches Discover: the header fuses into a slim bar carrying the
title, the count and the page-level actions, the filters move into the rail
column where they narrow the rail and nothing else, and the split fills the
window. Its seven chips fit at rail width, so unlike Discover's nineteen they
need no disclosure. Host gets the bar and the height; its resource monitor,
summary cards and tabs stay above the split, because those are read once while
the rail and the pane are worked in.
Two things the height change surfaced.
The console layout is a flex row with align-items:flex-start, so its body sizes
to content. Right for the pages it was built for, wrong for a split view, which
needs a ceiling to scroll inside: without it the Backends rail ran past the
viewport and over the footer. Pinned with :has() so only split-view routes are
affected.
The filters vanished when nothing matched. Both galleries swapped the whole
shell for an empty state, which took the search box and the chips with it, so
the page said "try adjusting your search or filters" while offering neither.
The shell now stays and the empty state moves into the pane.
Also fixes the Update control on Host, which had no className at all and
rendered as bare text, next to a status span that had picked up btn classes and
two copies of `fas` and so rendered as a button you cannot press. They have
swapped appearances back.
e2e: full suite 355 passing. The render-smoke selector learns .view-bar__title,
since the pages it checks no longer all use PageHeader.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(ui): keep the view mounted while searching, and bring rail grouping back
Searching replaced the whole view with a loader. The search box lives in the
rail column, so every debounced refetch unmounted the field being typed into
and dropped its focus with it. The list, the filters and the pane went too.
The shell now stays and the rail says it is busy: a sweep bar under its header
and the stale list dimmed, so the eye knows the answer is being replaced
without losing its place. A cold start still gets the skeleton, because there
is nothing to keep.
The condition for that is "nothing has loaded yet", not "the list is empty".
Those differ exactly when someone is editing a query that matched nothing, and
getting it wrong there would unmount the view on the keystroke after a
no-results search - the worst possible moment.
Grouping comes back on both galleries. It was removed because nine rows could
not fill five buckets, so a page turn rebuilt the rail's whole structure. That
was a symptom of the page size rather than of grouping: the rail now asks for
30 rows instead of 9 (Backends 60 instead of 21), which is enough for the
sections to read as structure and turns five times fewer pages. The order of
the sections is fixed, so what changes between pages is membership, not
arrangement.
Grouped while browsing, flat while searching, as before: once a term is typed
the buckets stand between the reader and the answer.
Also gives GalleryLoader a class and a testid instead of six inline style
declarations on a bare div, which is why nothing could select it.
e2e: full suite 359 passing, including a new spec asserting the search box
keeps its focus and its value across a refetch, and that a cold start still
shows the skeleton.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* perf(gallery): stop invalidating the VRAM estimate caches on every request
Searching or turning a page felt slow. It was not the search and not the
listing: /api/models answers in 3-9ms. It was the VRAM estimate, which the
gallery asks for once per row, and which took ~2.3s every single time however
often the same model was asked about.
pkg/vram already caches what makes that expensive - the remote content-length
probes, the GGUF metadata reads and the HF repo sizes. Those caches key on a
gallery generation counter, and AvailableGalleryModelsCached triggered a
background refresh on every call, with each refresh bumping the counter. One
page view is one listing request plus thirty estimate requests, each of which
re-read the gallery and started another refresh, so the generation moved
constantly and every cache entry was stale before it could ever be read. The
caches were dead in production.
Three changes, each doing one thing:
A refresh interval. The cached list is still served immediately; this only
decides how often re-fetching from upstream is worth starting. Five minutes,
as a package variable so tests can drive it without waiting.
A generation bump only when the gallery actually changed. An unchanged gallery
re-fetched on schedule must not throw away work that is still valid, which is
the difference between an estimate costing nothing and costing a network round
trip.
A separate "loaded" flag. The cache engaged on `cached != nil`, so a gallery
that legitimately holds nothing read as never-loaded and took the blocking path
on every call, bumping the generation each time. Found by the test for the
interval, which could not pass while this was true.
Measured against a live instance with 1,595 models:
one estimate, repeated 2.3s -> 2ms
a page of 30, in parallel 10s -> 0.04s
A first, genuinely unseen model still costs its remote probe. That is inherent;
what changed is that it is now paid once per model per gallery version rather
than once per request.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* perf(gallery): warm VRAM estimates at startup, and stop the UI waiting on them
Two halves of the same complaint: the gallery stalls on VRAM estimation.
Server side, the estimates are now warmed in the background at startup.
Estimating an entry nobody has asked about costs a remote probe of its weight
files, and the gallery needs one per row, so the first visitor was paying for
the whole page. The warm-up walks the gallery in the order the UI lists it, so
the first page is ready before anyone reaches it.
It is bounded and it never blocks: 300 entries at 4 at a time by default, on
its own goroutine, stopping with the server's context. Warming the whole
gallery would be thousands of probes on every boot, which is rude to the
upstream and slow to finish; warming nothing leaves the first page paying two
seconds a row. Anything past the limit still warms itself on first view.
LOCALAI_VRAM_WARM_LIMIT=0 turns it off for an air-gapped host,
LOCALAI_VRAM_WARM_CONCURRENCY=1 slows it for a metered link.
Client side, the page no longer waits on estimates it does not need yet. It
fired one request per row at once; a browser allows about six connections per
host, so thirty estimates took every slot and the request behind a click - the
variant list, an install - queued behind work nobody asked for. That is the
freeze: the list was already usable, and the UI was busy fetching sizes. Four
at a time leaves room for the interactive request to overtake, and a row whose
estimate is still in flight says "sizing…" rather than leaving a blank where a
number will appear.
buildEstimateInput moves to core/gallery as EstimateInput, since the handler
and the warmer both need it.
Measured against 1,595 models, from a cold boot:
page 1, 30 estimates in parallel 10s -> 0.04s
full warm-up (299 of 300 entries) 3m, in the background
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* chore: untrack data/.local_user_id and ignore the runtime data dir
`local-ai run` writes its instance state under ./data when started from the
repo root, which is exactly what a contributor testing a build does. The
identity file ended up committed on this branch by a `git add -A` while
verifying the gallery changes against a live instance.
Anchored, so it matches the runtime directory at the repo root and not a
`data` directory nested inside some package.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Add the recommended Q4_K_M build and an MTP-enabled variant with the shared vision projector. Tag the existing Qwythos 9B MTP entry so serving-feature ranking recognizes it.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* chore(deps): bump cogito to v0.11 ahead of the nib harness
nib is the agent harness that becomes 'local-ai chat'. It requires cogito
v0.11, so pull that bump forward on its own: minimal version selection would
apply it to LocalAI anyway, and both repos use cogito and cogito/clients.
Landing it separately keeps the harness change reviewable.
nib itself is not pinned yet. Nothing in LocalAI imports it, and 'go mod
tidy' runs as a goreleaser before-hook in CI, so an unimported require line
does not survive. It lands with its first importer.
No LocalAI call site needed a change. Both cogito.WithMaxAttempts callers
guard the argument above zero, so v0.11's new clamp is unreachable, and
LocalAI's Multimedia values implement only URL(), so v0.11's new
TypedMultimedia routing treats them as images exactly as v0.10 did.
Binary size (cmd/local-ai): 200,301,381 -> 200,336,045 bytes (+34,664).
A throwaway probe that links nib measured 201,042,243 bytes (+740,862 over
the pre-change baseline).
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): resolve and seed the agent state directory
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): write the agent config atomically and tighten its modes
Replacing config.yaml in place truncated it first, so an interrupted write
would have destroyed the api_key nib keeps in the same file. Stage through a
sibling temp file and rename over the target instead, and match nib's 0700
directory mode.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): probe the endpoint and classify failures
Probe lists what a LocalAI endpoint advertises and separates the two
failures that need different advice: nothing listening, and rejected
credentials.
go-openai reports a rejected key as one of two concrete types depending
on the error body, and both occur against a real LocalAI. The normal
error handler sends an OpenAI error envelope, which arrives as
*openai.APIError; the opaque-errors handler replies with a bare status
and no body, which arrives as *openai.RequestError. Classifying on only
one of them misses half the cases, so the status is read from either.
A cancelled probe is not reported as an unreachable server, because it
learned nothing about the endpoint, and neither is a reply that could
not be parsed, because something did answer. Both would otherwise send
the user off to start a server that may already be running.
The model list is returned verbatim and in server order. LocalAI lists
whatever it finds in the models directory, including stray archives and
dotfiles, and deciding which advertised ids are real belongs to whoever
presents them.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): resolve the model from flag, config, or the server
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(chat): pin that model resolution sorts a copy of the caller's slice
The sort spec asserted only on what the chooser was offered, so replacing the
defensive copy with an in-place sort of req.Available still passed all 37
specs. Assert the input slice's order after the call, so the guarantee cannot
be dropped silently by a later refactor.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): offer to start a server when none is reachable
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): bound the server wait and pin readiness and stop semantics
Set cmd.WaitDelay so a backend subprocess holding the child's stderr pipe
cannot block cmd.Wait forever, which would leave exited unclosed, burn the
whole shutdown grace on a clean exit, and leak the waiter goroutine.
Two test gaps closed alongside it: the readiness spec now counts polls, so
treating 503 as ready is observable, and Stop's single-interrupt contract is
pinned by giving StartedServer interrupt/kill hooks that a spec can count.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactor(chat): drive Stop through one process interface, hide exec plumbing
Two independent interrupt/kill func fields plus a nil check admitted wirings
no test could distinguish: the pair swapped, so a SIGKILL would strand the
backends SIGINT exists to let local-ai run clean up, or kill left nil, so a
wedged server never escalates. One two-method interface that *os.Process
already satisfies leaves nothing to swap and nothing to nil.
Also translate exec.ErrWaitDelay, whose text names an os/exec struct field,
into what the user can act on. os/exec only substitutes that sentinel when the
process exited without an error of its own, so no exit status is swallowed.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): replace the REPL with the built-in agent
local-ai chat is now the nib agent harness compiled into the binary: tool
use behind an approval gate, sub-agents, MCP, plugins, and skills, all
auto-configured against the local server.
The REPL goes with it. Its model listing and its 401 classifier were
duplicates of the ones Probe now owns, and the classifier was the version
that misreads a bare 401 with no OpenAI error envelope, so keeping either
would leave the package with two divergent answers to the same question.
github.com/mudler/nib lands in go.mod in this commit rather than earlier:
go mod tidy runs as a goreleaser before-hook on every PR, so a require
line with no importer is stripped before it reaches CI.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactor(chat): split the pre-agent phase out of Run and pin it
Everything before the handoff is testable and nothing after it is: once
app.Run owns the terminal there is no seam left. prepare draws that line,
takes interactivity as a parameter so the prompts can be driven over a
pipe, and hands Run the state dir, the model, and any server it started.
The questions move onto one prompter that owns its buffered reader. A
fresh bufio.Reader per question reads ahead and discards what it buffered,
so the model choice typed behind an answer to "start a server?" was lost
and the next question saw EOF.
choose answers with a list index and refuses an empty offer, so a value
that was never on the list cannot reach ResolveModel, which persists it
and starts every later run against it.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): bound each server check with a deadline
Nothing bounded the model listing, so pointing chat at an address that
accepts the connection and then never replies left the user with no
output and no offer to start a server.
The budget is context.WithTimeout rather than a cancel plus a timer.
Probe deliberately refuses to call an endpoint unreachable on a
context.Canceled, since a caller who gave up learned nothing about the
server, and only honours a deadline. A cancel-based budget therefore
expires as the one error that suppresses ErrUnreachable, exactly for the
hung servers the offer exists to rescue.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): tell the user when their model choice cannot be saved
The choice is meant to be asked for once. When saving it fails the user
is silently asked again on the next run, and the only trace was an
xlog.Warn: the agent runs at log level error, and a --log-level=error run
swallows it entirely.
ModelRequest gains Notify for exactly this class of problem, one that is
worth telling the user about but not worth failing over, and the chat
wiring points it at the same writer the question was asked on.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): stop a session's server when the process is signalled
A server started for the session is stopped by a deferred call, and a
signal skips deferred calls: a SIGTERM between the spawn and the exit
left 'local-ai run' reparented to init with nothing left that knew to
shut it down. Ctrl+C was already safe, but only incidentally, because the
child shares this process' foreground process group.
A signal handler rather than Pdeathsig on the child. Pdeathsig is
Linux-only and, in Go, is delivered when the OS thread that forked exits
rather than when the process does, so it can fire on a healthy parent.
Setpgid would break the Ctrl+C that works today by taking the child out
of the foreground group.
SIGHUP joins SIGINT and SIGTERM: a terminal program whose terminal is
gone has nobody left to talk to. The same context is what cancels the
agent, which nib leaves to its embedder.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): only skip the server checks for work that stays local
Two argument shapes were classified wrongly. Every 'mcp ...' invocation
counted as management, so 'local-ai chat mcp --stdio', which serves the
agent over MCP and needs a model like any other session, was handed an
empty one. And --init, whose shell snippet a user pastes into an rc file
long before any server exists, went the other way: it demanded a running
server to print a static string.
The mcp split is asked of nib's own IsMCPManageSubcommand rather than
restated here, so a verb added upstream cannot drift out of this list.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): exit with the agent's status instead of reporting it twice
nib writes what went wrong to stderr and returns nothing but an exit
code, so returning that error unchanged had main log "Error running the
application error=exit status 1" underneath the message the user had just
read. The refusal to render the full-screen interface into a pipe is the
one they meet in practice: it names --cli, and burying that hides the fix.
ExitCodeError says "already reported, exit with this status". main
honours it and prints nothing more, so a piped or redirected chat still
fails a script the way it should.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* style(chat): route interactive chatter through one writer helper
The prompts and notices all write to a terminal, where a failed write is
not worth failing the session over and the read that follows the question
reports the real problem. say says that once instead of five discarded
error returns.
The command's one-line help comes along: chat is no longer "an
interactive chat session".
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(chat): record why the agent gets this process' streams
Injecting them is what makes nib refuse to draw its full-screen interface
into a pipe and name --cli, instead of rendering onto a terminal the
caller may not own. The tradeoff is worth stating where the wiring is.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): stop the session's server on cancellation, not on the way out
The deferred Stop is reached only if the agent returns, and cancelling
the context does not make it: nib hands the TUI to bubbletea without the
context, so what actually unwinds a running session today is bubbletea's
own SIGINT and SIGTERM handler. SIGHUP has no such backstop, and
registering for it removed the default disposition that used to end the
process outright, so kill -HUP left a live TUI with a cancelled context
and the started server still running.
runSession watches the context alongside the agent and stops the server
the moment it is cancelled, so the guarantee no longer depends on what
the agent does with cancellation. Stop is idempotent, so the deferred
call stays correct and free.
The doc comment on shutdownContext described the mechanism it was
supposed to work by rather than the one that does. Corrected, bubbletea's
handler included.
ResolveModel now checks the chooser's answer against what it offered.
The shipped chooser answers by list index and cannot be wrong, but
ModelChooser is exported, the answer is persisted, and every later run
starts against it, so the invariant belongs at the consumer.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(chat): bump nib to v0.5.1
v0.5.1 carries four fixes that matter to 'local-ai chat':
- --init now names the embedder's command, so the emitted widget invokes
'local-ai chat' rather than a bare 'nib' the user does not have.
- A piped CLI session that succeeds exits 0 instead of failing with EOF.
- EOF at a tool-approval prompt denies the call rather than approving it,
and the session exits 3 (app.ExitCodeApprovalNoInput) so a script can tell
"answered" from "refused to act" without reading stdout. Read-only tools
are unaffected and still run. ExitStatus already unwraps app.ExitError,
so the code propagates with no change here.
- RunTUI passes the context to bubbletea and gives up bubbletea's own signal
handler, which makes shutdownContext the single owner of the signal and
stops a SIGHUP leaving a wedged TUI behind.
Verified against a live server on 127.0.0.1:8080: the three --init shells,
a piped prompt exiting 0, a denied 'touch' that left no file and exited 3,
a read-only 'ls' that still ran and exited 0, and a SIGHUP that unwound a
TUI running under a pty.
Two comment blocks in run.go described the old TUI behavior and are now
wrong, so they are corrected in the same change. No behavior change: both
shutdownContext and runSession are untouched, and stopping the server on
cancellation is still worth keeping independent of how promptly nib unwinds.
One known gap, not addressed here. The widget --init now emits runs
'output=$(local-ai chat --height 50%)', and runAgent injects Stdout
unconditionally, so under $(...) nib refuses the TUI for a non-terminal
stream. This is the cost the runAgent comment already anticipated, now that
the snippets no longer hardcode standalone nib. Ctrl+Space should not be
documented until that is decided.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): let nib own stdout, so the Ctrl+Space widget works
The widget 'local-ai chat --init' emits runs
'output=$(local-ai chat --height 50%)', which puts a pipe on stdout by
construction. runAgent injected os.Stdout unconditionally, and nib refuses
every mode but --cli when a stream it was handed is not a terminal, so
Ctrl+Space printed "Re-run with --cli to use the injected streams" and
inserted nothing. Verified against a pty before and after.
nib reads a nil stream as "not injected" and falls back to the process
stream, which is how an embedder asks for nib's own behavior. That is what
stdout needs: the interface renders on /dev/tty but writes the chosen
command to stdout even when stdout is a pipe, and that write is the whole
of the shell-capture idiom.
Stdin is deliberately left injected. A piped or redirected stdin really is
ignored by the interface, so the refusal is the honest answer there, and it
is the one users meet: 'echo q | local-ai chat' still says to re-run with
--cli, once, exit 1. Nilling stdin the way stdout is nilled would delete
that silently. Stderr is not gated by nib at all and is unchanged.
One case does change and cannot be kept: 'local-ai chat > out.txt' from a
terminal no longer refuses, because it is indistinguishable from the
widget. It renders on /dev/tty and writes the capture line to the file,
which is what standalone nib does.
The app.Options literal moves into agentOptions so the decision is
reachable from a spec rather than being a detail of a function that takes
the terminal. Both sides of the asymmetry are pinned: reinstating
'Stdout: opts.Out' fails "hands nib nothing for the process stdout", and
nilling stdin fails "hands the process stdin over".
Also rewrites the last comments describing the pre-v0.5.1 behavior.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(chat): say what the stream refusal actually keys on
Two comments still called it the refusal to render the interface "into a
pipe". That was true when both stdin and stdout were injected, but a pipe on
stdout no longer refuses, so the wording now points at precisely the case
that was un-refused to make Ctrl+Space work. Only a stdin that cannot be
read triggers it, and both comments now say so and name the command a user
meets it with, 'echo q | local-ai chat'.
The agentOptions doc also said a "file a caller chose" stays injected and
refused, which reads as though 'local-ai chat > out.txt' still refuses. It
does not: a shell redirect arrives as os.Stdout and is nil-ed like the
widget's pipe, because the two differ only in being a regular file rather
than a FIFO and nib's gate does not look at that. What stays injected is a
writer an in-process caller chose for itself. Says that now, in the doc and
in the spec comment that had the same ambiguity.
Comments only. No behavior change.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: local-ai chat is now the built-in terminal agent
`local-ai chat` was a plain chat prompt and is now an agent that runs
shell commands behind an approval gate, so the pages that described a
REPL were wrong rather than merely thin.
Adds a Terminal agent feature page at /features/terminal-agent covering
the approval gate, piped runs and their exit codes, Ctrl+Space, model
resolution, state directory, and the pass-through management commands
(including the `--yes` caveat that leaves a plugin installed but
disabled in a script).
The three-way "looking for something else" notice becomes four-way and
moves into an agentic-routing shortcode. Four hand-kept copies of the
same paragraph is what produced the drift the new page would otherwise
have added to; the shortcode takes `current=` so each page still marks
itself, and errors the build on a name that is not one of the four.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* website: the agent is in the binary, not a second install
The nib section sold a separate tool you also install, with a GitHub
link as the only way in, which is now the wrong order: the agent ships
compiled into local-ai, and the standalone binary is the second reason
to care rather than the first.
Leads with `local-ai chat`, keeps nib as the SSH-anywhere story, and
adds a docs CTA pointing at the new Terminal agent page. id="nib" is
left alone because localai.io/#nib is linked from outside.
The two credits on the demo clip named nib as the thing that drove the
machine; they now credit the agent in LocalAI, which is the same agent.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* website: fix the exit keys, the plugin warning, and the redirect gap
Three claims on the chat-agent pages that the code does not back.
try-it-out told readers to press Ctrl+D. nib has no Ctrl+D handler: the
full-screen interface quits on Esc or Ctrl+C, and Ctrl+D is only an exit
in --cli, where it arrives as ordinary tty EOF. That sentence had
replaced the removed /exit and /quit text, so the page was left with no
working way to leave a session. Document both modes, since they differ.
The plugin warning said nothing tells you the install stopped short. It
does: the command prints that the plugin was left disabled. What it does
not do is say so in its exit code, which is 0 either way. That is the
part a script cannot work around, and it is the reason to pass --yes.
Overstating it in the paragraph that gives the advice only makes the
advice easier to dismiss.
Redirecting stdout no longer refuses; the interface goes to /dev/tty and
only the yanked command reaches the file. It is what lets the Ctrl+Space
widget capture a command at all, since a redirect and out=$(...) are the
same thing to the stream gate. It was documented nowhere. A non-terminal
stdin is still refused, and the new text says which of the two it is.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): make the CLI flags outrank the agent config file
local-ai chat routed --endpoint, --model, --api-key, --trace-dir and --yolo
through nib's app.Options.Defaults. Defaults are seeds: they sit beneath the
config file, so the file silently undoes them. That made the flags accepted and
inert, and not in an edge case, since EnsureStateDir writes base_url on the
first run and the interactive picker writes model, so from the second run on
the file carried a value for both.
Observed against a live server: with base_url: http://127.0.0.1:9999/v1 in the
config and --endpoint http://127.0.0.1:8080 on the command line, the probe hit
8080 and every agent turn posted to 9999. With model: gemma-4-e2b-it-qat-q4_0
in the config, --model lfm2.5-8b-a1b was ignored on the wire.
nib v0.6.0 adds app.Options.Overrides, applied above the config file and above
the bare environment block. Move the whole block there: all five values are
decisions this invocation already made on the user's behalf, and a flag the
config file can undo is not a flag. Nothing is left in Defaults, because
LocalAI's one genuine seed, the initial base_url, is written into the config
file by EnsureStateDir rather than handed to nib.
Two limits come with the channel and are documented on agentOptions rather than
worked around. An override can only raise a field, since nib cannot tell "set
to the zero value" from "not set", so --yolo can turn approval off but nothing
on the command line turns it back on over an approval_mode: auto in the file.
And nib's own NIB_TRACE_DIR and NIB_YOLO are resolved after the config load and
still outrank these, deliberately, upstream.
The existing spec pinned that the right values reach app.Options, which they
always did, which is exactly why it could not see nib discarding them. The new
specs resolve the config the way app.Run resolves it, against a real config
file that disagrees with every flag, and one asserts Defaults stays empty.
docs/content/features/terminal-agent.md already documented --model as winning
over the saved model; that was false before this change and is true now, so no
docs edit was needed.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): document intentional config file read
Assisted-by: Codex:gpt-5 [gosec]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The post was written against the first draft of the release notes, when the
cycle stood at 214 PRs over thirteen days. It closed at 321 PRs over eighteen
days, and three of the larger user-facing changes landed after it was written.
- Correct the counts throughout: 321 PRs, eighteen days, 24 contributors
(11 first-time), gallery 1,221 to 1,505.
- Add sections for the three new capabilities: 3D generation as a modality
(Generate3D, FLAG_3D, /v1/3d/generations, trellis2cpp), audio.cpp serving
six audio endpoints from one process, and the operations bar becoming the
Activity page.
- Cover the two further hardening fixes (tar hardlink escape, cyclic $ref
stack overflow) alongside the TRL one.
- Note the Valkey store, systemd socket activation, persistent trace history,
in-place chat edits, the self-contained SYCL backend and the site split.
- Group the new-engine sections together rather than splitting them across
the operational ones.
Embeds the existing vllm-race and magpie clips, and adds a 3D generation clip
cut from the demo recording to the conventions in .agents/preparing-a-release.md
(no audio track, 14s, named for the feature). blog.css styled figure img but
not figure video, so a clip in a post rendered outside the card; both selectors
now share the rule.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
tests-qwen3-tts-cpp has been failing on master since 2026-07-31. The suite
loads every component fine and then stops: TTS() never returns from the
native call, so a job that takes ~5 minutes runs into the 20 minute Go test
timeout instead.
goroutine 74 [syscall, 19 minutes]:
github.com/ebitengine/purego.RegisterFunc.func4
qwen3-tts-cpp.(*Qwen3TtsCpp).TTS goqwen3ttscpp.go:154
qwen3-tts-cpp.init.func2.4 e2e_test.go:90
Not a flake: reproduced on master and again on an explicit re-run.
Bisected across this cycle's eight qwentts.cpp bumps by their own check:
10832, 10850, 10902, 10964, 11006, 11039 and 11127 all pass in ~5 minutes;
11241 (abab6b3) fails at 1h58m. That PR was merged with this check already
red, which is how the hang reached master.
35ebe537..abab6b3 is three upstream commits, and the only functional one is
26dd8adb, "predictor: unroll the frame into one cgraph and sample in standard
ops", which is consistent with a generation loop that never reaches its stop
condition.
Hold the pin at the last known-good commit. The bump entry is commented out
rather than left in place, because it tracks upstream master and would put
the hang straight back on the next nightly run. Both spots carry a pointer to
the other so the hold is discoverable, and restoring it is uncommenting four
lines once upstream is fixed.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Select the CPU_ALL_VARIANTS target for x86 GPU images so partial offload uses runtime-selected host kernels. Keep GPU arm64 builds on the portable fallback until their toolchains consistently provide gcc-14.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): drop merged MiniMax-M3 patch
The bumped llama.cpp revision includes the MiniMax-M3 parser and template detection, so the carried patch now rejects during backend preparation. Remove the obsolete patch while retaining the independent score-task patch.
Assisted-by: Codex:gpt-5 [systematic-debugging]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add the new 9B Fara computer-use model alongside its existing 27B sibling, with Q4_K_M and Q8_0 llama.cpp variants plus the required vision projector.
Assisted-by: Codex:gpt-5 [web]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Build the runtime CPU variant set alongside x86 GPU backends so partial offload uses the host's SIMD kernels instead of the scalar fallback. Keep arm64 GPU images on the portable binary until their builders consistently provide gcc-14.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(utils): reject tar hardlinks that escape the extraction root
ExtractArchive pre-scans archive members and rejects symlinks, but tar
hardlink entries carry a regular file mode and so pass that check.
Header.Linkname was never validated, so an archive could create a link
to a path outside the destination directory.
Validate Linkname with the same path check already applied to member
names. Hardlinks that resolve inside the extraction root still extract,
so ordinary archives are unaffected.
pkg/oci/image.go already resolves tar.TypeLink targets before using
them; this brings the archive extraction path in line with it.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Zelys-DFKH <zelys@dfkhelper.com>
* test(utils): cover hardlink overwrite and in-root hardlinks
The existing hardlink test names a link target two levels above the
extraction root, so its final assertion checked a path the link never
resolved to and could not fail. Point the target one level up instead,
at the path that assertion already names.
Add two cases. The first uses a .tar.gz, where ExtractArchive binds a
Tar config with OverwriteExisting set, and follows the link entry with a
regular entry of the same name. Before the fix that pair linked to a
file outside the root and then truncated it through the link, which the
plain .tar case does not reach. The second extracts a hardlink whose
target is an earlier member of the same archive, covering the claim that
ordinary archives are unaffected.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Zelys-DFKH <zelys@dfkhelper.com>
---------
Signed-off-by: Zelys-DFKH <zelys@dfkhelper.com>
Use the case-sensitive Hugging Face filenames and refresh the linked SHA256 values for both gallery variants.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Detect text-to-speech MLX repositories during model import and emit a TTS-ready mlx-audio configuration. Expose mlx-audio in the backend preference dropdown for repositories without complete metadata.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(sycl): make the intel llama.cpp backend self-contained on any host
The SYCL backend shipped an incomplete oneAPI runtime AND relied on a
host-provided GPU driver, so it only ran inside the build container. On a
bare host it died with "libze_loader.so.1 / libdnnl.so.3: cannot open
shared object file", and even with the host's Intel driver installed it
SIGSEGV'd during SYCL init when the host driver was built against a newer
glibc than the backend's bundled loader (rolling-release distros).
package_intel_libs now bundles the complete, coherent oneAPI runtime
(the missing MKL ILP64 / sycl_blas / tbb_thread + oneDNN + the dlopen'd
UR adapters, plus a sweep of the backend binaries' own direct deps) and
the Intel GPU userspace driver (libze_intel_gpu + libigdrcl + IGC + gmm)
with its OpenCL ICD manifest, mirroring how package_vulkan_libs bundles
Mesa. run.sh points the Level Zero and OpenCL loaders at the bundled
driver, and install-base-deps.sh installs it in the SYCL build image.
Bundling the driver is safe across kernels because it talks to the host
i915/xe via the stable DRM UAPI (unlike NVIDIA's kernel-locked
userspace).
Validated on Arch (glibc 2.43, i915): the backend loads and runs on an
Iris Xe with no host Intel packages installed.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
* fix(sycl): install a driver that exists, and let the user choose their own
The driver install added earlier in this branch asked apt for
intel-level-zero-gpu, which is not a package in Ubuntu 24.04. apt fails
outright on an unknown name, so neither driver was installed, nothing was there
to copy, and the images carried no driver at all.
It now comes from Intel's own repository, which has 25.18 for this Ubuntu
release, against 23.43 from late 2023 in the Ubuntu archive. The archive driver
does not know any card released since, so a machine with a recent Intel GPU
would end up carrying a driver that cannot drive it. Anything that goes wrong
during that install fails the build on purpose: an unreachable repository is a
passing problem that a retry fixes, while quietly carrying a different driver,
or none, is a difference nobody would notice until a user reports an idle GPU.
run.sh used to overwrite whatever driver the user had chosen. Level Zero uses
only the driver it is given, so on a machine with a card too new for the
carried driver, the GPU would go unused with no way back. Both that setting and
the OpenCL one are now left alone when already set, and the docs say how to
point a backend at the machine's own driver.
The OpenCL setting also used to be applied whenever the backend held a driver
list, even when the driver it named had not been copied, which leaves OpenCL
with nothing instead of falling back to the machine's own driver. It now
requires the copied driver to be present, and the packaging leaves out the list
entry of any driver it did not copy. The oneAPI images list a processor-only
OpenCL library, which was being carried with nothing behind it.
Two more corrections in the packaging. The scan for libraries a program is
linked against only looked at files named llama-cpp-*, so turboquant and bonsai,
which are also built for Intel GPUs, were left with the incomplete set of
libraries this branch set out to fix; it now looks at every program in the
directory. And a build that should carry a driver but ends up without one now
says so, which is what a stale prebuilt base image looks like: such a backend
still runs on a machine that has its own driver, so nothing fails and the only
other symptom is a user reporting an idle GPU.
Backends now also ask the driver to report how much graphics memory is free,
without which llama.cpp reads zero on an integrated GPU, since such a chip
shares the system memory instead of having its own. turboquant and bonsai get
the same run.sh handling as llama.cpp.
The driver is only carried by the builds that start through run.sh, because
run.sh is what points Level Zero and OpenCL at it. The Python backends for
Intel GPUs start differently and would never load it, so they keep using the
machine's own driver rather than carrying several hundred megabytes they cannot
use.
Checked in a container on Ubuntu 24.04: the install brings driver 25.18 with
the files where the packaging expects them, an unreachable repository fails the
build, and the copied set resolves on its own once the machine's Intel packages
are moved away.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
* fix(ci): rebuild every Linux backend when the GPU packaging script changes
scripts/build/package-gpu-libs.sh decides which GPU libraries end up inside an
image. The filter that builds the backend matrix listed it as an input of the
Python images only, so changing it rebuilt no Go and no C++ backend, even
though those run it from their own package.sh. A packaging fix aimed at the
Intel llama.cpp backend could merge and reach no image, which is the same
failure this rule was written to prevent.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
* fix(sycl): carry only the driver Level Zero uses, not the OpenCL one
llama.cpp reaches an Intel GPU through Level Zero, which hands the driver
programs that are already compiled and so needs only the back end of the
graphics compiler. The OpenCL driver can be handed source code instead, so it
needs the compiler's front end as well, and that arrives with its own copy of
clang. Carrying it cost about 139 MB in every backend built for Intel GPUs, and
took the carried set from 123 MB to 261 MB.
Nothing here takes that path. No LocalAI code selects an OpenCL device, each
backend image holds one backend, and the documentation never described OpenCL
as a way to run models: the only mentions are a stale clblas row in the
BUILD_TYPE table, for a llama.cpp backend that no longer exists and that no
build matrix entry uses, and the sycl-ls troubleshooting hint. Before this
branch the packaging carried the OpenCL loader and adapter but no driver, so
the path could not work in a released image either. There is nobody to keep
working.
The driver list that OpenCL reads is no longer carried, and run.sh no longer
sets OCL_ICD_VENDORS, so OpenCL inside a container keeps using whatever the
image provides rather than being pointed at a directory with no driver in it.
Checked in a container against the real 25.18 driver: the carried set is 123 MB
with nothing unresolved, and Level Zero still reports the GPU with the
machine's own Intel packages moved out of the way. Neither the Level Zero
driver nor the compiler back end names the front end or clang among the
libraries it opens by name, so the leaner set is complete for this path.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
---------
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
* feat(chat): edit saved conversation messages
Add inline edit, save, and cancel controls for stored user and assistant messages without triggering inference. Preserve structured message attachments and cancel edits when streaming starts.
Assisted-by: Codex:gpt-5
* test(chat): preserve seeded conversation on reload
The saved-message edit test reloads the page to verify persistence, but its init script was replacing localStorage with the original fixture on every navigation. Seed only an empty store so reloads exercise the data written by the application.
Assisted-by: Codex:gpt-5 [Codex]
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Direct repository-root users to the supported make docs target and document the equivalent direct Hugo invocation from docs/.
Fixes#10062
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The page reads as a list of features with nothing joining them, so two
things did not land.
The engines section never said these are the backends LocalAI loads. The
runtime section describes a core that pulls each engine in on demand, and
the engines section describes engines written from scratch, and nothing on
the page connected the two sentences. Readers were taking parakeet.cpp and
the rest for unrelated side projects by the same people. The lede now says
whose backends they are before it says anything else.
APEX was used as a known term on first appearance, in a section that opened
onto a benchmark table. Nothing said what it is or why it follows the
engines. It now opens by placing itself in the stack: the engine decides how
fast a model runs, the weights decide whether it runs at all, and APEX is
the second of those. Then the numbers.
Also drops "Most backends wrap somebody else's engine. These do not", which
is the machine-written antithesis shape, and fixes a list that broke its own
parallel halfway through.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m]
The star, fork, contributor and release counts were typed into the templates
by hand, so they only moved when somebody remembered. They had already
drifted: stars read 48,042 against 48,067, forks 4,314 against 4,320, and
contributors 224 against 225.
They move to website/data/stats.yaml, which .github/ci/refresh-site-counters.sh
rewrites from the GitHub API, run weekly by a new workflow. The contributors
and releases endpoints never report a total, so the script asks for one item
per page and reads the count out of the Link header. It refuses to write a
zero or a non-number, which is what a rate-limited or failed call looks like,
and the workflow commits only when a number actually moved. The Discord count
has no API behind it, so the script reads the existing value back and carries
it through.
The engine count was wrong in a second way. The hero said 18, the section
heading said "Eighteen engines", the timeline said "Nineteen engines of our
own", and the /engines/ page derived 19 from the data file. All of them now
derive from that same file, so they cannot disagree again.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m]
Add Q4_K_M and Q8_0 llama.cpp entries for the newly released Qwopus3.6-27B Fusion reasoning and coding merge, with MTP enabled.
Assisted-by: Codex:gpt-5 [Hugging Face API]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The timeline set six 15rem columns in a flex row with overflow-x:auto, which
needs 90rem and so scrolled sideways on any normal laptop. It is a wrapping
grid now, and the rule that carries the dots moves from the container onto
each item so a wrapped row still gets a line above it. Column gap is zero and
the items carry their own right padding, so the rule stays continuous.
Integrations move from a card grid to a reel. Any single integration is a weak
signal and the whole moving line is the strong one, so the count is doing the
argument. It pauses on hover and on keyboard focus, since the names are links.
The list grows from 8 to 26: Open WebUI, Dify, LibreChat, RAGFlow, Continue,
big-AGI, Nextcloud, Frigate, promptfoo, Mods, TypingMind, baibot, k8sgpt-
operator and others. Each was admitted only after opening that project's own
repository or docs and reading the line that names LocalAI. The ones that
failed that test are listed in the data file so nobody re-adds them.
The blog cards were hand-written, which is how one of them came to advertise
"Porting vLLM to C++", a post that does not exist, and how all three linked to
the blog index instead of an article. They range over the posts now.
The section intro used the "a changelog tells you what moved, these posts show
you what it does" shape, which is the standard machine-written antithesis. It
states what the posts contain instead, including the perplexity regression
that APEX costs, because publishing the price is the actual claim.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m]
The band led with three sentences of hedging and printed a commit count
next to each employer, so a one-commit entry beside a large name read as
weakness rather than as the modest, true claim it was. It now opens on the
contributor count, sets the employers as a sentence instead of a pill wall,
and keeps the caveat to one line. The counts stay in ecosystem.yaml, since
they are the provenance for the list and anyone re-checking it needs them.
Two "coverage" cards were not about this project. The modelslab.com piece
reviews Frikallo/parakeet.cpp, an unrelated project of the same name, and
the snailtext.app benchmark measures Parakeet through ONNX Runtime without
mentioning LocalAI at all. Both are removed, along with the contributor
card that duplicated the band's opening line.
Press was four posts from one vendor, which read as the whole of the
coverage rather than one enthusiastic outlet. SUSE collapses to a single
series entry, and Pulumi, Semaphore and Spectro Cloud join it. Each was
opened and checked against the project before being added. K8sGPT and
LlamaIndex join the integrations; both document LocalAI as a backend.
The quotes move above the lists so the section opens on its strongest
line, which is somebody else's. The hero gains a GitHub call to action,
the APEX collection link was returning 404 and is corrected, and the
footer no longer describes the site as a design mock.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m]
Follow-up to 42541dd4f, which routed lint to arc-runner-set. Both of its
jobs failed there in one second (run 30637392862): the runner image has
git, curl, unzip, tar, ldd and python3, but not make, and build-scripts
additionally needs gcc because the packaging-script tests compile a
throwaway binary and inspect it with ldd.
golangci-lint needs make twice over: `make protogen-go` (which also wants
curl + unzip to fetch protoc) and `make lint` itself. So both jobs go back
to ubuntu-latest.
gh-pages.yml stays on arc-runner-set and is unaffected: it uses no make and
no C toolchain, and setup-go / actions-hugo fetch their own toolchains.
The preflight steps stay. They cost about a second on the hosted pool, and
they are what turned this into a one-second named failure instead of an
opaque one midway through a build. When the runner image gains make + gcc,
re-routing is one runs-on line per job. Any such re-route must stay
push-only: lint also runs on pull_request, and fork PRs execute untrusted
code that must not reach a persistent self-hosted runner.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
The GitHub-hosted runner pool is shared per ACCOUNT, not per repo, so a
burst in one repo starves every other. On 2026-07-31 it reached zero
scheduled jobs for 35 consecutive minutes with 39 jobs queued, while
arc-runner-set completed 12 jobs without interruption across the same
window. Actions was healthy globally (other public repos were scheduling
normally), so this is an account-level throttle we cannot fix from inside
the workflows, only route around.
Site publishing and lint are small, run on nearly every commit, and gain
nothing from waiting behind a saturated hosted queue, so both move to
arc-runner-set, the label already proven in generate_intel_image.yaml.
lint.yml is routed for PUSH ONLY, and this is the important part: that
workflow also triggers on pull_request, and a fork PR executes untrusted
contributor code. Running that on a persistent self-hosted runner would be
a real compromise vector, so anything that is not a push to mudler/LocalAI
stays on the ephemeral hosted pool. gh-pages.yml needs no such clause: it
triggers only on push-to-master and workflow_dispatch, so it never runs
pull-request code. Both carry a repository guard so forks, which have no
such runner label, fall back to hosted instead of queueing forever.
Neither workflow uses sudo or apt, and both fetch their own toolchains via
setup-go / actions-hugo. A self-hosted image can still be leaner than the
hosted one, so each lint job opens with a preflight that names the missing
tool (curl/unzip/make for protoc and lint; gcc/ldd/python3 for the
packaging-script tests) rather than failing opaquely mid-build. Reverting
is one runs-on expression per job.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed backend verification performs separate registry requests for manifests and referrers. Reuse LocalAI's version-aware User-Agent there so the full install flow is attributable to LocalAI.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Adds the "who turns up around this project" section, split into three lists
because the evidence behind each one is a different strength and collapsing
them into a single logo wall would overclaim.
Contributors 21 companies whose engineers have commits here. Evidence is
the commit history plus the employer on that person's public
GitHub profile, so it is a claim about the person. Commit
counts are shown next to each name, including the ones that
are a single patch, because hiding that would be the whole
problem.
Integrations six projects that reference LocalAI in their own repository
or documentation, which anyone can verify without asking us.
Press four SUSE Communities articles about running LocalAI.
Names are set in type rather than fetched as logos. A logo reads as
endorsement, and a one-line typo fix from somebody who happens to work at a
large company does not support that, quite apart from what their trademark
policy says about it.
ADOPTERS.md is the mechanism for the stronger claim. An organisation that
wants to be listed as a user opens a pull request adding itself, which is both
the evidence and the permission, and is publicly auditable afterwards. The
file says plainly what the website does and does not claim, so the next person
to ask "can we add some big names" has the answer in the repository.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] [WebSearch]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(website): split the site, move docs to /docs, add a landing page
The Hugo docs site has always been localai.io itself, which left nowhere to
explain what LocalAI is or show what the team builds. This adds a separate
marketing site at the root and moves the documentation under /docs/.
Docs:
The existing site keeps its content tree and its Relearn theme, and now
builds with baseURL <root>/docs/. Its _index.md, which held a hand written
landing page, becomes a real documentation home.
Every previously published URL keeps working. GitHub Pages has no server
side rewrites, so .github/ci/gen-redirects.sh walks the built docs output
and leaves a meta refresh plus a canonical link at each old root path. It
covers bare .html files too, which is what keeps /gallery.html alive, and
it never overwrites a path the marketing site already owns.
Website:
A second Hugo site under website/ with its own layouts and no external
theme, so the marketing side does not have to fight Relearn's home rooted
menu and asset pipeline. CI builds both and merges them into one Pages
artifact.
The design is derived from the project logo rather than invented: the navy
of the triangle, the cyan of the llama, the purple of the speed bars. Those
offset bars became the motion signature. The background renders a real
depth-anything.cpp depth map as contour lines and switches to a
locate-anything.cpp style detection overlay over the engines section.
Also included: an /engines/ index driven entirely by data/engines.yaml, a
/blog/ section with five posts written from the release notes and the
engine benchmark suites, install.sh and a Kubernetes manifest since the
site advertises both, and a rule in .agents/ that release preparation now
includes a blog post and demo clips.
Every figure on the site is derived from the repository or the GitHub API,
not from memory. Correcting them against their sources found one error in
README.md: voxtral-tts.c is text to speech, not speech to text.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] [Agent]
* feat(website): add a star history chart, rewrite the history post in first person
The history post read like a changelog written by a committee. It is now in
Ettore's voice, first person, with the admissions left in.
The numbers paragraph in particular read like a directory listing. It now says
what the figures mean rather than which file they came from.
Adds an interactive star history chart, built from the GitHub stargazers API
rather than embedded from a third party, so the page makes no external request
and cannot break when someone else's service is down. The four releases the
post is organised around are marked on the curve, and the labels stack into
rows because three of them land within two months of each other.
The API stops paginating at 40,000 items, so the curve is measured up to
December 2025 and the segment from there to today's total is drawn dashed,
labelled as an estimate in the caption and in the tooltip. It is a straight
line between two known points, and the chart says so rather than implying it
is data.
Also drops "marketing site" from the README heading and everywhere else it
appeared, and calls it the main site instead.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
PR #11041 rewrote the testllama31inputResult1 fixture and un-escaped the
backslashes inside its Go raw-string literal, turning `[^"\\]` into
`[^"\]` and `["\\/bfnrt]` into `["\/bfnrt]`. The fixture is compared
line-by-line against the grammar built from PRIMITIVE_RULES in
bnf_rules.go, which is unchanged and still emits the doubled form, so
"generates a valid grammar from JSON schema" fails on every platform.
Restore the four fixture lines to their pre-#11041 form. The cyclic $ref
and depth specs added by that PR are untouched.
The regression reached master because only the DCO check ever reported
on #11041; its test runs were cancelled during the CI purge.
Assisted-by: Claude Code:claude-opus-5[1m] [Bash] [Edit]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Add the Qwen3.5-MoE POCKET-35B and Gemma 4 POCKET-26B releases with Q4, Q2, and compact IQ1 variants where available. Verify every artifact hash against its Hugging Face linked etag.
Assisted-by: Codex:gpt-5 [web]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* refactor(ui): add layout/text primitives and per-page CSS blocks
The React UI already shipped a design system (tokens, form grids, data
tables, stat cards, callouts) that the pages largely bypassed: ~2,000
`style={{ ... }}` literals across src/pages and src/components. Each one is
a spacing or colour decision made locally, so no two pages share a rhythm,
which is the main reason the app reads as unfinished rather than as one
product.
Two additions, both to App.css:
- A small semantic primitive layer: .stack / .hstack for vertical and
horizontal rhythm, .text-note / .text-sub / .text-meta / .text-mono for
the text roles the pages kept re-deriving, .tone-* + .icon-chip for
semantically tinted icons, plus scale-locked spacing and size steps.
Deliberately short and semantic, not a utility framework: the size and
spacing classes exist mainly so that an OFF-scale value stays an inline
style and therefore stays visible.
- Named blocks for the shapes fifteen pages actually have (.p2p-diagram,
.usage-tile, .tr-code, .mw-badge, .set-rail, ...), so those shapes are
defined once instead of per call site.
Two findings worth recording. The type scale is xs 0.6875 / sm 0.8125 /
base 0.875, and 0.75rem was in use roughly 100 times without being on it
(along with 0.7, 0.85, 1.1 and 0.625rem); all now snap to the nearest step.
And there were thirteen distinct table column widths across the app where
three or four would do; they are pulled into .col-w-* so the ladder is
visible in one place, ready to normalise separately.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
* refactor(ui): move repeated inline styles onto shared classes
Six passes over src/pages and src/components, each matching a whole
`style={{ ... }}` attribute exactly so the swap is provably equivalent:
- the identical "waiting for the first response" wrapper, repeated
verbatim in 21 files
- text roles (size + colour combinations) onto .text-note / .text-meta /
.text-sub / .text-mono
- semantic colours, type-scale steps, .panel-title, .list-row
- spacing and weight steps, .stack / .hstack rows
- table column widths, small pills, chart legend swatches
- the remaining shapes appearing three or more times
One class of bug is worth calling out, because it is what a careless
style-to-class conversion produces and it is invisible to every check we
run. Adding `className="x"` to an element that already had a className
leaves TWO className attributes; JSX keeps the last and silently drops the
first, so `<i className={icon} className="text-xs" />` loses its icon while
passing eslint, `vite build` and the full Playwright suite. 112 of these
were introduced and repaired here. The gate added in a later commit fails
on them.
No visual change is intended beyond snapping off-scale font sizes onto the
type scale. Verified after every pass: eslint 0 errors, vite build passes,
Playwright page-render-smoke + navigation 22/22.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
* refactor(ui): convert fifteen pages onto named classes, add inline-style gate
Full per-page conversions, each one reading the page, naming the shapes it
actually has, and leaving inline only what is computed at runtime:
P2P 205->0 Traces 50->0 ConfigFieldRenderer 23->0 NodeDetail 29->1
ModelEditor 25->1 NodeInstallPicker 29->1 FineTune 175->2
Nodes 33->2 ImportModel 35->3 Talk 43->4 AgentJobDetails 27->4
Usage 97->6 Settings 24->6 Middleware 54->8 Backends 87->45
Every remainder is genuinely dynamic: a data-driven badge colour, a
`width: ${pct}%`, a tooltip's coordinates.
Naming the shapes made reuse fall out on its own. Nodes reuses the P2P
setup shapes (both present the same "no workers yet, here is how to add
one" flow) and Model Editor reuses the Settings section rail, which was
byte-identical. Two shared *style objects* also turned out to be classes
wearing a costume and were deleted: `monoCell` in Usage and `hintStyle` in
ImportModel.
Some fixes fell out of the conversion. The evaluation toggle on the
fine-tuning page was a hand-rolled div that rendered as a clipped circle;
it is the existing Toggle component now. That page's empty state pointed at
a "New Job" button that was scrolled off the top of the page, and now
carries its own call to action. And `.input--file` is added at the system
level rather than as a local hack, because ImageGen and VideoGen truncate
their file inputs the same way today.
scripts/inline-style-gate.mjs is the ratchet that keeps this from
regressing. It does not forbid inline styles; it fails when the total goes
UP (same discipline as the coverage baseline) and when an element carries
two className attributes. eslint would catch the latter via
react/jsx-props-no-duplicate-props, but that needs eslint-plugin-react,
which this project does not depend on, so the check lives next to the tool
that causes the problem.
npm run lint:inline-styles # check against baseline
npm run lint:inline-styles:report # per-file counts, worst first
npm run lint:inline-styles:write # refresh after converting
Net: 2,061 -> 611 inline styles. eslint 0 errors, vite build passes,
Playwright page-render-smoke + navigation 22/22, gate green on both checks.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
refactor(backends): extract shared package-system-libs.sh from package.sh
The arch-detect-and-copy-system-libs block (Darwin rpath / x86_64 / aarch64
loader + libc/libstdc++/libgcc_s/libm/libgomp/libdl/librt/libpthread) was
inlined verbatim in 31 backend package.sh scripts. Extract it into a single
sourced scripts/build/package-system-libs.sh, the CPU-side counterpart to
scripts/build/package-gpu-libs.sh and its sourcing contract.
Consolidating the copies fixes three drift classes that had crept in:
- libgcc_s.so.1 and libstdc++.so.6 were listed twice in 9 backends
(acestep-cpp, crispasr, moss-tts-cpp, omnivoice-cpp, piper,
qwen3-tts-cpp, silero-vad, stablediffusion-ggml, whisper); the shared
script copies each once.
- libgomp.so.1 was omitted from opus. OpenMP consumers dlopen it rather
than link it, so the missing copy only failed at runtime; the shared
script always includes it.
- the Darwin @loader_path/lib rpath was applied only in piper and
silero-vad; both now pass their packaged binary to the shared script,
preserving that behavior. Every other backend passes an empty binary
path so no rpath is added, preserving its current behavior.
Each backend's pre/post packaging steps (binary copy, run.sh, ldd closure
walks, ggml variant bundling, espeak/OpenBLAS extras, the ds4 validate step)
are preserved verbatim; only the inline if/elif/else arch block is replaced
by a single source line.
Signed-off-by: supermario_leo <leo.stack@outlook.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
* fix(backends): fall back to copying links when the filesystem rejects symlinks (#10890)
Backend installation extracts the OCI image tar via containerd's
archive.Apply, which calls os.Symlink directly. On filesystems that do
not support symlinks (notably CIFS/SMB mounts, commonly used to back the
/backends volume) the syscall fails with "operation not supported" and
the whole install aborts, leaving an empty backend directory. The CUDA
llama.cpp image trips this on the libcublas.so -> libcublas.so.12.x
symlink.
When archive.Apply fails with a link-unsupported error, reset the
staging directory and re-extract with a pure-Go walker that still
attempts real symlinks/hardlinks first and degrades to copying the link
target's contents in place when the filesystem rejects them.
mutate.Extract already flattened the layers, so the tar carries no
whiteouts to interpret. Link copies are deferred to a second pass so
forward references resolve.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]
* fix(oci): check deferred Close in copyFilePreservingMode (errcheck)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]
* fix(oci): reject path-traversal tar entries in the link-copy fallback
safeJoin sanitized "../.." entries by clamping them under root instead of
rejecting them, so a malicious entry was silently redirected rather than
refused. Join without the leading-slash trick and reject any entry whose
cleaned path resolves outside root; absolute link targets are still mapped
under root (image-root relative) rather than escaping.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]
* fix(oci): silence gosec on the validated link-copy file ops
Use hdr.FileInfo().Mode() instead of converting the int64 tar mode to
os.FileMode (removes two G115 overflow findings), and annotate the tar
extraction file operations with justified #nosec comments: every path is
validated by safeJoin against the extraction root before use (G304/G305).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]
* fix(oci): build extraction image from downloaded layers
Avoid appending downloaded layers to the original remote-backed image, which duplicates the layer stack and reopens the source during extraction. Building from an empty image preserves the flattened whiteout semantics while keeping extraction local.
Assisted-by: Codex:gpt-5
* fix(oci): materialize chained links in dependency order
Retry deferred link copies until their targets exist so soname chains work on filesystems without symlink support. Document that copied links can increase backend storage usage on CIFS and SMB mounts.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(api): add /v1/detokenize endpoint
Closes#1649.
Mirror of the existing /v1/tokenize path, requested by @benniekiss in
the issue thread for "complete API workflow" use cases that need to
turn token IDs back into text without local processing.
- Add Detokenize gRPC RPC with DetokenizeRequest{tokens} /
DetokenizeResponse{content} messages.
- Implement in the llama.cpp backend using common_token_to_piece, the
same primitive TokenizeString already uses internally.
- Other backends inherit the default Unimplemented from base.Base, in
line with how Detect, Rerank, etc. are gated per-backend.
- Wire up the Go gRPC interface, server, client, and in-process embed
wrapper alongside their TokenizeString counterparts.
- Add the schema types, ModelDetokenize wrapper, HTTP handler, route
registration, RouteFeatureRegistry entry (gated by FeatureTokenize so
no new feature flag is needed), and the discovery map entry under
ai_functions.
- Regenerated swagger reflects the new endpoint and types.
- Update authentication.md to list /v1/detokenize alongside /v1/tokenize.
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
* test(e2e): add mock backend tests for /v1/detokenize
Add Detokenize to the mock gRPC backend and wire up two e2e tests in
the MockBackend suite: one that posts known token IDs and asserts a
non-empty content response, and a round-trip that tokenizes first then
detokenizes the returned IDs.
Addresses reviewer feedback on #9620.
Assisted-by: Claude:claude-sonnet-4-6
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
* fix(kokoros): implement detokenize in the Rust backend service
The Detokenize RPC added in this PR grows the tonic-generated Backend
trait. Unlike the other languages there is nothing to inherit a default
from — Rust trait impls must list every method — so
backend/rust/kokoros failed to compile:
error[E0046]: not all trait items implemented, missing: `detokenize`
--> src/service.rs:72:1
72 | impl Backend for KokorosService {
Go backends pick up the Unimplemented default from base.Base, and the
generated C++/Python servicer bases default to UNIMPLEMENTED, which is
why the Rust backend was the only one that broke. kokoros is the sole
Rust crate in the tree, so this is the full extent of the fallout.
Return Status::unimplemented("Not supported"), matching how this same
file already gates tokenize_string and ~20 other unsupported RPCs.
Fixes the tests-kokoros and backend-jobs-singlearch-4 (-cpu-kokoros)
failures on the previous head.
Assisted-by: Claude:claude-opus-5 cargo
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
---------
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
On 2026-07-30, 12 of the 23 queued runs of this workflow were commits like "add
1 new model to gallery" or a docs fix, each rebuilding all 18 container images.
That was roughly 216 queued jobs producing byte-identical output, in a queue
holding 1071 jobs with an oldest entry two days old.
Verified against the shipped Dockerfile before assuming it: the final stage
copies only entrypoint.sh, healthcheck.sh and the local-ai binary, there is no
go:embed of gallery/ or docs/, and the gallery is fetched at runtime from
github:mudler/LocalAI/gallery/index.yaml@master. A gallery-only commit produces
an identical image, and the gallery change reaches users through GitHub whether
or not an image is rebuilt, so nothing is delayed by skipping.
Add a `changes` job that decides once whether the push can affect an image; the
other 11 jobs take `needs: changes` and an `if:` on its output.
A job gate rather than paths-ignore on the trigger, for two reasons that both
fail silently if got wrong:
- paths-ignore on `push` also applies to tag pushes, and a tag created on an
existing commit carries an empty commits list. That would skip the release
image build with no failure anywhere. The gate short-circuits to build for
refs/tags/*, and for a base commit that is missing, zero or unresolvable --
the same run-everything posture the backend matrix filter takes for a
truncated diff.
- the merge jobs use `if: ${{ !cancelled() && ... }}`, and !cancelled() is
true when a dependency is skipped, so they need the gate named explicitly
or they would try to merge manifest lists for images never built.
Checked the decision logic against real commits from the queue: the two
gallery/docs commits resolve to build=false, the two code commits to build=true,
and all three fallback paths (tag, zero base, unresolvable base) to build=true.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Document all stateful container paths, explain upgrade behavior and UnRAID mappings, and correct the obsolete troubleshooting mount target.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
docs: clarify reverse proxy bulk job guidance
Mention ingress controllers as another place to configure equivalent upstream response timeouts, and include an example private LocalAI URL for trusted bulk jobs.
Assisted-by: Hephaestus:openai/gpt-5.5 [opencode]
Signed-off-by: Owen Adirah <owenadira@gmail.com>
Stage the negative GPU-layer sentinels expected by the upstream argument parser, then restore LocalAI resolved values unless a passthrough flag explicitly overrides them. This avoids the parser assertion that terminated the backend for any generic option.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Version-pin bumps dominate PR volume: 48 update/* PRs in the week to
2026-07-30, from 16 pins, each a two-line diff. bump_deps.yaml runs a 28-entry
matrix daily and opens one PR per moved pin; CRISPASR and the gallery checksum
produced one every day, ik-llama-cpp six in seven days. Almost all of them edit
nothing but a single backend/*/<name>/Makefile.
Neither image-pr.yml nor build-test.yaml can observe such a change. `make build`
is `go build ./cmd/local-ai`, GoReleaser builds that plus ./cmd/launcher, and
the core image's final stage ships only entrypoint.sh, healthcheck.sh and the
binary. The per-backend trees are copied into the builder but nothing in them
reaches the output. That is 7 + 3 jobs per bump PR that cannot fail for a reason
the diff caused, roughly 410 jobs a week.
Add backend/{cpp,go,python}/** to the paths-ignore of those two workflows. The
inputs that do reach the binary are deliberately outside those prefixes and so
still trigger a full run: backend/backend.proto (protogen-go), go.mod/go.sum
(the go mod tidy before-hook), and backend/Dockerfile.* .
Not applied to the workflows that genuinely read that tree:
test.yml TEST_PATHS names ./backend/go/{cloud-proxy,local-store,
valkey-store}/...
lint.yml .golangci.yml carries backend/-scoped rules
tests-e2e.yml the e2e suite drives real backends over gRPC
backend_pr.yml its whole job is rebuilding the changed backend
Simulated against the change shapes that occur in this repo. Pin bumps and
python requirement bumps skip; backend.proto, go.mod, a core Go edit, a
backend/Dockerfile edit and any mixed diff all still run, since paths-ignore
skips only when every changed file matches.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
.agents/ci-caching.md stated that `cache-to: type=registry,mode=max` "exports
the cache mount data into the registry cache, so subsequent builds restore it".
BuildKit does not do that. A `--mount=type=cache` lives in the builder's local
state and is not part of a registry cache export, and every CI job gets a fresh
runner with a fresh builder, so /root/.ccache starts empty on every build.
The compile script already prints `ccache -s` after a `ccache -z`, so the
evidence was sitting in the logs:
job 89766266951 llama-cpp cublas-13 6369s 0 / 889 hits (and 0 / 1778)
job 89766267281 llama-cpp hipblas 8160s 0 / 537 hits
job 90210828110 llama-cpp cublas-12 5673s 0 / 813 hits
The first two are the control. Their commit, 90355cd44, changed exactly one
file: backend/go/magpie-tts-cpp/Makefile, nowhere near llama.cpp. The engine
source was byte-identical to the previous build, which is the case this section
claims ccache serves, and the hit rate was still 0.00%. A restored-but-stale
cache would show partial hits; 0-of-N is an empty cache.
So llama-cpp, ik-llama-cpp, turboquant, bonsai, ds4 and privacy-filter pay the
ccache wrapper overhead and get nothing back, and multi-hour C++ builds
recompile identical translation units every time.
Record this rather than silently extending it. Wiring the same mount into
Dockerfile.golang, which covers 215 of the 434 matrix entries, measured 18%
faster locally on a rebuild after a source edit with a 71.5% hit rate, but only
because that test reused a single builder across both builds. In CI it would be
a no-op. The note spells out what would actually work (ccache remote_storage or
sccache with a real backend, or round-tripping the cache dir through
actions/cache) and what each costs.
Also correct the composite-actions list, which still named test.yml as a
free-disk-space consumer after #11219 removed it.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Dockerfile.golang builds 215 of the 434 matrix entries: every ggml/C++ engine
wrapped in Go. Each of those Makefiles clones an upstream repo at a pinned SHA
and compiles it once per SIMD variant (depth-anything-cpp builds four: avx,
avx2, avx512, fallback), and those variant targets depend only on the clone.
They cannot observe a change anywhere else in the LocalAI tree.
The compile sat below `COPY . /LocalAI`, so any edit anywhere invalidated it and
recompiled C++ that had not changed. Move it above that COPY, behind a copy of
only the backend's own directory.
This lands the compile in the part of the image the registry cache already
restores. Measured on two real CI builds of this Dockerfile (jobs 90551029008
and 90551028904, both fresh runners): 13 of 17 layers CACHED from
quay.io/go-skynet/ci-cache. The uncached tail is exactly `COPY . /LocalAI`, the
git-config RUN and the build RUN. Putting the engine above the COPY moves it
from the uncached tail into the cached region.
Local measurement, depth-anything-cpp CPU, rebuild after editing a Go file
outside the backend:
master 78s, 216 C++ objects compiled
this change 25s, 0 C++ objects compiled (67% faster)
Note what this deliberately is not. An earlier attempt wired a
--mount=type=cache ccache into the same RUN. BuildKit does not export cache
mounts to a registry cache, so that measured well locally and is a no-op in CI
(see the ccache section of .agents/ci-caching.md). This change relies only on
ordinary layer caching, which the 13-of-17 figure above shows already works
here.
The layer copies the backend's whole directory rather than just the Makefile:
the CMake targets also need CMakeLists.txt and the file list differs per
backend. The cost is that editing a backend's own Go sources invalidates its
engine layer. The expensive cases are unaffected, since a shared-build-input or
backend.proto change, the weekly full-matrix cron and a tag push all rebuild
every backend while touching none of their directories.
Scoped to one backend for now: only depth-anything-cpp gains the `engine`
target. The other 27 fall through the `make -n engine` guard and build exactly
as before, verified against local-store and silero-vad. Rolling the target out
to the remaining 12 backends that define VARIANT_TARGETS is mechanical once this
is confirmed against the registry cache on master.
One caveat on merge: inserting layers shifts the cache keys, so the first build
of each entry after this lands is a full miss. It pays for itself on the second.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Measured over the week to 2026-07-30, 97% of CI wall-clock is queueing and 3%
is execution: a median 5-hour queue against a 4-20 minute median job. With the
queue saturated, throughput is concurrency divided by service time, so cutting
execution time raises the drain rate directly. Two steps stood out as paying
nothing for what they cost.
test.yml: drop the free-disk-space step (~3.1min per run, ~22 h/week). That
action exists to make room for docker buildx layers and this job runs no buildx
step. It was also sized for a `make test` that downloaded multi-GB GGUF/whisper
fixtures and built llama-cpp/whisper/stablediffusion-ggml; the test-suite reorg
moved all of that into tests/e2e-backends and tests/e2e-aio, as the Makefile
test target already records. Its tool-cache:true wipe was additionally deleting
/opt/hostedtoolcache, forcing setup-go and setup-node to re-download toolchains
that ship preinstalled on the runner.
build-test.yaml: build only the host target on pull_request. The three-platform
cross-compile (linux/amd64, linux/arm64, darwin/arm64) is the bulk of that job's
~6.6min median, ~47 h/week, and nothing consumes a PR's binaries. goreleaser's
--single-target still runs every before-hook (protogen-go, react-ui, go mod
tidy), so the "is the release build broken" signal is unchanged. master pushes
and tags keep building all three.
Also record why the Linux Go workflows pass cache: false to actions/setup-go,
since it reads as an oversight and is not. Set up Go has a median of 11 seconds
on those runners, so there is nothing to win, and the repo already sits at
GitHub's 10 GB Actions cache ceiling with 31 entries, where each setup-go entry
is 222-375 MB on Linux and up to 1.4 GB on macOS. Re-enabling it would evict
something that is earning its space.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(vllm): apply Options[] engine flags before engine init (#11130)
CLI-style flags in a model's `options:` array (`--quantization:gptq_marlin`,
`--enable-prefix-caching`, `--kv-cache-dtype:fp8_e5m2`) were discarded: the
backend only ever read `tool_parser`/`reasoning_parser` out of Options[], and
did so *after* `AsyncLLMEngine.from_engine_args()`, where nothing it set could
still reach the engine.
Map `--` prefixed options onto the AsyncEngineArgs dataclass before the engine
is constructed. Names are normalized the way vLLM's CLI spells them
(`--enable-prefix-caching` -> `enable_prefix_caching`), values are coerced to
the target field's type (bare flag -> True for booleans), and unknown or
uncoercible flags warn and are skipped instead of failing the load, since
Options[] is a bag shared with backend-level settings. Field types come from
the annotation's base so `Literal["auto", "float16"]` (vLLM's dtype) is not
mistaken for a float.
Precedence is typed proto fields -> `options:` -> `engine_args:`. The
production engine_args defaults seeded in hooks_vllm.go therefore skip any key
the user already set as an option, otherwise the later engine_args pass would
silently override it. Parser lookups now accept both spellings, so
`--reasoning-parser:qwen3` selects LocalAI's parser as well.
The helper's tests are stdlib-only and run in the lint workflow's
dependency-light job via `make test-python-helpers`.
Assisted-by: Claude:claude-opus-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
* backend(audio-cpp): add the native build scaffold
Links 0xShug0/audio.cpp engine_runtime through its public framework headers
and serves Health/Status. Model loading and the audio RPCs follow.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): keep the build-tree rpath at $ORIGIN
Upstream sets CMAKE_BUILD_WITH_INSTALL_RPATH in its own directory scope, so
CMake was appending its build-tree library dir to our target and baking an
absolute build-host path into the shipped binary. Set BUILD_WITH_INSTALL_RPATH
on the target so a package that forgets to bundle libggml*.so fails on the
build machine too, instead of only on a user's box.
Also document why EXCLUDE_FROM_ALL must stay on the add_subdirectory call,
correct the claim that Ubuntu ships no gRPC CMake config, stop the pin comment
from repeating the assignment token that bump_deps.sh rewrites, and make
test-engine fail rather than pass when no test is registered.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): parse namespaced model options
Splits option entries on the first colon so path values survive, and routes
load./session. prefixes to the upstream load and session option maps.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): reject out-of-range numeric model options
std::atoi is undefined once the digits exceed long and in practice wraps, so
device:2147483648 was accepted and handed the ggml backend selector a device
index of -2147483648 from a function whose error text promises a non-negative
integer. Parse with strtol and reject on ERANGE, on a value above INT_MAX, and
on any unconsumed trailing input. The error strings are unchanged.
Name the whole entry in the unknown-key error too: an entry like ':value' has
an empty key and left the user nothing to grep for in their YAML.
Tests look keys up through a helper instead of map::at, so a prefix off-by-one
fails one named check rather than aborting the binary and skipping the rest of
the suite, and cover the overflow, negative and non-numeric paths.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): route LocalAI RPCs onto audio.cpp tasks
Task-major resolution over the family's advertised capability set, with the
voice-reference and instructions signals selecting cloning and voice design,
and a streaming-to-offline fallback for server-streaming transcription only.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): use upstream's 'spk' task name and pin the preference order
The SpeakerRecognition short name was 'spkrec', which audio.cpp neither prints
nor parses; a name copied out of audio.cpp was rejected and a pinned 'spkrec'
would not survive the engine boundary. Emit 'spk', keep 'spkrec' as an
input-only alias, and correct the known-tasks lists.
Three assertions were vacuous because their fixtures advertised a single task,
so reversing a preference order or dropping the RPC name and the attempted
pairs from the capability error all passed. Give them fixtures that can tell
the orderings apart.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): convert sample, time and PCM units
Integer nanosecond conversion so 44.1 kHz stays exact, float seconds for the
VAD and diarization messages, and saturating s16le encode so an overshooting
sample cannot wrap to the opposite sign.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): harden seconds_to_samples against NaN and overflow
seconds_to_samples is the one entry point fed by untrusted-shaped input: a
float-seconds timestamp off the wire, or a boundary from a model that diverged.
Its guard covered only the low side, so NaN and out-of-range values fell through
to an undefined double-to-int64 cast and came back as INT64_MIN. A hugely
negative sample index used later as an offset or a length is a wild pointer
rather than merely a wrong timestamp. Reject NaN with the !(x > 0) form and
saturate before the cast.
Also round instead of truncating there. These functions exist to cross the float
seconds boundary the VAD and diarize messages use, and truncation lost a sample
about half the time on the samples-to-seconds-and-back round trip, starting at
n=1.
Pin the decode scale at INT16_MIN, pin nanosecond truncation on a nonzero
fraction, and record why the clamp argument order in f32_to_s16le is
load-bearing for NaN.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): map NaN PCM samples to silence explicitly
f32_to_s16le relied on std::min argument order to keep a NaN sample away from
std::lround, whose result is unspecified for NaN. That was too subtle to rest on
a comment, and the comment was itself wrong: it warned against a spelling that
the outer std::max already catches, while three real spellings leak, including
std::clamp, which is the idiomatic C++17 way to write the same clamp and so the
likeliest future edit.
Divert NaN before the clamp and encode it as 0. A NaN sample rendered as a
full-scale click is worse audio than a dropped one, and this unit converts audio
that may have originated off the wire.
Pin it with an exact-value check rather than a range check, since all three
outcomes the plausible spellings produce are finite and inside full scale, plus
an invalid-operation check that fails unless the NaN is diverted before any
ordered comparison. That second check is what catches modernizing the clamp and
dropping the guard together.
Also bound the seconds round-trip comment, which claimed unconditionally what
holds only below roughly 2^23 samples, and document NaN, saturation and that
bound in the header.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): assemble transcripts from runtime spans
The top-level transcript text is TaskResult.text_output verbatim. audio.cpp
carries text nowhere else: speech_segments, speaker_turns and word_timestamps
hold spans and labels only, so deriving the text from them empties the
transcript for any producer that omits word timing, VibeVoice diarized ASR
included. Fixtures cover every observed producer shape.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): keep a nested speaker turn's own label
A segment sourced from speaker_turns re-derived its speaker by greatest
overlap. A turn's overlap with its own span is the largest possible, so a turn
nested inside another speaker's turn could only tie with the container, and the
tie went to whichever came first. sortformer_diar binarizes each speaker
independently and sorts by start sample, so the container always comes first
and the interjecting speaker was silently erased from DiarizeSegment.speaker.
choose_segment_spans now carries the label out with the span.
Also pins the nearest-segment fallback against measuring from either endpoint
or from segment position, which a trailing-only stray word could not do, and
exercises the empty-word guard in join_words. Two fixtures that pin a rule but
do not mirror any pinned family are relabelled defensive.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serialize runs with a wedge-aware guard
audio.cpp sessions are not reentrant and a wedged CUDA call cannot be
cancelled, so a plain mutex would pile every worker thread behind a stuck GPU.
Callers waiting past the configured bound, or arriving while the holder has
already overrun it, fail fast instead.
A caller that queues behind a healthy run deliberately does not stamp the
clock: only the thread that takes the lock does. Stamping on arrival would
restart the wedge clock on every request and hide a stuck run from everyone
behind it, which is the pile-up this guard exists to prevent.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serialize inference through an InferenceLane
One audio.cpp model is loaded per backend process and its sessions are not
reentrant, so concurrent gRPC handlers have to take turns. Serialization alone
is not enough: a wedged GPU call cannot be cancelled from userspace, so an
unbounded queue behind one stuck run would swallow every gRPC worker thread
until the process is useless.
InferenceLane gives handlers a lane with room for one runner. LaneEntry occupies
it for a scope and gives it back on every exit, including an exception, and is
the only way to take the lane at all: occupy/vacate are private with LaneEntry
as the sole friend, so a caller cannot acquire without holding something that
releases. LaneEntry is immovable on purpose, because a moved-from entry would
have to stop releasing while the lane still recorded it as occupied.
A caller either waits indefinitely or brings a millisecond budget. A bounded
caller that cannot get in fails instead of waiting on, and a bounded caller
whose budget is already shorter than the age of the run in the lane fails
immediately, which is what stops a queue forming behind a wedged run. The two
failures carry different text: one names the wait it exhausted, the other states
the measured age of the run without claiming to know why it is long, since a
short budget meeting a legitimately long run lands there too.
The run's age is stamped only after acquisition. A waiter that published itself
as holder would restart the measurement and hide a genuinely stuck holder from
every caller behind it.
Budget negotiation and the overrun decision are pure functions taking their
inputs explicitly, so both are covered without threads or sleeping. The
per-model ceiling arrives as an int of milliseconds; a request may tighten it
and may never loosen it.
Replaces the previous run_guard unit, which was a derivative of an
Apache-2.0 file upstream and could not stay in an MIT tree. Written from a
behaviour contract with no reference to the removed code.
Tests: 65 checks, standard library only, single translation unit, clean under
-Wall -Wextra. Mutation tested at 23/23 killed; two of those mutants exposed
missing coverage and the tests were extended until they died. ThreadSanitizer
clean.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): make the B10 test able to fail, and document LaneEntry
Review of the previous commit found the B10 test could not fail for the reason
it was named. It aged the in-flight run to about 120 ms and then tried two
budgets, 30 ms and 60 ms, both under that age, so both callers took the
fail-fast path. "The two failure modes do not share one message" was comparing
two fail-fast messages that differ only in the budget they print, and the
timeout path was never reached. The second budget is now 400 ms, well over the
run's age, so that caller queues and times out, and a new check asserts which
path each caller took instead of inferring it from inequality. A mutant that
makes the fail-fast path emit the timeout message previously died only on B4 and
B8 checks; it now also dies on B10.
Comment-only changes elsewhere. LaneEntry now says it is not reentrant and does
not detect reentrancy: a second entry on a thread that already holds the lane
surfaces as LaneUnavailable with a positive budget, but parks silently in
unbounded mode, which matters because a handler may hold one across a whole
stream. The immovability note now names the shapes that work, an optional
emplaced in place or a unique_ptr, rather than saying to hold the entry
indirectly without saying how; all three documented forms were compiled before
being written down, which is how the note came to say that an optional of an
immovable type cannot itself be returned.
The header's explanation of why fail-fast exists is reworded. Two clauses traced
back to a specification written after reading the Apache-2.0 upstream header,
and while that was judged de minimis, this unit was rewritten precisely to carry
no upstream expression at all.
The margin table in the report was also wrong about which wall-clock margins are
load-sensitive: there are four, not one, and the tightest is the B3 arrival
check, which is now flagged at the call site. No margin value changed and none
moved across 65 runs.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): gate model loading on the audio.cpp family
Refuses any GGUF without an audiocpp.model_spec.family key and any non-GGUF
path without an explicit family option, so the model loader's greedy backend
probe cannot bind an unrelated llama.cpp GGUF to this backend (#9287).
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): load models and cache sessions per task
Loads one ILoadedVoiceModel and creates an IVoiceTaskSession lazily per
(task, mode), so the same model serves both the unary and streaming RPCs.
LoadModel derives the family from GGUF metadata or an explicit option and
fails with INVALID_ARGUMENT otherwise, so a failed load is a gRPC error the
backend probe can see.
audiocpp_backend::Task mirrors engine::runtime::VoiceTaskKind positionally,
and drift there is silent: every unit still compiles and every test still
passes while the backend runs a different task. Two mechanisms pin it. The
static_asserts in loaded_model.cpp catch an insertion or a reorder, and
-Werror=switch on that one file turns an appended upstream enumerator into a
build failure rather than a warning in a 600 file log.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): stop aborting the process on SIGTERM
The signal handler called grpc::Server::Shutdown directly. Shutdown takes an
absl::Mutex, which is not async-signal-safe: the handler can interrupt a thread
already holding that mutex, and abseil's deadlock detector responds by aborting.
Every SIGTERM therefore ended in exit 134 and a 'dying due to potential
deadlock' stack rather than a drained shutdown.
The handler now sets a lock-free atomic and returns. Server::Wait moves to a
helper thread so the main thread can poll that flag and call Shutdown itself,
outside any signal context. A condition variable would not have helped, because
notifying one from a handler is not async-signal-safe either.
SIGTERM and SIGINT both exit 0 with no stack trace, where both previously
exited 134.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): correct the status, lifetime and state contracts of LoadedModel
An environment fault during session creation was reported as UNIMPLEMENTED. A
missing libggml-cpu-*.so surfaced to the client as 'family silero_vad advertises
vad/offline but refused to create the session: Failed to initialize CPU
backend', which tells LocalAI the model cannot do this and must never be
retried, and sends an operator hunting a capability bug instead of a packaging
one. A throw from create_task_session is now a plain runtime_error, so it maps
to INTERNAL. Only a null return, where the family genuinely declined, stays a
CapabilityError.
The model.'s task: option was parsed and then dropped: it lived in a local that
died at the end of LoadModel and had no route to RequestShape::pinned_task.
LoadedModel now keeps it and exposes pinned_task().
The global model becomes a shared_ptr reached through snapshot(). An audio RPC
runs for seconds and cannot hold g_model_mu for its duration, so under a
unique_ptr a Free arriving mid-request would destroy the model underneath it.
Handlers now take a counted reference and whichever finishes last does the
teardown, outside the lock.
session_for documents the streaming state contract rather than resetting the
session itself. Resetting on a cache hit was tried first and is not possible:
silero_vad throws 'session prepare() must be called before Silero VAD reset()',
so it would turn an ordinary second fetch into a hard error. start_stream's base
implementation is already a reset, so a caller that runs prepare then
start_stream per stream gets a clean session; a probe against the bundled
silero_vad confirms an identical replay when it does and a carried-over stream
when it does not.
Also: an unknown backend: name is rejected before the model loads rather than
after; MainGPU is parsed instead of passed through std::atoi, which turned
'gpu1' into device 0 silently; and device carries a device_set flag, because 0
is both the default and a real device index, so MainGPU was overriding an
explicit device:0 that the neighbouring threads: handling promises will win.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serve the VAD and Diarize RPCs
Both emit float seconds, converted from the runtime's sample-index spans, and
both take a counted reference to the loaded model through snapshot() and hold it
for the whole call: a Free arriving mid-request drops only the global's
reference, so whichever request finishes last destroys the model instead of one
of them running on freed weights. An AddressSanitizer build reproduces exactly
that heap-use-after-free inside ggml_vec_dot_f32 when the handler keeps a raw
pointer instead, which is why the shape is what it is.
The inference lane is taken before session_for, not after. session_for reads and
writes an unsynchronised session cache and the offline run calls prepare(),
which mutates the session, so both belong inside the lane.
Diarize routes before it reads the input file, so a family that cannot diarize
at all says so rather than complaining about the audio first. Its per-segment
text stays empty because audio.cpp's SpeakerTurn carries a span and a speaker
label only, and nested or overlapping turns are passed through untouched: a
sortformer turn inside another speaker's turn is correct output for overlapped
speech, and LocalAI is overlap-tolerant downstream. Duration counts frames
rather than floats, so a stereo input does not report twice its length.
Verified end to end against upstream's bundled silero_vad, which needs no
download, using the bundled 16 kHz speech asset: a synthetic tone returns
nothing, correctly, because silero detects speech and a sine is not speech.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): enforce ModelIdentity on VAD and Diarize
audio-cpp was the only C++ backend without the model-identity guard, and no
later task in the plan added it. pkg/grpc/server.go enforces checkModelIdentity
on exactly these two RPCs, for the reason #10952 records: in distributed mode a
worker can recycle a stopped backend's gRPC port for another model's backend,
and the controller's liveness-only probe cannot tell a stale cached route from a
live one. Without this guard a stale route gets a different model's VAD or
diarization answer back with a 200.
The loaded identity lives on LoadedModel rather than in a separate global, which
is where this differs from llama-cpp. A handler holding the model through
snapshot() then necessarily judges against the identity that model was loaded
with, and a concurrent reload cannot swap one without the other. The refusal is
NOT_FOUND carrying the verbatim grpcerrors.ModelMismatchSentinel substring.
session_for and run_offline now take a const LaneEntry & proof-of-holding
parameter. The rule that both must run under the inference lane was prose, which
is exactly how the plan came to specify the inverted order; it is now a compile
error. Restoring the inverted order fails to build rather than racing on an
unsynchronised session map with a mutating prepare().
Diarize's speaker-hint comment claimed the dropped hints were "not a silent
failure". From the caller's side that is what they are, and backend.proto
documents num_speakers as forcing, so the comment now says plainly that the
forwarding is dead for sortformer and that the family which lands must either
honour num_speakers or refuse it. read_audio_file inspects the error_code from
exists(), so an unsearchable parent directory no longer reports as a missing
file. The VAD handler records the stimulus that actually works, since silero
correctly ignores synthetic tones and the next task would otherwise rediscover
that.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): make the lane and identity guards structural
Two hardenings ahead of the eleven handlers still to be written, both of which
get harder to retrofit later.
The lane proof-of-holding parameter was a const reference, which binds to a
temporary, so session_for(rpc, shape, model->acquire(0)) compiled. Each such
temporary dies at the end of its own full-expression, releasing the lane between
two calls that must share one: precisely the split the parameter exists to
prevent, and the form a future author is most likely to reach for because it
reads as tidy. A non-const reference requires an lvalue, so the temporary form
now fails to compile while the named-local handlers build unchanged. The header
comment no longer implies the check is total either: it proves a lane was taken,
not that it is this model's lane.
The identity check was two lines each handler had to remember, with nothing
failing if a new one forgot them and no C++ equivalent of
model_identity_modalities_test.go to notice. snapshot() becomes
snapshot_unchecked(), whose only legitimate caller is Status, since HealthMessage
carries no ModelIdentity. Handlers go through snapshot_for(), which takes the
counted reference, refuses when nothing is loaded, and runs the identity check
before anything can route. Every handler already has to call something to obtain
the model, so the guarded call is now the shortest path and skipping it means
deliberately typing snapshot_unchecked. A convention that has to be remembered
can rot; this cannot.
Verified: the temporary-argument and inverted-order forms each fail to compile
with the expected diagnostic, the real handlers build, and bypassing the guard in
Diarize alone turns the identity test red on that RPC while VAD stays green.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serve the AudioTranscription RPC
Adds result_map, the engine-to-proto boundary, and wires the offline
transcription RPC.
The handler branches on the ROUTED task: for Asr the request's prompt is
whisper-style decoding context and becomes a request option, for Alignment
the same field IS the transcript to align and becomes the text input.
Routing has already decided which.
The result text is TaskResult.text_output verbatim and is never derived
from the segments. audio.cpp carries transcript text in text_output and
nowhere else, so deriving it returns an empty transcript for every
producer that reports segments without word timing. transcript_assembly
already enforces that; this commit's job is not to undo it at the proto
boundary, and result_map_ctest pins it there.
read_audio_file now takes the sample rate the caller needs. Both file-fed
speech handlers ask for 16 kHz mono, for two reasons: silero_vad and
sortformer_diar refuse anything else outright, which turned an ordinary
44.1 kHz upload into INTERNAL, and nemotron_asr emits word timestamps in
its own 16 kHz feature domain whatever the input was, so only a 16 kHz
buffer makes the emitted nanoseconds right. Zero keeps the file's native
rate and channels, which is what source separation will need.
LoadedModel::check_can_serve answers a capability refusal before the lane
is taken and before the input file is read. Routing is a pure read of the
immutable capabilities, so a model that cannot serve an RPC no longer
waits out somebody else's run to say so. VAD and Diarize use it too.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): stop linking sentencepiece's vendored protobuf
engine_runtime links sentencepiece, whose default SPM_PROTOBUF_PROVIDER
builds the protobuf-lite 3.14.0 sources it vendors. The generated
backend.pb.cc is built against the toolchain's protobuf 3.21.12. Both
ended up in the binary: 476 google::protobuf:: symbols came from the
archive, 278 of them also defined by libprotobuf.so, and the archive won,
because once ld pulls a member in for sentencepiece's own code every
reference binds to the definitions that member carries.
The visible symptom is one function.
ParseContext::ParseMessage(MessageLite*, const char*) is what a generated
_InternalParse calls for a submessage field and for nothing else, so flat
messages parsed and nested ones did not: a TranscriptResult carrying
segments serialized to correct bytes that the same process could not read
back, and TranscriptLiveRequest, a oneof of submessages, could not have
been parsed at all. Underneath that, 3.21 generated code was running 3.14
arena, ArenaStringPtr and ExtensionSet code.
-Wl,--exclude-libs does not fix it. It makes those symbols LOCAL in
.dynsym and the parse still fails, because the binding was decided at
static link time and no visibility flag revisits it.
Setting SPM_PROTOBUF_PROVIDER to "package" before add_subdirectory points
sentencepiece at the protobuf the generated code was already built
against. Zero google::protobuf:: definitions remain in the executable
afterwards, every nested message round trips, and citrinet_asr, which
parses a SentencePiece ModelProto at load time and would break first if
this were wrong, still tokenizes and transcribes correctly.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): fix the segment text a transcription response is built from
Segment text is not decoration. core/http/endpoints/openai/transcription.go
routes response_format text, srt, vtt and lrc through
schema.TranscriptionResponse, which builds the entire body out of
Segments[].Text and never reads the top-level text. So for those four
formats the segment text IS the response.
nemotron_asr emits one word_timestamp per SentencePiece token, and the
word boundary is carried as a LEADING SPACE on the piece ("So", "me",
" call"). join_words inserted a space unconditionally, so
response_format=text returned "So me call me na ture ," while the
correct sentence sat unread in the top-level field. The separator is now
chosen from the words themselves: whole words are space-joined, subword
pieces are concatenated, and one leading space anywhere selects the
latter. Concatenating the real nemotron pieces reproduces text_output
exactly, verified end to end.
This does not touch the top-level text, which is still text_output
verbatim. The rule that forbids deriving the transcript from the segments
is about the direction segments -> text; segment text has no source other
than its words.
Two smaller corrections in the same area:
timestamp_granularities ["word"] set only "word_timestamps", a key no
family in the pinned upstream reads. It now sets "return_timestamps",
which qwen3_asr does read and which both runs its forced aligner and
shortens its chunk window, so asking for word granularity no longer
silently returns nothing.
The request-option comment claimed more than it delivered. prompt,
translate and temperature are read by no ASR family, and are forwarded
only so a family adopting them works unchanged; the comment now says so
per key, and gives TranscriptRequest.diarize the same explicit treatment
threads already had.
Also: the shipping target now carries -Wall -Wextra -Wpedantic, which it
never did, so "the build is clean" starts meaning something; and
fill_transcript_result no longer swallows a null response pointer, since
answering OK with an empty transcript is the one failure mode this unit
exists to prevent.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serve the AudioTransform RPC
Covers voice conversion, singing voice conversion, speech to speech and source
separation, the four tasks LocalAI's AudioTransform can represent.
AudioTransformResult carries one dst while htdemucs and mel_band_roformer
produce several named stems from a single run, so inference runs ONCE, every
stem is written as a sibling file <dst-stem>.<name>.<ext>, and params[stem]
selects which one dst receives, defaulting to vocals and falling back to the
first output. An unknown stem name is INVALID_ARGUMENT listing the real stem
names rather than a silent substitution, and the selection happens before the
first write so a refused request leaves no files behind. params[stem] is
consumed here and is not forwarded into the engine's request options.
The stem decision lives in stem_selection, which is stdlib only and therefore
tested by backend/cpp/run-unit-tests.sh. It also validates the names, because
they come from the model (htdemucs reads them from the GGUF's config.sources)
and each becomes a component of a path this backend writes: a name carrying a
path separator would escape the caller's output directory, and two stems
sharing a name would silently overwrite one another.
Both files are read at their native rate and channel count. Separation forces
it, since demucs and roformer refuse any rate but 44.1 kHz and lose the stereo
image that separates a centred vocal from a wide mix. The conversion families
all resample internally (seed_vc, vevo2, miocodec, chatterbox were each
checked), so passing the file through unchanged is also strictly better than
band limiting it to 16 kHz first.
Verified end to end against htdemucs f16 on a 44.1 kHz stereo mix: four stems
plus dst, dst byte identical to the selected stem, params[stem] selecting a
different one, an unknown stem refused with no files written, and mono input
preserved as mono output. Also against miocodec for the single output path,
where params[stem] is refused rather than ignored.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): refuse an impossible stem early, and stop blaming the caller for a failed write
Four fixes from the first review of the AudioTransform RPC.
check_can_serve now returns the resolved route, so params[stem] on a route that
is not source separation is refused from the route instead of after a full
inference: 11 ms rather than the 4.5 s a miocodec conversion costs, and far
worse on seed_vc or vevo2. The post-run refusal stays as the backstop for a
separation-routed family that returns no stems anyway. The typo'd-stem-name
case still needs the run, since no framework header publishes the stem names
before one.
Stem names carrying control bytes are refused. GGUF strings are length prefixed
and demucs reads its sources from JSON, so an embedded NUL survives to here:
two names differing only after the NUL are distinct std::strings, so the
duplicate check passes them, and then path::c_str() truncates both and they
open the same file. That is exactly the silent overwrite the duplicate check
exists to prevent, with the .wav lost as well.
A failed write is now INTERNAL rather than INVALID_ARGUMENT. The destination is
LocalAI's own generated-content directory, not anything the caller named, so a
full disk or a permission fault there is a server fault and is worth retrying,
which is the opposite of what INVALID_ARGUMENT tells a client. An empty output
path stays INVALID_ARGUMENT.
Two comment corrections and one clarification: the separators' required rate is
their checkpoint's declared samplerate rather than a hardcoded 44100, seed_vc
resamples with soxr and falls back to sinc-hann, and the "no files left behind"
guarantee covers a refused request, not a write that fails partway through the
loop.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(audio-transform): stop folding every upload to 16 kHz mono, and name the separation stems
Two defects that made source separation unusable through LocalAI's own API,
even though the backend served it correctly over gRPC.
/audio/transform normalized every upload to 16 kHz mono s16 through
utils.AudioToWav, with no way past it. htdemucs and mel_band_roformer refuse
any rate but their checkpoint's own and separate a centred vocal from a wide
mix using the stereo image, so every separation request through the HTTP API
died with "HTDemucs prepare() sample rate mismatch: expected 44100, got 16000"
while the same call over gRPC worked. The fold is not wrong, it is
backend-specific: LocalVQE's echo cancellation genuinely wants 16 kHz mono and
needs the reference in the same shape. So it becomes a declaration,
BackendCapability.AudioTransformInputMono16k, set for localvqe and for nothing
else. A backend that declares nothing gets its upload unchanged, which means no
backend has to opt in to work. utils.AudioToWavPreservingShape is the
non-folding conversion: a 16-bit PCM WAV passes through byte for byte at any
rate and channel count, anything else is transcoded to WAV with its rate and
channel layout kept.
The other defect is that the run-once stem design bought nothing. A separation
backend writes every stem beside dst from one inference, but AudioTransformResult
carried only dst, so the other three were files no caller could find and a
caller wanting all four had to run four separations. AudioTransformResult grows
a repeated AudioTransformStem, the backend fills it, core/backend validates that
each path really is inside the generated-content directory it handed over, and
the endpoint publishes them as an X-Audio-Stems JSON header beside the existing
X-Audio-Input-Url. JSON because a stem name is the model's own string and could
contain any separator a hand-rolled format would use.
Verified end to end through the HTTP endpoint with htdemucs f16 on a 44.1 kHz
stereo file: 200 with a 44.1 kHz stereo body, all four stems named and fetchable
through /generated-audio/, body byte identical to the selected stem, and
params[stem]=drums returning a different one. The same upload sent to a model
whose backend is localvqe still reaches the backend as 16 kHz mono, confirmed
both by the engine's own rate refusal and by the persisted input file.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(audio-transform): reject extensible WAV from the passthrough, escape stem URLs, convert stems with dst
Four fixes from the second review, plus one bug they made visible.
isPCM16Wav tested only the bit depth, and go-audio's IsValidFile never looks at
the format tag, so a 16-bit WAVE_FORMAT_EXTENSIBLE (0xFFFE) upload was passed
through untouched where the old fold would have transcoded it. audio.cpp's WAV
reader accepts 16-bit only when the tag is 1, so such a file died with
"unsupported WAV encoding". Extensible is what many DAWs and Windows tools
write and music files are this endpoint's new headline input, so it is a
first-contact failure rather than a corner. The check now requires tag 1, with a
spec that fails against the old implementation.
Stem URLs are percent-escaped. A stem name is the model's own string and legally
contains a space, a '#', a '?' or a '%'; an unescaped '#' truncates the URL
before the request is even sent. The name field keeps the raw name.
sample_rate and response_format are applied to the stems as well as to dst.
Applying beat documenting: dst IS one of those stems, so leaving them alone
broke the "dst duplicates the selected stem" invariant the whole design rests
on, and both conversions are no-ops when unset. A stem whose conversion fails is
dropped from the header rather than advertised in the wrong shape.
Verifying that turned up why it had never been noticed: the two fields were
never bound at all. The request arrives as multipart/form-data and echo's binder
falls back to the FIELD NAME without a form tag, matching only
case-insensitively, so "SampleRate" never matched "sample_rate" and "Format"
never matched "response_format". Both were documented in the endpoint table and
silently ignored. Two form tags fix it, and with them the conversion is
observable end to end.
Docs: audio-transform.md now documents what LocalAI does to an upload before the
backend sees it, which backend gets the 16 kHz mono fold and why, params[stem],
and the X-Audio-Stems header with a worked example.
Also records the known limitation that the fold lookup is on the bare backend
name, so pinned variants (vulkan-localvqe) do not match, and points at
IsLlamaCppBackend as the suffix-tolerant precedent.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serve the TTS and SoundGeneration RPCs
TTSRequest.voice is treated as a speaker reference clip when it names an
existing regular file, which makes routing prefer VoiceCloning, and as a named
preset otherwise, in which case it travels as VoiceReference::cached_voice_id.
Both the clip and SoundGenerationRequest.src are read at the file's own rate and
channel count: upstream's own CLI and server do exactly that, every consuming
family resamples internally and mostly with a better resampler than ours, and
ace_step and stable_audio resample their input per channel, so a downmix here
would delete the stereo image they are built to consume.
The request builders live in their own unit rather than in grpc-server.cpp's
anonymous namespace so they can be tested; grpc-server.cpp has a main() and
cannot be linked into a test binary. The option keys are the whole point of
these functions, so each one was grepped against the pinned upstream and the
accounting is written down beside it. instructions maps to "instruct", which is
what upstream's own server maps the OpenAI field to and what qwen3_tts and
omnivoice read, and to "caption" for irodori_tts; the style tag is spelled
"instruct" too, because "instructions" is looked up nowhere. duration maps to
"duration_seconds", read by all three generation families, with the proto's own
name kept only as a forward-tolerant alias. Keys that no family reads say so.
Both handlers answer a capability refusal before taking the lane and before any
file read, so a model that cannot synthesise does not queue behind somebody
else's run to be told no.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): stop emitting an empty style language, and name the missing clip
StyleCondition::language was set whenever has_language() was true, with no
!empty() guard, while the language option twelve lines below had one.
core/backend/tts.go sets Language unconditionally, so has_language() is true on
every request LocalAI sends and carries "" when the caller named none. An
engaged-but-empty style language is worse than an absent one: supertonic reads
text_input->language behind its own !empty() guard and then overrides it from
style->language with no guard at all, so "" replaced its "en" default and its
tokenizer threw "invalid Supertonic language: ". Every /v1/audio/speech request
that set instructions and no language would have been an INTERNAL against a
supertonic model. A plain request never saw it, because the style condition only
exists when instructions are non-empty, which is why the chatterbox end to end
run did not catch it.
TTS also stops discarding the Route that check_can_serve already returns. A
family routed to voice cloning without a reference clip used to be refused from
inside its own prepare(), which meant an INTERNAL naming neither the RPC nor the
field to set; chatterbox advertises clon and no tts, so that was every
preset-only request to it. It is now an INVALID_ARGUMENT naming
TTSRequest.voice, answered in about 4 ms, and it cannot misfire because
has_voice_reference is what selected cloning in the first place. Reading
CapabilitySet::supports_speaker_reference to generalise this stays a follow-up.
The src read carries a written caveat rather than a family blocklist, because
ace_step's editing routes legitimately need src: setting src on a stable_audio
model corrupts the heap and aborts the process in the pinned upstream, and the
only thing keeping that off the network is that
schema.ElevenLabsSoundGenerationRequest has no field for it. Nobody reading that
Go schema would know why, so the reason is recorded where the field is read.
build_tts_shape is extracted so TTSStream cannot describe the same request
differently, and it arrived untested: two mutations of it survived until a
test_tts_shape case was added.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serve the TTSStream and AudioTranscriptionStream RPCs
TTSStream leads with a streaming WAV header carrying 0xFFFFFFFF sizes, matching
the convention backend/go/vibevoice-cpp established, so an HTTP client can start
playback before the full PCM exists. Its chunks are read from
StreamEvent::named_audio_outputs and not audio_output: supertonic, omnivoice and
voxcpm2 all put their streamed audio there and leave audio_output empty until
the very end, so reading the obvious field yields a stream with no audio in it.
The finish_stream result is the family's own merged whole rather than a tail, so
it is emitted only when nothing was streamed.
Streaming transcription sends incremental deltas and degrades to a single delta
plus the final result on families that offer no streaming ASR, which is the same
message sequence with fewer deltas. The four streaming ASR families disagree on
what partial_text means: nemotron_asr, vibevoice_asr and higgs_audio_stt report
incremental fragments while voxtral_realtime reports the whole hypothesis and
reports it twice, so the reconciliation lives in one tested unit rather than in
the handler. nemotron_asr reports only through the stream event sink, and only
from inside finalize, so the audio driver installs one and clears it again
before returning: the session is cached and a sink left holding the caller's
frame is a use after free waiting for the next stream.
begin_stream is now the only implementation of the streaming state obligation,
prepare then start_stream. Streaming sessions are cached, and what clears the
previous stream is start_stream's reset; a family override that dropped it would
break every call site with no compile error, so there is one call site.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): keep streaming deltas on UTF-8 boundaries, refuse dtypes that abort
TranscriptStreamResponse.delta is a proto3 string, whose wire format requires
valid UTF-8. voxtral_realtime reports its hypothesis as a concatenation of raw
token BYTES (tokenizer_text.cpp:171-183), so the cumulative difference between
two consecutive reports is eventually a lone continuation byte, and the C++
runtime serializes that with only a logged warning while the Go runtime refuses
to unmarshal it: the client loses the remaining deltas AND the final_result.
Measured on a trace of a non-ASCII sentence, 11 of 31 messages failed to
unmarshal and every accented character was lost. TranscriptDeltaTracker now
holds back an incomplete trailing sequence and merges it into the next fragment;
reconcile flushes it, which it always can because the final text is complete.
The same trace now unmarshals in full with zero failures.
A streaming buffer whose float count is not a whole number of frames is refused
rather than truncated. The integer division dropped the tail floats from the fed
audio and therefore from the transcript, with no diagnostic; vibevoice_asr
refuses the same thing from the other side of the call.
A supertonic GGUF whose weights are not f32 is refused at load. It reaches
ggml_concat with mismatched operand types and ggml_abort takes the whole backend
process down on the first request, so nothing downstream can report it: the
model loads, then every request kills the process. Attributed rather than
assumed, the unary TTS path aborts identically, and upstream records that
package as untested. The refusal names the orig package and says what to run
before deleting the guard.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): stop a repeated lead byte from orphaning the next delta
The first UTF-8 fix closed the cumulative half only. Rule 2 discards a fragment
the known text already starts with, and when that fragment is the LEAD BYTE of a
new character it looks exactly like a repeat of an older character beginning
with the same byte. It was discarded rather than held, its continuation bytes
then arrived alone and began the next delta, and utf8_complete_prefix_length
only ever inspected the trailing sequence, so a delta invalid at the FRONT went
out whole. Through a real Go proto.Unmarshal the review's four-character repro
gave 3 deltas, 2 unmarshal failures and a lost transcript.
Reachable from the incremental families, not only from voxtral: nemotron_asr's
decoder cuts at a byte offset and vibevoice_asr's common_prefix_size compares
bytes, so both split characters. Measured over 30,000 randomized incremental
traces, 53.28% of Japanese traces and 9.52% of French ones carried at least one
delta the Go runtime refuses.
Two changes. Rule 2 no longer judges a fragment that ends mid-character, so the
lead byte is held instead of swallowed and the character survives intact; the
cost is a few duplicated bytes in a shrinking cumulative report, which no pinned
family produces. release() additionally drops leading orphan continuation bytes,
so no delta can begin mid-character whatever the rules above it decide. Losing a
byte keeps the stream alive; emitting one ends the RPC and takes the
final_result with it.
Post-fix all 60,000 traces produce zero unmarshal failures, and the cumulative
streams plus both pure-ASCII incremental streams are byte-identical to the
previous commit, so nothing changed for the families already working.
The weight-dtype allow list moves to family_gate, where it is stdlib-only and
pinned by a test rather than only by a comment. Two comment citations corrected.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): read only an exact repeat as a repeat, not any prefix
Rule 2 discarded any partial the known text merely started with. For a cumulative
family that is a duplicate; for an incremental family it is an ordinary short
fragment that happens to coincide with the start of the transcript, and it was
dropped, silently corrupting the text. Pure ASCII, no multi-byte character
anywhere: the fragments "pure ", "ascii ", "trans", "c", "ri", "p", "t" left the
client holding "pure ascii transcrit". Over 5,000 randomized traces per
transcript, 9.50% of pure-ASCII and 29.12% of French traces ended with the client
holding something other than final_result.text, with a 200 and no diagnostic.
Both incremental families emit fragments that small routinely, since nemotron_asr
cuts at a byte offset and vibevoice_asr at a common prefix.
Narrowing rule 2 to an exact repeat drives that to zero on all six transcripts
and changes no cumulative stream at all: 30,000 randomized cumulative traces are
byte-identical to the previous commit.
What rule 2 guarded was established from upstream rather than from its own
comment. The only duplicate any pinned family produces is voxtral_realtime's,
where process_available_stream_chunks feeds each event to the sink from inside
its loop and returns the last of the batch, so that event arrives twice with
byte-equal text. A duplicate is an exact repeat, so equality still covers it. The
case given up is a cumulative report that SHRINKS, which no pinned family can
produce: voxtral decodes a token vector that is only push_back'ed and cleared by
reset(), so within a stream it can only grow.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serve the AudioTranscriptionLive RPC
The one bidirectional stream this backend serves. The client sends a
TranscriptLiveConfig, then TranscriptLiveAudio frames; the server acknowledges
with ready, emits deltas as the audio arrives, and sends final_result once the
read side closes. There is no offline fallback: live transcription has to
consume audio incrementally, so a family with no streaming ASR is refused
rather than served a batch run, which is what this RPC's Streaming-only
mode_candidates list already says.
The driver is a new sibling of run_streaming_audio, run_streaming_live, because
the audio does not exist yet: instead of slicing a buffer it pulls frames from
the caller until the read side closes. It installs the same ScopedStreamSink in
the same order, which is not optional, since nemotron_asr returns a bare event
from process_audio_chunk and reports every partial through the sink from inside
finalize(). It buffers the wire's frames up to the family's own preferred window
rather than feeding whatever size the client's audio callback produced, and it
does not call finish_stream at all when no audio arrived, because nemotron_asr
throws "finalize requires streamed audio" and an empty transcript is the
truthful answer to transcribing nothing.
Three things the handler had to get right and one it cannot:
- The audio contract. A live request carries no samples, but nemotron_asr's
streaming prepare() throws without an audio contract, and
build_preparation_request derives it from TaskRequest::audio_input, so that
field is an EMPTY buffer holding only the rate and the channel count.
- 16 kHz or a refusal. The families express their spans in their own 16 kHz
feature domain whatever the input was, and live frames cannot be resampled
on the way in the way a file can, so an 8 kHz session would return
timestamps 2x off with a 200. core/backend hardcodes 16000 anyway.
- A mid-stream Config is refused. backend.proto calls it a decoder reset, but
deltas already on the wire cannot be retracted, so a reset would leave the
final text contradicting the transcript the client assembled. Ignoring the
message would hand a client that believes it reset the decoder a transcript
that silently continues the audio it thought it discarded.
- The stale-route identity check cannot run here: TranscriptLiveRequest
carries no ModelIdentity in either arm of its oneof, so snapshot_for does
not instantiate for it. snapshot_unchecked's comment now names that as a
second legitimate class of caller and says the fix is a proto change.
eou and eob stay false. They exist for cache-aware models that emit
end-of-utterance and end-of-backchannel tokens; audio.cpp's StreamEvent has no
equivalent signal, and a client uses eou to decide the speaker yielded the turn,
so a guess inferred from silence cuts people off mid-sentence.
The lane is held for the whole stream, which is as long as the user keeps
talking: the streaming session is stateful and cached, so a concurrent run would
interleave two callers' audio and corrupt both transcripts.
Verified against nemotron_asr over a real connection with a 14 s WAV in
512-sample frames: ready first, 59 incremental deltas with no repeated prefix,
concat(deltas) equal to final_result.text, word timestamps in nanoseconds, eou
and eob false. citrinet_asr answers UNIMPLEMENTED naming the family and listing
asr/offline. A config followed by a close returns an empty final_result rather
than hanging, and a first message that is not a config is INVALID_ARGUMENT. Two
concurrent streams both return the complete transcript.
Two cleanups on lines Task 12 touched, folded in. The DtypeAllowList terminator
is now asserted at compile time: the reported out-of-bounds read did not exist,
the single entry does terminate, but the loops have no other bound and any edit
that widened an entry would walk off the end. And the dtype guard now
short-circuits on "is there a table entry" through a new predicate rather than
on the emptiness of the description string, which would have skipped the check
on an entry with an empty allow list, i.e. on precisely the entry that refuses
every dtype.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): bound the lane a live stream can hold
AudioTranscriptionLive holds the model's inference lane for the whole stream,
which is correct (the streaming session is stateful and a concurrent run would
interleave two callers' audio) and newly dangerous. Every other RPC holds the
lane across compute, or across a write to a slow reader, and both of those
terminate on their own. A live stream instead blocks in a client-driven read,
and a peer that goes silent WITHOUT closing the stream never terminates
anything: the lane stays taken and every other request against that model queues
behind a client that stopped speaking.
live_watchdog is a one-shot idle timer that ends the stream when no frame has
arrived inside a window. It is standard library only, so it is unit tested
without an engine. gRPC's synchronous Read has no timeout and cannot be given
one, so the only way to unblock it is ServerContext::TryCancel, which decides
the wire status itself: the client sees CANCELLED rather than the
DEADLINE_EXCEEDED the handler returns, the reason is logged, and the lane coming
back is the point. When it fires the read loop throws rather than reporting
end-of-input, so the driver does not go on to finalize a decode nobody is
waiting for.
It is armed only after the lane is taken and disarmed as soon as the read side
closes, and both ends matter. Arming earlier would cover acquire(), which
legitimately blocks while another live stream runs, so a queued caller would be
cancelled for waiting its turn. Disarming later would cover our own decode,
where a window overrun is not a peer going quiet and cancelling would throw away
the transcript the client is waiting for.
The window is the new live_idle_timeout_ms option, 30 s by default, 0 meaning no
limit. core/http/endpoints/openai/realtime.go drives a 300 ms ticker and feeds
every tick that produced new audio while a turn is open, so 30 s of silence is a
hundred ticks that delivered nothing. It is also longer than any pause a speaker
takes mid-utterance, which is the case that must never be cut off, and
backend.proto lets one stream span many utterances, so a client that pauses
longer between them raises the option rather than discovering it.
Two smaller corrections in the same handler:
- check_can_serve now runs BEFORE the sample rate check.
pkg/grpc/grpcerrors/errors.go degrades to the file path on UNIMPLEMENTED and
on nothing else, so a live-incapable model asked at a wrong rate was
answering INVALID_ARGUMENT and costing the caller its fallback.
- a negative sample rate is refused instead of silently becoming 16000. Zero
still means 16000, which is what the proto documents; -1 is malformed rather
than absent and gets the same refusal every other bad rate gets.
And one thing recorded rather than changed, at the handler: "live" here means
incremental INPUT, not low latency, and with the pinned families it does not yet
mean incremental OUTPUT either. nemotron_asr's process_audio_chunk only appends
to its buffer, so its whole decode and every delta happen inside finalize(),
after the client closes its send side. The policy-window buffering is inert for
that family and matters only for vibevoice_asr and higgs_audio_stt.
Verified on the wire with live_idle_timeout_ms:3000. A silent client acked at
371 ms and was cancelled at 3.371 s; a second live stream opened one second
later received its ack 2.37 s in, i.e. at the instant the first was cancelled,
and then transcribed successfully on the same cached session. Without the
watchdog it would still be waiting. Re-ran the live transcription (ready first,
59 incremental deltas, concat equal to the final text, word timestamps in
nanoseconds, eou and eob false), the citrinet refusal at both a right and a
wrong rate (UNIMPLEMENTED either way now), and Task 12's AudioTranscriptionStream
on nemotron_asr, which is unchanged.
Mutation testing the watchdog found a weakness in its own test: the destructor
test slept past the window inside the watched scope, so a destructor that
DETACHED the thread instead of joining it passed unnoticed. The test now uses a
window longer than the scope, which kills that mutant, and says why.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): refuse the unsupported RPCs with a reason
AudioEncode, AudioDecode, AudioTransformStream, AudioToAudioStream and
VoiceEmbed have no counterpart in audio.cpp's VoiceTaskKind. Each now returns
UNIMPLEMENTED naming the loaded family, what that family does support, and the
upstream limitation, instead of the generated base class's bare status. The
reasons live in a table in capability_routing.cpp so they are data rather than
literals copied into five handlers, and so a test can assert every one of them.
The five claims this was planned against were re-read at the pinned upstream
e800d435d130dc776baf6f3e6129bb62b1495c89, and one did not hold. "audio.cpp
streams tts and asr only" is false: silero_vad advertises vad with
RunMode::Streaming. The refusal stands on the narrower claim that survives, that
no family advertises streaming for any task AudioTransform routes to, and a test
asserts the refuted wording does not come back.
VoiceEmbed is the one refusal whose request carries a ModelIdentity, so it runs
the #10952 check before answering: a stale route must get NOT_FOUND and the
router's sentinel, not "audio.cpp cannot embed speakers" about a model that is
not loaded here. It cannot use snapshot_for, whose no-model branch would tell
the caller to load a model when no model can help, so it takes the reference
through snapshot_unchecked and checks identity itself. That function's comment
now names three classes of caller instead of two.
The two bidirectional surfaces refuse without reading their stream, verified
with a client that writes a config and eight frames first and gets the status
rather than hanging.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): correct the vevo2 clause, and assert the absences
Review found a false clause in the AudioToAudioStream refusal. It said s2s is
"offline voice conversion ... which converts one clip into another speaker's
voice", which is true of miocodec and false of vevo2: vevo2's s2s route is
`editing` and only `editing` (default_route_for_task and route_matches_task in
src/models/vevo2/session.cpp), documented as "Edit source speech into new target
text while using the target voice" and requiring --target-text, so it rewrites
what was said. vevo2's voice conversion is its separate vc task. It now reads
"offline clip-to-clip processing against a target voice, declared only by
miocodec (voice conversion) and vevo2 (speech editing)", and a test asserts the
miscast cannot come back. The conclusion is unchanged: neither family converses.
That defect was undetectable on the wire, since vevo2 does not load here, which
is the argument for upstream_absence_ctest.cpp. It links engine_runtime purely
to interrogate make_default_registry() and asserts the five premises the refusal
reasons rest on: no codec task kind, no family advertising spk, no streaming for
sep/vc/svc/s2s, miocodec advertising exactly vc and s2s, and s2s advertised by
exactly miocodec and vevo2. The last two are exact sets, so an addition fails
here rather than leaving a message stale. A positive control proves the registry
is populated and the query works before any absence is believed, and every
assertion has a reproduced negative control. This turns an AUDIO_CPP_VERSION
bump from "remember to re-read five prose paragraphs" into a test failure.
unsupported_surface now switches over UnsupportedRpc with no default label, so
-Wswitch reports a sixth enumerator added without a row at build time; the
runtime bounds guard it replaces is deleted.
The AudioTransformStream reason had a true premise and an overreaching
conclusion: an offline sep family could be buffered into a stream, as other
LocalAI backends do. It now says this backend declines to offer a buffered
offline call in disguise, rather than implying impossibility.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): make the missing-switch-case diagnostic fatal
unsupported_surface() switches UnsupportedRpc onto the table row that explains
it, with no default label, so -Wswitch reports an enumerator nobody handled. As
a warning that is not enough: adding a sixth enumerator and building the shipping
target gives exit 0, a binary and one warning, and the trailing
`return surfaces[0];` then answers the new RPC with AudioEncode's codec reason.
That is a confident, specific and false statement about audio.cpp on the wire, on
the one code path whose entire job is to be truthful about what this backend
cannot do, and it is worse than the runtime fallback it replaced, which at least
named itself as a bug in this file.
capability_routing.cpp therefore joins loaded_model.cpp on the existing
-Werror=switch pin, whose comment already made this argument for the engine enum.
The comment now covers both files. The pin stays per-file rather than
project-wide because upstream's own ace_step/vae_decoder.cpp has unhandled
-Wswitch cases of its own.
Verified: a sixth enumerator now fails `make grpc-server` with exit 2 and no
binary; appending a 14th VoiceTaskKind upstream still fails loaded_model.cpp, so
the two pins fire independently; both reverted clean.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): package the backend image
Bundles the dependency closure for the from-scratch image, the dlopened ggml
CPU-variant shared objects that ldd cannot see, and upstream's bundled
silero_vad and marblenet_vad assets so VAD works with no download.
The bundled loader sits in the package ROOT rather than at lib/ld.so. run.sh
execs it, which makes /proc/self/exe name the loader, and this backend has two
consumers of that path: ggml discovers the libggml-cpu-*.so by listing
dirname(/proc/self/exe), and resolve_model_path expands bundled:<name> under the
same directory. Rooting the loader makes the binary, the ggml objects and
assets/ share the one directory all three resolution mechanisms agree on.
llama-cpp's lib/ld.so layout would need assets/ moved into lib/ as well.
The image builds against apt gRPC and protobuf, like Dockerfile.ds4 and unlike
Dockerfile.privacy-filter. The from-source gRPC that install-base-deps.sh and
the base-grpc-* images supply vendors protobuf 26, which pulls abseil into
message_lite.h; with SPM_PROTOBUF_PROVIDER=package that collides with
sentencepiece's vendored mini-abseil and every absl::internal reference becomes
ambiguous. Noble's protobuf 3.21.12 predates the abseil dependency and is the
pair every earlier verification of this backend ran against.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): exempt the driver libraries from the packaging gate
package.sh already left libcuda.so* and libnvidia-* to the host when copying,
because the driver has to match the kernel module on whatever host runs the
image, but the validation gate had no matching exemption. With BUILD_TYPE=cublas
ggml is static and links CUDA::cuda_driver, so grpc-server carries DT_NEEDED
libcuda.so.1 and the gate would have rejected the very absence the copy loop
created, failing every cublas build in CI. One regex now feeds both.
Building a control for that found a second defect: ld.so --list refuses to trace
an object with an unresolvable dependency at all, exiting 127 without emitting a
per-library line, so the "=> not found" rule was dead code and no exemption could
have applied to it. The gate now traces with LD_TRACE_LOADED_OBJECTS and
LD_LIBRARY_PATH, which reports the missing name and exits 0, and which is also
what run.sh does at run time.
Adds a layout assertion so a future move of the loader into lib/ fails the build
instead of shipping a package that resolves bundled: models into lib/assets and
finds no ggml CPU backend, and records for Task 16 that the Darwin script must
not be a straight copy of privacy-filter-darwin.sh, which never calls package.sh
and would silently drop assets/.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): register the backend with CI and the gallery
Adds the five Linux matrix entries (cpu amd64/arm64 sharing a tag-suffix so the
manifest merge fires, cuda 12, cuda 13, vulkan), the path-filter case that keeps
later PRs touching backend/cpp/audio-cpp/ from getting zero CI jobs, the
bump-bot entry pointing at the AUDIO_CPP_VERSION pin in the backend Makefile,
the gallery meta plus its -development variant and the image entries for every
variant, and the Makefile docker-build wiring.
The matrix entries carry base-image only, with no builder-base-image, unlike
the llama-cpp and privacy-filter blocks they sit next to. The prebuilt
quay.io/go-skynet/ci-cache:base-grpc-* images ship a from-source gRPC whose
protobuf v26 depends on abseil, and this backend's sentencepiece is built with
SPM_PROTOBUF_PROVIDER=package, so it sees real abseil's absl::lts_20240116::
internal alongside its own vendored plain absl::internal and every
absl::internal:: reference becomes ambiguous. Building against base-grpc-amd64
fails at sentencepiece-static.dir/error.cc.o with "reference to 'internal' is
ambiguous". Dockerfile.audio-cpp installs apt's gRPC/protobuf 3.21.12 itself,
which is also the pair every unit and end-to-end run of this backend has been
verified against, and the CUDA toolkit therefore has to come from base-image.
No Darwin matrix entry and no metal gallery entries: the Metal build needs
scripts/build/audio-cpp-darwin.sh, a backends/audio-cpp-darwin make target and
a routing step in backend_build_darwin.yml, none of which exist yet, so an
entry added now would be routed to build-darwin-go-backend and look for
backend/go/audio-cpp/. The inferBackendPathDarwin case and the
DARWIN_BESPOKE_BUILDERS membership are in place, inert, so that adding the
entry later is a one-line change that cannot be claimed by the generic Go path.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): pin the CUDA architectures, drop the vulkan variant
Upstream sets CUDA_ARCHITECTURES to `native` on the engine_runtime target
whenever CMAKE_CUDA_ARCHITECTURES is unset at root scope, and docs/build/
linux.md says so outright. ggml's own default does not rescue it: it
list(APPEND)s in the ggml subdirectory scope, which never reaches the root
scope where the engine_runtime property is decided. No CI runner has a GPU for
`native` to enumerate, so both cublas entries would have gone red on the very
commit that first turns a CUDA build on.
Pin the list in backend/cpp/audio-cpp/Makefile, selected by CUDA_MAJOR_VERSION,
which Dockerfile.audio-cpp now forwards from the CI build-arg it was previously
discarding. The values are copied from ggml's own version guards rather than
invented, so engine_runtime and ggml compile for the same set: CUDA 12 keeps the
Maxwell/Pascal/Volta virtual archs and stops at 120a-real, CUDA 13 drops them
and adds 121a-real. The `a` suffix is used rather than `f` because the latter
needs CMake 3.31.8 and Ubuntu Noble ships 3.28.3. Verified by driving CMake
3.28.3's own CUDA architecture validator over both lists, with 120f-virtual as
the rejected control.
Drop the vulkan matrix entry, its two gallery entries, the vulkan capability
key on both metas and the Vulkan tag. Every other vulkan backend gets its Mesa
ICD drivers from .docker/install-base-deps.sh, which package-gpu-libs.sh then
bundles; Dockerfile.audio-cpp calls neither and installs only libvulkan-dev and
glslc, so the image would ship a Vulkan loader that finds no GPU. No CI job runs
a vulkan image against real hardware, so that would have passed green and failed
in users' hands. BUILD_TYPE=vulkan stays supported for local builds.
Also note on the cublas entries that cuda-major-version now selects the
architecture list and that cuda-minor-version and the base-image tag encode the
same toolkit, and correct the stale entry counts on matrixEntryKey.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): build for Darwin Metal
Bespoke C++ Darwin path like ds4 and privacy-filter: an includeDarwin matrix
entry, a backends/audio-cpp-darwin make target, a gated workflow step, and the
metal image entries plus metal/metal-darwin-arm64 capability keys in the
backend gallery.
The build script deliberately does NOT reassemble the package the way
privacy-filter-darwin.sh does. It runs the backend's own `make package` and
copies the result, so the Darwin package keeps the root-level layout the Linux
one has: grpc-server, run.sh, the ggml objects and assets/ in one directory,
with lib/ for the dylib closure. Hand-assembling would drop assets/, and
assets/ is what makes the bundled: model paths resolve with nothing downloaded.
The dylib walk is a full transitive closure rather than the single level ds4
and llama-cpp do, because Homebrew's grpc++ pulls libgrpc, abseil, upb, cares
and OpenSSL that grpc-server does not link itself, and a level-1 walk ships a
package that only works on a machine that already has Homebrew grpc.
Two fixes folded in, both in the backend Makefile:
- an EMPTY CUDA_MAJOR_VERSION fell through to the CUDA 12 architecture list,
which contains 120a-real and so needs nvcc >= 12.8. A local
BUILD_TYPE=cublas build on a 12.0-12.7 host failed to compile where
upstream's documented default (native) worked. EMPTY now maps to native,
12 and 13 keep their lists, and any other non-empty value is an error on
cublas builds. CI always passes a major, so CI is unaffected.
- the Darwin branch now points CMake at Homebrew's keg-only libomp. AppleClang
ships no OpenMP runtime and nothing is symlinked into /opt/homebrew, so
FindOpenMP finds neither the library nor the header, and audio.cpp calls
find_package(OpenMP REQUIRED) whenever ENGINE_ENABLE_OPENMP is on. Without
the hint the macOS build would have died at configure time. If the keg is
absent the build disables OpenMP instead of failing.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): make the Darwin fallbacks loud and the rpath walk complete
Review follow-up on the Darwin Metal build.
The OpenMP fallback was silent. If brew --prefix libomp ever comes back empty,
CI produced a green Metal package with 108 #pragma omp directives across ~30
files compiled out, and clang says nothing about an ignored omp pragma without
-Wsource-uses-openmp, so the only trace was one absent flag inside a set -x
cmake line. That regression would have been blamed on Metal. It now warns.
The @rpath arm of the dylib walk had no live candidate when it was written, on
the reasoning that a Metal build links ggml statically. The OpenMP fix in the
same commit made libomp.dylib one, and whether Homebrew records it as an
absolute opt path or as @rpath/libomp.dylib is not observable from Linux. The
walk now expands @rpath, @loader_path and @executable_path against the object's
own LC_RPATH entries, and only fails when nothing on disk answers, printing the
rpath list with the error so a failure on a machine nobody can attach to
explains itself.
Also: ADDITIONAL_LIBS now go through the closure rather than a bare cp, so they
are deduplicated and their own dependencies bundled; build/darwin/lib is
created explicitly instead of relying on package.sh pre-creating it; the libomp
probe uses nested ifneq rather than $(and ...), which needs GNU make 3.81 and
would otherwise expand empty and take the OFF branch on an older make; and
-DOpenMP_ROOT is quoted like its CUDA sibling.
Verified with a Linux harness that runs the script verbatim against a stubbed
otool: a level-2 transitive dep, an @rpath dep reachable only through LC_RPATH,
and an ADDITIONAL_LIBS dep are all bundled, a dependency cycle terminates,
system libraries are skipped, the packaged tree has assets/ at the root beside
grpc-server with the dylibs in lib/, and both failure paths exit non-zero.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): make bundled: reachable from a model YAML
resolve_model_path() tested the bundled: prefix on `candidate`, which prefers
ModelFile and falls back to Model. LocalAI fills ModelFile by joining ModelPath
onto the configured model string (pkg/model/loader.go, LoadModelWithFile), and
only sets it from a managed artifact otherwise, so a model YAML saying
`model: bundled:silero_vad` arrives as ModelFile "/models/bundled:silero_vad"
and Model "bundled:silero_vad". The prefix therefore never matched through the
normal load path: it matched only for a hand-written LoadModel call that left
ModelFile empty, which is exactly how task 15 verified it, and every model YAML
using the form failed with "model path does not exist:
/models/bundled:silero_vad".
Both fields are now checked, Model first, so the zero-download VAD path the
package ships assets for is reachable the way it is documented. A caller that
puts the form in ModelFile still works, so task 15's verification stands.
Compiled clean; the runtime check could not run on this host, whose system
libprotobuf/libre2 have gone missing (the pre-existing grpc-server binary no
longer resolves its libraries either), so it wants a container run.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): advertise the backend and document its options
Registers audio-cpp as preference-only in /backends/known: the family lives in
GGUF metadata that an importer cannot read from a remote repo, and one repo
hosts thirty families, so there is no honest auto-detect signal. Modality is a
single string and the import form chips on a fixed key set, so it registers as
tts with the other modalities named in the description rather than under an
invented key the UI would bucket as "other".
Adds a features page covering the option namespacing, the routing table per
endpoint, the RPCs this backend declines and why, the bundled VAD path, the
separation stem behaviour, and the family gotchas (supertonic needs the orig
package; chatterbox advertises cloning and no plain tts; nemotron_asr defers
its whole decode to finalize so live transcription emits nothing until the
client half-closes, unlike higgs_audio_stt and voxtral_realtime). Every option
name and family capability in it was read off the pinned upstream checkout.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): test resolve_model_path, and correct the family names
The bundled: fix in 842443cd7 shipped without a test, which is how the bug got
there: task 15 verified the form with a hand-written LoadModel that left
ModelFile empty, and that is the one shape the server never produces. Four cases
in streaming_driver_ctest, which already links loaded_model.cpp, pin the
PRODUCTION shapes instead. The first fails against the pre-fix source (returns
the joined /models/bundled:silero_vad); the other three are the branches the
bundled: lookup now runs in front of and must fall through for.
Three family names in the docs were the source directory rather than the
registered family, on pages whose whole argument is that these names cannot be
guessed: demucs is htdemucs (demucs/loader.cpp:22), roformer is
mel_band_roformer (roformer/assets.h:15), and moss is TWO families,
moss_tts_local and moss_tts_nano. The hyphenated ASR names are underscored to
match, here and in the compatibility table.
The supertonic dtype note claimed more than the evidence carries. The f16 abort
is a local observation, identical through TTS and TTSStream; upstream's
docs/gguf.md leaves the 16-bit column untested and records q8_0 as "No
(unsupported weight dtype)", which says unusable rather than fatal. Both are
still refused, because the allow list is what the family can run. Corrected in
family_gate.h, family_gate.cpp and the docs together, since the docs inherited
the wording from the code.
The importers tripwire says in the file that it is a tripwire: it exercises no
audio-cpp behaviour, and the registration assertion lives in backend_test.go.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* gallery: add audio.cpp models covering every served RPC
One representative model per RPC group of the audio-cpp backend, plus the two
bundled VAD models, which need no download at all because the assets ship
inside the backend package.
Every hash was computed with sha256sum on the downloaded file. Quantizations
come from upstream's tested-status table in docs/gguf.md rather than a default
of q8_0: supertonic ships the orig package (its q8_0 is recorded as an
unsupported weight dtype and its f16 aborts in ggml_concat), and nemotron_asr
and htdemucs ship f16 because their q8_0 builds are recorded with drift while
16-bit is a clean pass.
Diarization and separation use the diarization and audio_transform usecases,
not transcript: /v1/audio/diarization and /audio/transform filter the default
model on FLAG_DIARIZATION and FLAG_AUDIO_TRANSFORM respectively, so a
transcript flag would have hidden both models from their own endpoints. The
forced aligner sets parameters.language, which the transcription endpoint uses
as the fallback when no language form field is sent, because the family
requires both a transcript and a language.
All ten entries were run twice: once against the raw gRPC server, and once
installed with local-ai models install and called through the HTTP endpoint.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* gallery: correct the audio.cpp entries' licenses
Swept all ten entries against the real upstream named in audio.cpp's
tools/model_manager.py rather than against the audio.cpp repo's own license.
Three were wrong:
supertonic apache-2.0 -> openrail weights come from
mlx-community/supertonic-3-mlx, and
both it and Supertone/supertonic are
openrail
citrinet apache-2.0 -> other pulled from NGC
nvidia/nemo/stt_en_citrinet_256,
governed by the NGC Terms of Use
sortformer other -> cc-by-nc-4.0 nvidia/diar_sortformer_4spk-v1 is
CC BY-NC 4.0, and the gallery already
uses that exact string, so there is no
reason to obscure a non-commercial bar
The license field is one word, so citrinet and sortformer also gained a
sentence saying why they are restricted. The other seven were confirmed
correct against their sources.
Also drops an unverified claim from the nemotron description. It said the
model drives the realtime transcription session; that endpoint actually calls
TranscribeStream, and the live RPC reaches LocalAI only through
realtime_semantic_vad.go. Neither path was exercised here, so the description
now states only the two calls that were.
MarbleNet gains the NeMo upstream under urls: for parity with silero.
No sha256, quantization, usecase or model choice changed.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(audio-transform): bound sample_rate, keep same-named uploads apart
Four defects the whole-branch review found on the Go side, plus two comment
corrections.
sample_rate is a disk-exhaustion hazard. The branch added the `form:` tag that
makes the field bind for the first time, so the resample path went from dead to
live, and utils.AudioResample interpolates the int straight into ffmpeg's -ar
with no bound. Measured with ffmpeg 7: -ar 999999999 on a 0.01 s clip writes
20 MB and exits 0, which scales linearly to the reported 3.9 GB for one second,
into a GeneratedContentDir nothing sweeps, and convertStems repeats it once per
separation stem. Clamped to 8000..192000 in the handler, before the temp dir and
before the model is touched, and rejected with a 400 outside it.
The low end was reported as "a 0-byte file". It is not: -ar 1 writes a 78-byte
header with no audio behind it, whose declared data size still claims 70 bytes,
so go-audio parses it as a 35 SECOND file and a size check does not see it. The
guard therefore compares the declared data chunk against the bytes actually on
disk, and AudioResample now fails rather than returning a WAV carrying nothing.
Both parts of a transform request land in one temp dir, and the raw copy was
named only after the client's basename, so `-F audio=@mic/clip.wav
-F reference=@loopback/clip.wav` wrote "raw-clip.wav" twice. Since
AudioToWavPreservingShape hardlinks an already-PCM16 WAV rather than copying it,
the reference part's os.Create truncated the inode audio.wav pointed at: mic and
reference came out identical, which makes an echo canceller null everything and
return near-silence with a 200. The raw copy now carries the form field name.
audio-cpp had no BackendCapabilities entry, so VoiceCloningForModel returned nil
before it ever consulted the model's tts.voice_cloning override and every
`voice: "profile:<id>"` request was refused with a 400, on a backend that ships
audio-cpp-chatterbox whose family serves cloning and not plain TTS. Registered
with its RPCs, usecases and the reference-audio contract, and deliberately
without the 16 kHz mono fold, which its separation families cannot survive.
GetBackendCapability was exact-match only, so every pinned gallery variant read
as an unknown backend: vulkan-localvqe lost the 16 kHz mono fold that used to be
unconditional and started failing inside LocalVQE, and the usecase gate does not
stand in for it because BuildFilteredFirstAvailableDefaultModel returns early
once the client names a model. Lookup now falls back to the meta name by
stripping the gallery's hardware prefix and release-channel suffix, exact match
first so nothing can be shadowed. Same class as #10945.
Also corrected: the AudioTransformRequest comment claimed echo's binder falls
back to the field name, which it does not in either direction (bindData binds
ONLY tagged fields and `continue`s otherwise; `model` arrives from
setModelNameFromRequest's c.FormValue). And the stable_audio `src` heap
corruption caveat now lives on ElevenLabsSoundGenerationRequest, where the Go
developer who would add the field can see it, instead of only in C++.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(audio-cpp): refuse a task pin the RPC cannot serve, and stop empty frames holding the lane
The model's `task:` option is copied into the request shape by all nine
handlers, which is correct, but resolve_route then replaced the RPC's candidate
list with the pin WHOLESALE and never asked whether the pin was something that
RPC routes to. One pin therefore bled across all nine surfaces, and because the
family still supported the pinned task the result was a wrong 200 rather than an
error. Reproduced live: nemotron with task:asr made Vad return 200 with zero
segments after a full ASR decode, so 14 seconds of speech was reported as
silence, and Diarize did the same; silero_vad with task:vad made
AudioTranscription return 200 with empty text and four segments whose spans were
VAD segments, which combined with response_format in {text,srt,vtt,lrc} building
the body solely from Segments[].Text yields a well formed SRT of four timed
EMPTY cues. It also contradicted the documented contract, that a family which
cannot serve a request is refused rather than rerouted.
A pin is now checked against the RPC's admissible task set before it is adopted,
and the refusal names both the pin and the RPC. The set is derived from
task_candidates with every shape flag set rather than restated, so a task added
to an RPC's candidates cannot become inadmissible by omission. Every legitimate
pin survives, and the test asserts all fifteen of them alongside the eight
crossings that must not.
The live watchdog was defeated by empty frames. idle.touch() ran on ANY message,
before the has_audio and pcm.empty() filters, so a peer writing unset-oneof or
zero-length frames faster than the window held the lane indefinitely while
feeding the decoder nothing. There is one lane per model and one model per
process, so that is a single client denying the whole backend, which is what the
watchdog exists to prevent, and the thrown text already said "no audio frame
arrived". The touch moved below the filters, which are now a named predicate so
the distinction is testable rather than a call order nobody can see.
Three comments corrected against measurement rather than reasoning:
- CMakeLists claimed zero google::protobuf:: definitions remain in the
executable. nm -C --defined-only reports 2515, and that is expected: they are
generated code, sentencepiece::ModelProto's own _InternalParse among them. The
claim that holds, and the one the ABI fix is actually about, is that no
vendored protobuf RUNTIME is linked and ParseContext::ParseMessage is
UNDEFINED in the executable, resolving to libprotobuf.so.
- refuse_cloning_without_a_clip's "cannot misfire" paragraph had its reasoning
backwards. Routing picks VoiceCloning as the FALLBACK when there is no clip,
which is the case being caught; chatterbox, which ships in the gallery,
advertises clon and no tts at all, so every voice-less request lands there.
- audio_units read "2.1 min at 96 kHz" for index 11289602, which is 1.96 min.
2.1 min is 96 kHz's OWN first failure at 12288002. Both were remeasured and
the note is now a per-rate table.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* build(audio-cpp): exclude the upstream checkout from the C++ gate, harden the darwin walk
run-unit-tests.sh pruned */llama.cpp/* but not */audio.cpp/*. It is safe today
only by luck: upstream's 44 tests all put "test" at the FRONT of the filename
(17 test-*.cpp, 27 test_*.cpp, zero *_test.cpp), so the glob misses every one of
them, and nothing enforces that. This gate runs on every PR for every backend
and compiles each match as a standalone translation unit with nothing but
nlohmann/json on the include path, so the day upstream adds or renames one test
the gate goes red repo-wide on an Apache-2.0 file nobody here wrote.
audio-cpp-darwin.sh now logs the raw otool -L output and the parsed LC_RPATH
list unconditionally, before the walk. Both awk filters in that script assume a
column layout nobody working on this can observe, since it runs only on the CI
Mac, and a green first Darwin run proves nothing about the assumption: an awk
that silently matched nothing yields an empty dependency list, which reads
exactly like "no non-system dependencies" and packages happily. Both filters
otherwise feed process substitutions, so their input never reached the log.
It also lists every symlink in the package and fails on one that cannot resolve
inside the image. A dangling link does not fail anything else here, because
every assertion tests with -e, which follows links; it fails at dlopen on a
user's Mac. Links are NOT banned outright, which the review suggested but which
would break the libggml.dylib -> libggml.0.dylib chain the `cp -a` above exists
to preserve. What is banned is a link that resolves on the build host and will
not resolve in the image: a broken one, or an absolute one pointing outside the
package.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* style(audio-cpp): drop em dashes from the audio-cpp capability entry
Follow-up to a84b3c4b9, no behaviour change.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(config): key the voice-cloning model rule on the resolved backend
Making GetBackendCapability strip the gallery hardware prefix and release
channel fixed pinned variants of /audio/transform, but VoiceCloningForModel
kept keying its per-backend switch on the caller's spelling. A pinned name
therefore resolved the capability by stripping and then missed every case in
the switch, falling through to the permissive default: cuda12-vibevoice-cpp
advertised voice cloning for the realtime 0.5B model, metal-coqui for
tacotron2, cuda12-crispasr for a pure ASR model, cpu-qwen3-tts-cpp for
CustomVoice. Each of those is a model that cannot clone, so /v1/audio/speech
accepted a profile: voice it had to fail on inside the backend rather than
rejecting it with a 400, and the UI advertised the capability too.
resolveBackendCapability now returns the key the entry was found under, and
callers that branch on backend identity use that key instead of the name they
were handed. The exact-match-first order is unchanged, so a backend genuinely
registered under a variant-looking name still keys on its own name.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* gallery(audio-cpp): declare audio_transform on the chatterbox entry
Chatterbox advertises VoiceCloning AND VoiceConversion
(src/models/chatterbox), and the entry's own description already said so, but
known_usecases listed only tts. /audio/transform selects its default model by
FLAG_AUDIO_TRANSFORM, so voice conversion was reachable only by naming the
model explicitly and was invisible to every usecase-driven surface. It is the
one audio.cpp task with a shipped gallery model and no way to find it.
Verified against the real model rather than inferred from the capability list:
AudioTransform with chatterbox-q8_0, speech as audio_path and a speaker clip
as reference_path, returns a 5.08 s 24 kHz mono WAV at -25.5 dB mean and zero
stems, which is the single-output shape voice conversion should have.
The description now says which endpoint reaches that half and warns that
installing this next to a source-separation model gives /audio/transform two
candidates, so the model should be named rather than defaulted.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* gallery(audio-cpp): add voice-design and singing-voice-conversion entries
Two of the three audio.cpp task kinds that had no gallery model now have one.
Both were driven end to end against the real weights through the backend
before being written, not inferred from the capability tables.
audio-cpp-irodori-voicedesign covers vdes. TTS carrying `instructions` routes
to the vdes task, so the voice is described in words rather than supplied as a
clip. Verified: "a calm elderly woman speaking slowly with a warm, gentle
tone" over an 8.76 s 48 kHz mono render at -16.8 dB mean, and a closed-loop
citrinet pass recovers the sentence with the accent drift expected from a
Japanese-first model read by an English recogniser.
audio-cpp-seedvc-singing covers svc, and pins task:svc because nothing else
can reach it. seed_vc advertises svc and ordinary voice conversion, no request
signal means "this input is singing", and auto-routing resolves the tie to
voice conversion every time. Verified with the pin: 5.04 s 44.1 kHz output
whose closed-loop citrinet transcription is exact.
s2s deliberately has no entry, and the reason is not effort. miocodec is the
only upstream family whose speech-to-speech route needs no text, and it
returned audio with correct duration and level but no recoverable speech in
four independent attempts: the stale build, v2 q8_0, v2 orig (the variant
upstream records as a clean Pass), both tasks, and matched 44.1 kHz inputs on
both sides. vevo2's route refuses with "Vevo2 text/prosody route requires
text_input or target_text", and session.cpp:897 fills target_text only from
request.text_input, which AudioTransform has no field to carry. The same
vevo2 weights convert voice correctly through the default route with an exact
ASR round trip, so the model and the plumbing are both healthy; it is the s2s
route specifically that this RPC cannot express.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* backend(audio-cpp): carry transform text through params, add the s2s entry
AudioTransform is audio-in / audio-out and its proto message has no text
field, but not every task it routes to is audio-only. vevo2's speech-to-speech
route is a text and prosody route: session.cpp:897 fills refs.target_text from
request.text_input and nowhere else, and the run refuses without one with
"Vevo2 text/prosody route requires text_input or target_text". The params map
is the only channel this RPC has that reaches the engine, so the text travels
through it and apply_transform_text_input unpacks it after the params have
been copied into task.options.
Before this, s2s was not awkward to reach through /audio/transform, it was
unreachable, and it was the last audio.cpp task kind with a real model and no
way to get to it.
target_text is canonical and text is its alias, the order vevo2's own option
table declares them in, so a request setting both gets the canonical one
rather than whichever the map happened to store first. An empty value falls
through to the next candidate instead of ending the search. language rides
along only when a text was found: on its own it conditions nothing, and
manufacturing a text_input for it would route a plain separation request
carrying a language hint through the text path. The keys are left in
task.options rather than erased, because vevo2's loader advertises target_text
as a request option and a family reading it there keeps working.
Nine tests, all confirmed failing on behaviour against a stub that returned
false before the implementation was written. Verified end to end afterwards:
vevo2-q8_0 with task:s2s and params[text] returns a 5.12 s 24 kHz output whose
closed-loop citrinet transcription is exact, and htdemucs separation with no
text param still returns its four stems, with and without params[stem].
audio-cpp-vevo2-speech-to-speech ships that route. Every audio.cpp task kind
with a loadable family now has a gallery entry; spk remains the only gap and
has no family upstream at all.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* docs(audio-cpp): document params[text] and the pinned transform tasks
The text channel and the two task pins are both invisible from the endpoint
contract alone: nothing in the AudioTransform form tells a reader that a
speech-to-speech model needs the line it is resynthesising, and nothing says
that asking for singing voice conversion without task:svc silently gets plain
voice conversion instead. Both are the kind of thing a user only discovers
from a refusal or, worse, from output that looks right and is not.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(utils): annotate the two G304 sites this branch introduced
gosec flags os.Open on a variable path, and both new call sites in ffmpeg.go
are its alerts on this PR. Neither is reachable by an outside caller: isPCM16Wav
opens the exact path it is about to hand ffmpeg as input, which in the upload
path is a server-created temp file named from path.Base of the client name so
no traversal survives, and wavAudioBytes opens AudioResample's own dst, a name
this package derives from src and has just had ffmpeg write.
Annotated in the repo's existing style rather than restructured, with the
reason spelled out, because a bare suppression is worth nothing to the next
reader. The three other G304 sites in this file, in passthroughWAV and
isTargetWav, predate the branch and are left untouched.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(audio-cpp): build arm64 with gcc-14 for the armv9.2 SME variants
The arm64 CPU image failed to build:
cc1: error: invalid feature modifier 'sme' in
'-march=armv9.2-a+dotprod+fp16+sve+i8mm+sve2+sme'
ggml's CPU_ALL_VARIANTS table includes armv9.2 variants compiled with +sme, and
Ubuntu Noble's default gcc-13 rejects that feature modifier. Every entry in the
table has to compile even though a host only ever dlopens the one its own CPU
supports, so a single unbuildable variant fails the whole image. gcc-14 accepts
it, which is exactly the fix llama-cpp already carries in
.docker/llama-cpp-compile.sh; this is the same problem reached by a different
Dockerfile.
Applied to every arm64 BUILD_TYPE rather than to the CPU one alone, and that
differs from llama-cpp on purpose. llama-cpp needs it only for its pure-CPU
image because its GPU builds run llama-cpp-fallback, which builds no variant
table. This backend's Makefile turns ENGINE_ENABLE_CPU_ALL_VARIANTS on for
every non-Darwin build, GPU included, so an arm64 GPU image would hit the
identical error. The matrix has no arm64 GPU entry today, which is precisely
why gating on an empty BUILD_TYPE would leave the trap armed for whoever adds
the first one.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(model): deterministic, file-type-filtered backend auto-detect (#9287)
When a model config declares no explicit `backend:`, Load() fell into a
trial loop built by ranging the external-backends Go map (random order)
with no filtering, returning the first backend whose gRPC LoadModel
succeeded. An unrelated installed backend - e.g. the "opus" audio codec -
could therefore win a GGUF/LLM model load, so a model that should run on
llama.cpp wrongly tried to use opus.
Extract the candidate selection into a pure, testable function
SelectAutoLoadBackends that:
- sorts the candidate list deterministically (no more map-order
nondeterminism), and
- for a `.gguf` model, filters to LLM-capable backends (via
core/config.BackendCapabilities) and puts llama-cpp first, so an
incompatible audio/codec/image backend can never win the trial loop.
If filtering would leave zero candidates, the full sorted set is returned
unchanged, so a previously-loadable model is never made unloadable.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(model): break core/config <-> pkg/model import cycle in backend auto-detect
The #9287 auto-detect change made pkg/model/autoload.go import core/config
for the backend capability table. core/config already imports pkg/model
(runtime_settings_registry.go uses model.DefaultWatchdogInterval), so this
closed a core/config -> pkg/model -> core/config import cycle and broke the
build and golangci-lint.
Invert the dependency so the lower-level pkg/model no longer imports the
higher-level core/config. pkg/model exposes RegisterLLMCapableBackendFunc and
uses the registered predicate; core/config (which owns the capability table)
registers it from an init(). The deterministic, GGUF-type-filtered selection
behaviour is unchanged. When the predicate is unwired the GGUF filter is
skipped, preserving the existing zero-candidate fallback.
The unit test now injects a fake capability predicate so SelectAutoLoadBackends
is exercised independently of the core/config table.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
* fix(grammars): reject cyclic $ref in JSON-schema grammar to prevent stack-overflow crash
JSONSchemaConverter.visit resolved $ref entries by recursively calling
itself with no cycle detection. A client-supplied grammar_json_functions
schema whose $defs contains a self- or mutually-referential $ref (e.g.
{"A": {"$ref": "#/$defs/A"}}) made visit recurse until the goroutine
stack was exhausted, producing a fatal "stack overflow" that kills the
whole process rather than failing the single request. The schema is
converted synchronously in the /v1/chat/completions handler before any
backend call, so this is an unauthenticated remote crash. Fixes#11020.
Track the $ref targets currently on the recursion stack and error out
when one is re-entered, while popping after each descent so sibling
(non-cyclic) reuse of the same $ref is still allowed.
Signed-off-by: Tai An <antai12232931@outlook.com>
* fix(grammars): add a bounded recursion depth and cover llama31 $ref cycles
Addresses the review on #11041. The stack-set approach catches cyclic
$ref chains, but a deeply nested yet acyclic client schema (thousands of
nested arrays/objects) can still recurse through visit until the
goroutine stack is exhausted, which is the same unauthenticated remote
crash surface as #11020.
- Add a bounded depth counter to JSONSchemaConverter.visit (incremented
with a defer-based cleanup, capped at maxSchemaDepth = 256, far above
any realistic schema) so an over-deep schema fails the request with an
ordinary error instead of crashing the process.
- Apply the same cyclic-$ref guard and depth bound to
LLama31SchemaConverter.visit, the other production grammar entry point
named in #11020, which previously had no cycle detection at all.
- Regression tests: a deeply nested acyclic schema is rejected while a
moderately nested one still builds, plus direct/indirect $ref cycle
and depth tests for the llama31 converter.
Signed-off-by: Tai An <antai12232931@outlook.com>
* test(grammars): make llama31 cycle fixtures valid function-call shapes
The two new llama31 $ref-cycle specs asserted on "cyclic $ref" but the
converter requires each top-level oneOf alternative to carry its
function-name property before descending, so both fixtures failed
earlier with "no function name found in the schema" and never reached
the cycle guard.
Give each fixture a valid llama31 shape: construct the converter with
NewLLama31SchemaConverter("function"), put "function": {"const": "test"}
on the top-level alternative, and hang the cyclic $ref under an
arguments property, so all 29 grammar specs pass and the assertions
genuinely observe the cyclic $ref error.
Signed-off-by: Tai An <antai12232931@outlook.com>
---------
Signed-off-by: Tai An <antai12232931@outlook.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
backend_pr.yml and test-extra.yml already filter themselves, so a gallery-only
or docs-only PR costs them about one job each. The image and Go workflows had no
filter of any kind, so a one-line gallery/index.yaml edit queued 20 jobs: 7
container image builds, 3 GoReleaser/darwin launcher builds, 3 unit test jobs, 2
golangci-lint, 1 e2e, 1 yamllint, plus the 3 that correctly stop after their
detect step. A docs-only PR queued the same.
This matters more than the job count suggests. Measured over the week to
2026-07-30, 97% of CI wall-clock is queueing and 3% is execution: a median
5-hour queue against a 4-20 minute median job. Cutting job count is the only
lever that shortens feedback time. The volume is there to cut, too: 13
gallery-only PRs merged that week with 10 open at once, and 78 of the 137 PRs
opened were bot-generated.
Add paths-ignore for gallery/**, docs/**, examples/** and **/*.md to the
pull_request trigger of image-pr.yml, build-test.yaml and tests-e2e.yml, and
add gallery/** to lint.yml, which already excluded the rest. That drops 13 of
the 20 jobs. None of the four can observe such a diff: gallery metadata is
parsed at runtime and never copied into an image, docs and markdown never enter
one at all, GoReleaser and the launcher take no such input, the e2e suite drives
backends over gRPC directly, and golangci-lint runs new-from-merge-base so a
diff with no touched Go lines is a no-op. The build-test exclusion also frees
macOS capacity, which is the scarcest runner class.
The two checks that do validate the gallery are deliberately left alone.
test.yml still runs core/gallery/variants_lint_test.go, which reads the real
gallery/index.yaml and asserts the index invariants, and yaml-check.yml still
lints the syntax.
paths-ignore skips a run only when every changed file matches, so a PR touching
the gallery and Go code still runs everything. master carries no branch
protection and no rulesets, so a skipped workflow reports no status and nothing
waits on it; .agents/ci-caching.md records that constraint for whenever required
status checks are introduced.
image.yml on master push is left unfiltered on purpose: skipping it would stop
the master and latest tags being republished for a gallery commit, which is a
publishing decision rather than a cost one.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* gallery: add Nanbeige4.2 3B GGUF variants
Add Q4_K_M and Q8_0 builds of the compact Nanbeige4.2 agentic and reasoning model, grouped as install-time variants.
Assisted-by: Codex:gpt-5
* gallery: simplify Nanbeige4.2 model name
Apply the maintainer-requested canonical model name while retaining the quantization variants under the entry.\n\nAssisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add the official Q4_K_M build and seven APEX quality and size variants for llama.cpp.
Assisted-by: Codex:gpt-5 [Hugging Face API]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
GenerateImage hardcoded TilingParamsSetEnabled(vaep, false), so tiled VAE
decoding was unreachable from a model config even though all four upstream
setters were already bound in main.go.
Sampling runs in latent space, but the final VAE decode expands to full
resolution and needs one large compute buffer. At 1024x1024 that buffer
exceeds 8GB, which fails on two kinds of device: cards without the VRAM
for a full-frame decode, and drivers that cap a single allocation
regardless of how much memory is free. Mesa RADV reports a 4GiB
maxMemoryAllocationSize, so a Radeon 8060S with 74GiB of device-local
heap still cannot serve that decode:
[INFO ] sampling completed, taking 251.82s
[INFO ] decoding 1 latents
ggml_vulkan: Requested buffer size exceeds device buffer size limit:
ErrorOutOfDeviceMemory
[ERROR] vae: failed to allocate the compute buffer
[ERROR] decode_first_stage failed for latent 1
Every sampling step completes and then the run is discarded at the last
stage, so the whole generation is wasted.
Add three options, parsed in Load and applied per generation:
vae_tiling:true enable tiled decoding (bare flag also works)
vae_tile_size:512 tile size, or 512x384 for a rectangle
vae_tile_overlap:0.25 overlap between tiles
Tiling stays off unless requested, so existing models are unaffected. Tile
size and overlap only reach the library when the operator set them, which
keeps upstream's defaults rather than pushing a zero, and an unparseable
value is treated as absent for the same reason.
Truthy spellings match what load_model already accepts for its own bool
options, and the bare-flag form matches diffusion_model, so no new
convention is introduced.
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
Add the CAJAL GGUF gallery template and gallery index entry for local llama-cpp installs.
Assisted-by: Codex:gpt-5
Signed-off-by: Ching Kao <0980124jim@gmail.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
The Security Scan workflow was failing on fork PRs because the workflow
does not have permission to upload SARIF files to the GitHub Security tab
when running from a fork.
This change adds '!github.repository.fork' checks to all steps
to prevent the workflow from running on fork repositories.
This fix should be applied to the main repository so that
all forks inherit the correct configuration.
Fixes#10322, #10318, #10320, #10321
Co-authored-by: ghshhf <ghshhf@users.noreply.github.com>
fix(chatterbox): pin cublas12 torch/transformers and setuptools so the backend loads
The cuda12-chatterbox gallery backend fails to load on a fresh install
because several deps in requirements-cublas12.txt are unpinned:
- torch/torchaudio: unlike requirements-cublas13.txt and
requirements-cpu.txt, this file has no --extra-index-url, so pip pulls
a wheel whose CUDA runtime (cu130) is newer than the host driver
supports ("NVIDIA driver on your system is too old"). Add the cu124
index and pin torch/torchaudio 2.6.0+cu124.
- transformers: resolves to 5.x, which dropped LlamaConfig.rope_theta
that chatterbox-tts 0.3.1's T3 config still reads. Cap to <5.
- setuptools: 81+ dropped pkg_resources, which perth imports under a
bare try/except and silently sets PerthImplicitWatermarker=None,
making ChatterboxTTS.__init__ raise 'NoneType' object is not callable.
Cap to <81 in requirements.txt.
Fixes#11070
Signed-off-by: Tai An <antai12232931@anaiguo.com>
Co-authored-by: Tai An <antai12232931@anaiguo.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
response.create accepts a metadata map and ResponseCreateParams has carried
the field all along, but triggerResponse never copied it onto the Response it
emits, so both terminals went out with metadata omitted.
That field is the only thing tying a terminal event back to the
response.create that asked for it. Our own doc comment on ResponseCreateEvent
says so — "the metadata field is a good way to disambiguate multiple
simultaneous Responses" — and it is what makes an out-of-band response
(conversation: "none") usable at all: a client running one alongside the
spoken conversation has no way to tell its own answer from the conversation's,
so it waits for a reply it already received and gave away.
Found from the client side: a headless text turn injected into a live session
was answered correctly in about a second, and the caller still blocked until
its own two-minute timeout because it could not recognise the answer.
Carry the map on liveResponse so all three terminals (in_progress, cancelled,
completed) report it, and leave it omitted when response.create sent none.
Assisted-by: Claude:claude-opus-5 gofmt
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat: add Valkey Search vector store backend
Add a new built-in Go gRPC store backend 'valkey-store' that implements the
four Stores RPCs (Set/Get/Delete/Find) against the Valkey Search module (FT.*)
using the pure-Go github.com/valkey-io/valkey-go client. It is selected via the
existing per-request 'backend' field on /stores, so there is no proto or HTTP
API change, and it mirrors the in-memory local-store while adding persistence
across restarts and opt-in HNSW.
Each vector is a Valkey HASH keyed by hex(little-endian float32); the index is
created lazily on first Set (FLAT+COSINE by default), cosine similarity is
derived as 1-distance, and namespaces get a collision-resistant token. Includes
unit tests (valkey-go mock) and env-gated integration tests against
valkey/valkey-bundle, plus build/matrix/gallery wiring and docs.
Assisted-by: Kiro:claude-opus-4.8 golangci-lint
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* Address review feedback: recover persisted index dimension, harden Find
- Load now recovers the persisted vector DIM from FT.INFO (not just index
existence), so a post-restart Set/Find validates against the real DIM
instead of silently re-learning a wrong one and dropping mismatched
vectors from the index. This also restores Find's dimension check after
a restart.
- StoresFind treats a dropped/missing index as an empty store (empty
result, no error) and clears the stale indexCreated flag, matching
local-store's empty-store behaviour.
- StoresSet reuses checkDims for its per-key length check so the four RPCs
share one dimension-guard implementation.
- Add unit tests for FT.INFO dimension recovery, loadIndexState, and the
dropped-index Find path.
Assisted-by: Kiro:claude-opus-4.8
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* Address review feedback: TLS ServerName/CA, Find nil-check, config fail-fast
Addresses external review comments on the valkey-store backend:
- StoresFind now rejects a nil/empty query Key before dereferencing it,
so a malformed gRPC request can no longer panic the backend.
- TLS: derive ServerName (SNI) from the VALKEY_ADDR host so certificate
verification works for IP-addressed endpoints, and add VALKEY_TLS_CA_CERT
(custom CA bundle) and VALKEY_TLS_SKIP_VERIFY (testing-only) knobs.
- Config integer parsing now fails fast on a malformed value (e.g.
VALKEY_HNSW_M=1x6) instead of silently defaulting, matching the
fail-fast behaviour of the index-algo/distance-metric validation.
- Add VALKEY_DB (SELECT n) support for logical-DB isolation.
- Cap the human-readable part of a namespace token at 64 chars so a very
long model name cannot produce an unbounded key prefix / index name
(the appended short hash keeps distinct namespaces collision-free).
- Document the KNN-query injection-safety invariant (fields are constants)
and why StoresGet uses a single aggregate DoMulti deadline for reads.
- Unit tests for the Find nil/empty-key guard, fail-fast HNSW parsing,
and VALKEY_DB parsing/validation; docs + .env updated for the new vars.
Assisted-by: Kiro:claude-opus-4.8 golangci-lint
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* Address review feedback: configure valkey-store via model config
richiejp asked that the valkey-store backend take its configuration from
a model config rather than process-wide VALKEY_* environment variables,
so multiple stores can each have their own Valkey config within one
LocalAI process. This removes every env access from the backend and
routes config through the model-config seam every other backend uses.
- config.go: loadConfig(opts *pb.ModelOptions) now parses the model
config `options:` list (key:value strings, split on the first ':')
instead of os.Getenv. Option keys mirror the old VALKEY_* names without
the prefix (addr, index_algo, distance_metric, ...). Defaults, fail-fast
validation and the mandatory client name are unchanged.
- store.go: Load threads opts into loadConfig; TLS comments/errors renamed
off the VALKEY_* names.
- core/backend/stores.go: StoreBackend and NewVectorStore take a
*config.ModelConfigLoader, resolve the per-store ModelConfig by store
name, and pass its Options (and Backend when unset) to the backend via
WithLoadGRPCLoadModelOpts. No config -> default backend + built-in
defaults, preserving the zero-config experience.
- Endpoints/routes/application: thread the config loader to StoreBackend.
- Unit + integration tests: configure via options; the integration test
passes addr through the model-config path (VALKEY_ADDR is now only the
test harness locating the server).
- docs + .env: document the model-config options, drop the env var table.
Assisted-by: Kiro:claude-opus-4.8
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* Remove valkey-store informational comment from .env The backend is configured via model config, not env vars — the comment was unnecessary noise in .env. The configuration is already documented in docs/content/features/stores.md.
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* feat(valkey-store): gate Load on NamespacePrefix to refuse autoload probing Mirror local-store's pattern: reject model names without store.NamespacePrefix so the model loader's greedy autoload probe cannot bind an arbitrary model name to the vector store backend (the #9287 failure mode). Also adds unit tests for the gate covering: prefixed namespace, prefix alone, unprefixed model name, empty model, and nil opts.
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* feat(valkey-store): add username_env/password_env credential indirection Add support for resolving Valkey credentials from environment variables named in the model config, mirroring cloud-proxy's api_key_env pattern. This keeps secrets out of model YAML files and lets distinct store configs each reference their own credentials. Options: username_env / password_env name the env var holding the value. The direct username / password options still work and take precedence when both are set (backward compatible). Includes 5 unit tests and updated stores.md documentation.
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* fix: correct rebase artifacts in backend-matrix.yml and Makefile Fix two issues introduced by the conflict-resolution script during the rebase onto master: 1. .github/backend-matrix.yml: valkey-store entries were merged INTO the cloud-proxy entries (duplicate keys in same YAML map items) instead of being separate list items. This broke cloud-proxy Linux builds and the cloud-proxy darwin entry lost its build-type/lang. Fixed by making them standalone entries and restoring cloud-proxy exactly as on master. 2. Makefile: duplicated .NOTPARALLEL and docker-build-backends lines. Collapsed to single lines that are master's current content plus the valkey-store additions. Also adds the three optional pickups from #10801: - /valkey-store in .gitignore (the built binary) - valkey-store row in docs/content/reference/compatibility-table.md - valkey-store line in backend/README.md
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
---------
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
Co-authored-by: Daria Korenieva <daric2612@gmail.com>
* docs(gpu): add gfx1151 / ROCm 7.x and fix ROCm section
- Fix typo: "deditated" → "dedicated", "ROCm6" → "ROCm"
- Add ROCm 7.x to requirements (alongside ROCm 6.x)
- Add Ubuntu 24.04 to tested OS list
- Add AMD Strix Halo / gfx1151 section with kernel params,
required env vars (HSA_OVERRIDE_GFX_VERSION, ROCBLAS_USE_HIPBLASLT),
and Docker Compose example
- Add gfx1151 to the list of compiled GPU targets
- Add ROCm version column to verified devices table
- Add gfx1151 / Radeon 8060S (ROCm 7.11.0) as verified device
* fix(docs/gpu): correct gfx1151 section — env vars, image tag, safety warning
- Add all 4 required env vars (HSA_OVERRIDE_GFX_VERSION, ROCBLAS_USE_HIPBLASLT,
HSA_XNACK=1, HSA_ENABLE_SDMA=0) with descriptions in a table
- Fix Docker Compose example to use the ROCm 7.x image tag (-gpu-hipblas-rocm7),
not the ROCm 6.x image
- Add explicit warning: GGML_CUDA_ENABLE_UNIFIED_MEMORY must NOT be set
(even =0 activates hipMallocManaged due to getenv != nullptr check)
- Add --force-recreate note (docker restart does not update container env)
- Add tested hardware note (Geekom A9 Mega / Ryzen AI MAX+ 395)
* docs(gpu): single ROCm image — drop -rocm7 tag suffix
Per maintainer feedback on PR #9229: there is only one ROCm/hipblas
main image, and it ships with ROCm 7.x by default — no separate
-rocm7 tag.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Document the reverse-proxy settings needed for long-running and multimodal requests, and distinguish edge-generated 504 responses from the optional LocalAI busy watchdog.
Assisted-by: Codex:gpt-5 [Codex]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(3d): add Generate3D RPC, FLAG_3D capability, and /v1/3d/generations endpoint
Adds the plumbing for image-conditioned 3D asset generation (binary
glTF / GLB output), modeled on the video generation path:
- backend.proto: Generate3D RPC + Generate3DRequest (staged image src,
glb dst, seed/step/cfg_scale/texture_steps, quality and background
enums, params map for backend-specific extras)
- pkg/grpc: thread Generate3D through client, server, embed, base and
the backend interfaces; connection-evicting and distributed-node
wrappers (in-flight tracking + file staging) included
- core/config: FLAG_3D usecase (guessed only for the trellis2cpp
backend), '3d' canonical usecase string mapped to the Generate3D
method, and a '3d' output modality
- REST: POST /v1/3d/generations (+ unversioned alias) returning
OpenAIResponse with a /generated-3d URL or b64_json; conditioning
image accepted as URL, base64, or data URI; quality/background
validated at the edge; .glb served as model/gltf-binary
- auth: '3d' route feature (default ON); /api/instructions entry
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(trellis2cpp): add the trellis2.cpp image-to-3D backend
Wraps localai-org/trellis2cpp (C++/GGML port of Microsoft TRELLIS.2,
pbr-textures branch) as a Go+purego backend, following the
stablediffusion-ggml pattern:
- backend/go/trellis2cpp: purego bindings to the flat C ABI (v9,
asserted at startup), eager pipeline load with model-set validation
(refuses non-trellis GGUFs; degrades coarse/geometry-only/textured
exactly like the upstream demo), Generate3D via t2_generate +
t2_bake_glb writing a binary glTF to dst. Weight-free unit tests
cover resolution/validation/param mapping — CI never downloads the
multi-GB GGUF set or runs inference.
- CPU SIMD variants build into per-variant directories (the shared
libggml sonames collide across variants, unlike sd-ggml's flat
renamed-.so scheme); run.sh picks one via /proc/cpuinfo.
- CI wiring: backend-matrix entries (cpu, cuda12/13, vulkan
amd64+arm64, l4t, l4t-cuda13, darwin metal), index.yaml meta +
latest/master image entries, bump_deps tracking of the pbr-textures
branch, changed-backends.js mapping, top-level Makefile targets.
- Importer: auto-detects trellis GGUF repos/URIs (registered before
llama-cpp so the .gguf match isn't stolen) and expands any trellis
URI to the full 10-file component set spanning the three LocalAI-io
HF repos.
- Gallery: trellis2-4b (full PBR + 1024 cascade) and
trellis2-4b-geometry (512 untextured) with verified sha256s.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(ui): 3D generation page with native GLB viewer and IndexedDB history
Adds a Studio tab + /app/3d page for the new image-to-3D endpoint:
- GlbViewer ports the trellis2cpp demo's dependency-free WebGL2
renderer (quaternion trackball, metallic-roughness PBR, ACES,
hidden-line wireframe with a bounded index budget) and pairs it with
a minimal GLB parser for the two forms t2_bake_glb emits — dense
vertex-PBR (linear COLOR_0 + _METALLIC_ROUGHNESS, uploaded as
normalized integers) and the opt-in UV-atlas textured form. Parsing
happens before any GL so stats and errors render without WebGL2.
- use3DHistory stores past generations (params, input thumbnail, and
the GLB blob itself) in IndexedDB with keep-newest-20 eviction —
GLBs are multi-MB binaries localStorage can't hold — and the page
offers a download button for the active GLB.
- Wiring: CAP_3D capability constant (FLAG_3D — the exact string
/api/models/capabilities serves), threeDApi, router entries, Studio
tab, vite dev proxy, en locale keys.
- e2e: render-smoke entry plus a focused spec that feeds a real
one-triangle vertex-PBR GLB through the parser/viewer and exercises
IndexedDB persistence, selection, deletion, and API errors.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(3d): address API correctness and UX issues
Keep 3D generation on the LocalAI-specific /3d/generations route and ensure authentication and permissions cover it.
Propagate distributed transfer failures, publish a portable ARM64 backend image, honor importer overrides, and align discovery, upload validation, and touch controls.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(3d): add previewable print remeshing
Add a single-detail CGAL Alpha Wrap workflow for existing Trellis GLBs, including PBR reprojection, API documentation, tracing, and an in-browser preview before download.
Allow the remesh route to enforce its 512 MiB upload cap independently of the smaller global default so generated high-resolution meshes can be processed.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* build(trellis2cpp): centralize remesh dependency pins
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(kokoros): implement Generate3D stub for new proto RPC
The Generate3D RPC added to backend.proto for the trellis2cpp backend
made tonic's generated Backend trait require generate3_d, breaking the
kokoros-grpc build. Return unimplemented like the other unsupported
modalities.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
backend/backend.proto is consumed by every language, so its SHARED_BUILD_INPUTS
rule could only ever be always/always: 417 Linux plus 56 Darwin builds. It fires
on ~1.3% of commits (10 of 767 over six months), which made it the single
largest CI cost driver in the repo.
On 2026-07-29 the queue reached 2178 jobs against 8 concurrent runners. Four
runs totalling 935 of those jobs were triggered by nothing but a proto edit. The
largest, 378 jobs on master, came from PR #11158, whose entire proto diff was
six lines adding `bool cache_prompt = 8;` to one message. No backend that does
not read that field behaves any differently for it.
Make the rule content-aware. changed-backends.js resolves backend.proto at the
base revision (the contents-API pattern already used for backend-matrix.yml) and
hands both texts to protoChangeIsAdditive(), which compares them structurally so
a comment reflow, reindent or field reorder does not read as a change. An
additive-only edit (new field with an unused number, new message, new enum
value, new RPC) suppresses the rule and rebuilds nothing; a removed, renumbered,
retyped or renamed field, a dropped RPC or a changed option still rebuilds
everything, as does an unresolvable base revision.
Every other matched rule is untouched, so a PR that edits the proto and
scripts/build/ is still a full rebuild, and the weekly full-matrix cron remains
the backstop for stale wheels.
Verified against all ten proto commits of the preceding six months: the nine
with a resolvable parent all classify as additive, and controls covering a
retyped-and-renumbered field, a deleted RPC, identical revisions and a
reindent-plus-comment-reflow all classify correctly.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(llama-cpp): route Score through the slot loop
Score previously bypassed the slot loop with a direct llama_decode: a
conflict guard aborted the whole process if scoring raced generation, the
config validator had to reject score alongside chat/completion/embeddings,
and every candidate re-decoded the full shared prompt.
Add SERVER_TASK_TYPE_SCORE to the (patched) upstream server so score tasks
are scheduled like any other slot work: generation and scoring serialize
naturally, the shared prompt is decoded once per call, and the slot's
prompt cache carries the conversation prefix across calls. Context
checkpoints at the score boundary and at the cache-divergence point keep
SWA/hybrid/recurrent models (e.g. LFM2.5) from re-prefilling the whole
prompt per candidate: warm-turn scoring on a 6-option set drops from ~8s
to ~0.5s on a desktop CPU.
The conflict guard and the validation split are removed; declaring score
with generation usecases on one config is now supported and shares the
slot cache.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(realtime): classifier wire types and pipeline config
Wire types and YAML config for realtime classifier mode: sessions carry a
localai_classifier extension (options with canned replies/tool calls,
softmax threshold, normalization, history trimming, fallback modes, and a
deterministic wake-word address gate), mirrored by pipeline.classifier in
the model YAML and surfaced in the config-meta registry. The
localai.classifier.result server event reports the full score distribution
per turn.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(realtime): classifier response flow
Classifier-mode responses: instead of autoregressive generation, each user
turn is prefill-scored against the option list (router.ScoreClassifier
prompt/candidate shapes over the Score primitive) and the winning option's
canned reply and tool call are emitted through the existing response
machinery. Below-threshold turns take the configured fallback (none /
canned reply / generate); empty transcripts and unaddressed turns (wake
word not mentioned) skip scoring entirely. The scoring probe defaults to
the latest user message only — small scorers echo canned replies from
prior turns back as the top option otherwise.
Built for hardware that can afford prompt processing but not decode: with
slot-based Score the option list stays KV-cached across turns, so a turn
costs roughly one forward pass over the new words.
session_update_error events now carry the validation cause instead of a
generic message.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(realtime): bound the VAD tick's scan window and buffer retention
The VAD tick loop re-scanned the entire input buffer every 300ms and only
trimmed it on zero-segment ticks or commits. Audio that keeps producing
segments without a committing pause (steady noise a mic pipeline lets
through, music, continuous speech) grew the buffer toward the 100MB cap
with each tick rescanning all of it — O(n^2), measured at ~3.3ms of silero
per buffered second: past ~90s retained, ticks run back to back and pin
~4 cores until the stream stops.
Silero's recurrent state only carries a few hundred ms of context, so
rescanning old audio buys nothing. Clip the slice handed to the VAD to the
largest silence the commit test can need to measure (server_vad silence
window or the semantic eagerness fallback) plus a warm-up margin, and
rebase the returned segment times so every downstream consumer keeps
whole-buffer coordinates. An open turn whose clipped window is all silence
now commits (the silence outran the window) instead of being discarded as
no-speech. Independently, retain at most 90s of raw buffer, rebasing the
live-feed and EOU cursors on trim — this also bounds the previously
unbounded VAD-error path. Turn boundaries are otherwise unchanged: no
forced commits, no new coordinator states.
pipeline.turn_detection.vad_window_sec can widen the scan window; values
below the automatic floor are ignored. The tick body is extracted into
vadTick so specs can drive turn detection synchronously (same shape as
classifySoundWindow); the babble reproduction that pinned 4 cores now
plateaus under 10% of one core.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(backend): let per-model threads override the global default
ModelOptions overrode a set per-model threads value with the app-level
--threads whenever the latter was non-zero — and WithThreads defaults it
to the physical core count, so it always was. The YAML threads: knob has
been dead config: a tiny VAD model could never opt down from the global
pool size.
SetDefaults already fills an unset per-model value from the app config,
which is the intended precedence; resolve threads through a helper that
honors it (explicit threads: 0 still means unset).
Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* chore(gallery): single-thread the silero VAD
Silero is a ~2MB recurrent model with no exploitable graph parallelism:
measured per-call latency is identical at 1 and 10 ORT threads, while
every extra pool thread just spin-waits between the realtime loop's
frequent tiny inferences.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* docs(realtime): classifier mode, VAD scan window, threads precedence
Document the realtime classifier mode (options, threshold guidance,
wake-word address gate, empty-transcript handling), the VAD scan window
and 90s buffer retention (pipeline.turn_detection.vad_window_sec), the
per-model threads precedence, and the M3 classifier note in the realtime
state-machine design doc.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* perf(llama-cpp): score all candidates in one batched decode
One scoring call is now a single SERVER_TASK_TYPE_SCORE task: the slot
decodes the shared prefix (prompt + longest common candidate token
prefix) once, then forks one sequence per candidate off it
(metadata-only for the unified KV cache, copy-on-write for recurrent
state) and decodes every candidate's unique tail in one llama_decode.
Previously each candidate was its own task that restored the boundary
checkpoint and re-decoded its full tail sequentially, paying
per-candidate task and decode overhead.
The context reserves SERVER_SCORE_FORK_SEQS extra sequence ids (and
recurrent-state cells) beyond the parallel slots via the new
common_params::n_seq_score_forks. Forking requires the unified KV cache
(already this backend's default) since per-sequence streams would shrink
n_ctx_seq; an explicit kv_unified:false disables forking and Score calls
that need it fail cleanly. Candidates beyond the fork/output budget
decode in successive chunks.
Wire contract and scores are unchanged: per-token logprobs are stitched
from the shared region and the forked tails. Verified bitwise
deterministic call-to-call and independent of candidate order (no
cross-fork leakage via equal-length candidate swap); ranking matches the
per-candidate implementation on the drone battery (winner softmax
0.99996 vs 0.99997), and >16-candidate chunking, prefix-of-another and
empty candidates all pass.
Measured on a desktop CPU: warm /api/score calls 0.52s -> 0.23s; warm
realtime classifier turns 196-303ms. The 9-candidate drone turn decodes
~17 unique tail tokens in one batch instead of nine sequential ~220ms
checkpoint-restore tasks.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(realtime): gate scoring capacity by model usecase
Reserve llama.cpp scoring slots only for models that explicitly declare the score usecase, while allowing score to coexist with chat and completion. Reject incompatible unified-KV settings and classifier activation on models without scoring capacity.
Propagate application defaults when resolving realtime and preload pipeline stages so unset thread counts are resolved consistently without overriding explicit model settings.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(ci): honor APT mirrors in the prebuilt llama-cpp compile step
The builder-prebuilt path installs gcc-14 with apt directly and ignored
the APT_MIRROR/APT_PORTS_MIRROR build args the from-source path already
honors, so an ubuntu mirror outage broke every arm64 backend build. Pass
the args into the stage and run apt-mirror.sh (already in the build
context via COPY . /LocalAI) before the apt step.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(realtime): classifier argument slots via constrained completion
Hybrid classify-then-complete: a classifier option's canned tool call can
declare typed argument slots (number | enum | string, with defaults and
prompt hints) referenced as "{{name}}" in the arguments template. When
the option wins, the slots are filled by a short grammar-constrained
completion that continues the exact scoring prompt — rendered by the same
cached ScoreClassifier, so the llama.cpp prompt cache is already warm —
with the chosen route JSON re-opened at the first slot field. A GBNF
grammar pins the field skeleton and frees only the values; temperature 0,
a couple dozen tokens at most (~300ms on a desktop CPU for two slots).
Slot declarations and hints ride the option descriptions in the shared
system prompt, informing scoring and the fill alike at no per-turn token
cost. The localai.classifier.result event carries the final arguments and
a fill_latency_ms. On inference failure the slots' defaults apply; a slot
without a default fails the response (or falls through with
fallback.mode: generate). Slot filling requires completion alongside
score in the scoring model's known_usecases.
Verified end-to-end on the Pi drone demo: "fly forward three meters" in
distance mode classifies forward and infers {"distance": 3, "units":
"meters"} in ~310ms, and the drone flies exactly 3 units.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(realtime): splice filled slot values into classifier replies
A classifier option's spoken reply can now reference its tool's argument
slots ("Going forward {{distance}} {{units}}."): the values inferred by
the slot-fill completion — or the recovery defaults — are spliced into
the reply as plain text before it is emitted, so what the assistant says
confirms what it actually inferred. Placeholders without a value stay
literal, and options without slots are untouched.
FillToolArguments now returns the raw slot values alongside the spliced
arguments JSON to make the reply templating possible.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(realtime): harden classifier slot completion
Reserve context for constrained slot filling, size completions from their encoded output, and encode enum grammar literals as valid JSON. Reject empty enum values and cover the failure modes with regression tests.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(realtime): prewarm the classifier scoring prompt on registration
Swapping a session's classifier option list (a voice-switched command
mode, for instance) made the next turns pay a full re-prefill of the new
option-list prompt — measured 2.4s vs 0.3s warm on a desktop CPU, and
worse: on hybrid-memory models like LFM2.5, whose state cannot be
partially rewound (llama.cpp can only restore checkpoints), *every*
probe change re-prefilled from scratch whenever the last checkpoint
missed the probe boundary, so even same-list turns intermittently cost
full prefills.
Registering an option list (pipeline seed or session.update) now fires a
best-effort background prewarm: two throwaway scores with distinct
probes. The first prefills the new option-list prompt; the second,
diverging exactly where per-turn probe text starts, plants the backend's
rewind point (KV checkpoint) at the stable-prefix boundary that every
real turn reuses. The prewarm hides behind the canned mode-switch reply
— by the time it finishes speaking, the cache is warm. Idempotent per
option set, detached from the registering request's lifetime.
Measured on the drone demo (LFM2.5-1.2B, desktop CPU): first turn after
a mode switch 2374ms -> 340ms; intermittent same-list full prefills
(1.3-2.1s) all -> under 0.5s. For clients that swap lists frequently,
options: [parallel:2] on the scoring model additionally keeps one slot
per list via prefix-similarity routing (+26MB RSS, unified KV).
Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* perf(llama-cpp): checkpoint scoring at the caller-declared stable prefix
Hybrid-memory models (LFM2.5 shortconv, Qwen3.5 deltanet — where new
small models are headed) cannot rewind their state, so any prompt-cache
reuse that needs a rewind falls back to a full re-prefill. For classifier
scoring that meant every probe change re-processed the whole option-list
prompt: the server's checkpoints were placed reactively (at wherever the
previous task happened to diverge), so a checkpoint past the next
divergence was erased rather than restored — measured as intermittent
2-10s turns on prompts with a 95%+ common prefix.
The classifier now computes the probe-invariant prompt prefix once (the
byte-wise common prefix of two synthetic probe renders) and declares its
length with every Score request; the server maps it to a token boundary
and forces a KV checkpoint exactly there on each score prefill. That
checkpoint sits at or before every future divergence under the same
option list, so it always survives and always restores — repeat scoring
costs probe+candidates regardless of how the probe changes.
Also:
- prewarm reruns on every option-list registration instead of memoizing
per list: with boundary checkpoints a redundant rewarm costs two
probe-sized decodes, while skipping one after a slot eviction (three
lists sharing fewer slots evict in LRU cascades) silently moves a full
re-prefill onto the user's next turn
- new llama.cpp backend option rs_seq:N exposes bounded recurrent-state
rollback outside speculative decoding; measured impractical for
deltanet-scale states (65GB for 64 snapshots on Qwen3.5-4B) but cheap
insurance for small-state models
- docs: the multi-list recipe (parallel:N + sps:0.5 — the default slot
similarity threshold funnels distinct lists onto one slot)
Measured on the drone demo (LFM2.5-1.2B scorer, desktop CPU), steady
state: every turn 285-421ms including mode switches, vs 2.4s post-switch
and intermittent 1.3-2.9s re-prefills before.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(realtime): align classifier cache guidance
Document the single-score prewarm behavior and clean the vendored score patch formatting.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(llama-cpp): guard score task for fork backends
TurboQuant and Bonsai reuse the primary gRPC server against llama.cpp forks that do not carry LocalAI's slot-based Score patches. Compile the Score integration only for the patched primary backend and return UNIMPLEMENTED from fork builds instead of referencing absent task types and common_params fields.
Assisted-by: Codex:gpt-5 [gh]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(dev): generate gRPC code before commit lint
The coverage phase regenerates ignored protobuf bindings, but lint runs first and can fail against missing or stale output. Generate the pinned bindings before lint so the gate always type-checks the current schema.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Add the official Q4_K_M and Q8_0 builds plus the DFlash speculative-decoding pairing for llama.cpp.
Assisted-by: Codex:gpt-5 [Hugging Face API]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(sglang): keep nvidia-modelopt on a stable release
Every cublas sglang image currently fails to build:
Failed to build `nvidia-modelopt==0.46.0rc0`
Call to `wheel_stub.buildapi.build_wheel` failed
ModuleNotFoundError: No module named 'wheel_stub'
sglang[all] pulls nvidia-modelopt in through its `diffusion` extra with no
version bound of its own, and install.sh adds a GLOBAL --prerelease=allow so
that flash-attn-4, which only ships 4.0.0b* wheels, can resolve. Unbounded plus
prereleases-allowed picks 0.46.0rc0, whose build backend imports wheel_stub
without declaring it in build-system.requires. EXTRA_PIP_INSTALL_FLAGS also
starts with --no-build-isolation, so nothing installs wheel_stub and the build
dies. Latest stable is 0.45.0 and resolves cleanly.
Bounding this one package rather than dropping the global flag, because the
flag is load-bearing for flash-attn-4 and this is the narrower change with the
smaller blast radius. Raise the bound when 0.46.0 final ships.
This is invisible on master because the backend build is path-filtered: sglang
is only rebuilt when sglang changes. It surfaces on any PR touching a shared
build input such as backend/backend.proto, which rebuilds the whole matrix.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(nemo): build the darwin venv on Python 3.12
The darwin nemo image fails to build:
ModuleNotFoundError: No module named 'maturin'
nemo_toolkit pulls in text2num, a Rust extension built with maturin, whose
macOS arm64 wheels start at cp311: 3.0.2 publishes cp311, cp312, cp313 and
cp314 and no cp310. libbackend.sh defaults PYTHON_VERSION to 3.10, so pip finds
no wheel, falls back to the sdist, and dies in the PEP 517 hook because
EXTRA_PIP_INSTALL_FLAGS carries --no-build-isolation and nothing installs the
build backend. Taking the prebuilt wheel avoids the source build entirely, so
the runner needs no Rust toolchain.
Darwin only, deliberately: the Linux profiles resolve a cp310 manylinux wheel
for the same package and have no reason to move. The override is set after
libbackend.sh is sourced and before installRequirements, the same shape
sglang's install.sh already uses for its l4t13 profile.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(nemo): pin the darwin portable-Python patch level too
The 3.12 bump alone traded one failure for another:
curl: (56) The requested URL returned error: 404
make[1]: *** [nemo-asr] Error 56
libbackend builds the portable-Python URL from
cpython-${PYTHON_VERSION}.${PYTHON_PATCH}+${PY_STANDALONE_TAG}, and
PYTHON_PATCH defaults to 18 because the default interpreter is 3.10.18. Setting
only PYTHON_VERSION asked for a 3.12.18 that was never released.
Patch 11, not the 12 that sglang/install.sh pairs with 3.12 for l4t13: at the
20250818 tag python-build-standalone published 3.12.12 for linux aarch64 but
not for aarch64-apple-darwin, where 3.12.11 is the newest. Both URLs were
checked against the release assets rather than assumed to match.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
* feat(cli): support systemd socket activation
Serve the API from a single stream listener inherited through the systemd activation protocol while retaining the existing address bind path when no listener is provided. Validate activation metadata, preserve the public-bind safety check, and document an on-demand systemd setup.
Assisted-by: Codex:gpt-5
* fix(cli): satisfy listener cleanup lint
Make the best-effort close explicit so errcheck accepts the deferred systemd listener cleanup.
Assisted-by: Codex:gpt-5 [golangci-lint]
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Some Transformers processors used by MLX-VLM, including Qwen vision models, import both PyTorch and Torchvision. Include them in the Metal backend environment so model loading does not fail with missing-library errors.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The Anthropic translate provider builds the upstream request from scratch and
never emitted cache_control, so prompt caching was impossible for OpenAI-format
clients routed through cloud-proxy — even though the entire system prompt + tools
prefix is re-sent on every agentic turn.
Add an opt-in cache_prompt flag (ProxyOptions.cache_prompt; model YAML
proxy.cache_prompt: true). On a translate+anthropic model, buildAnthropicRequest
injects cache_control:{type:ephemeral} on the stable prefix — the system block,
the last tool, and the last message block (at most 3 of Anthropic's 4 allowed
breakpoints). Anthropic then serves the repeated prefix at the cache-read rate
(0.1x input) on subsequent calls, cutting cost on multi-turn/agentic workloads.
No effect in passthrough mode, for non-Anthropic providers, or when unset.
System is widened to any so it can carry the block form required to attach
cache_control, while still marshalling as a bare string when caching is off.
Adds a unit test asserting exactly three breakpoints when on and none when off,
and documents the option in docs/content/operations/cloud-proxy.md.
Assisted-by: Claude:opus-4.8
Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
* feat(ui): record finished gallery operations in a bounded history ring
The operations panel drops an operation the moment it succeeds, so a user
who steps away cannot tell whether an install finished, failed or was never
started. OpCache now keeps the last 50 terminal operations, recorded from
the point where an op leaves the cache.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(ui): pin the history ring's dedupe, outcome order and start stamp
Review of the history ring found four gaps. The dedupe guard and the
bounded seen set were unreachable through the exported API and so had no
coverage; an in-package spec file now drives opHistory directly. The
outcome switch claimed an ordering was load bearing that nothing pinned,
so an errored op that never reached Processed now has a spec.
Two behaviour fixes come with it. StartedAt was the zero time for ops
recovered from the store or replicated from a peer, since neither path
stamps a start time, which would have rendered as a two-millennia
duration; it now falls back to the finish time. Reusing a cache key with
a fresh job ID orphaned the previous stamp, so Set and SetBackend now
drop it.
The comment on the outcome switch described a state the code cannot be
in: CancelOperation sets Cancelled and Processed synchronously before the
handler removes the entry, so status.Cancelled already covers the cancel
endpoint. The !Processed clause stays for the dismiss endpoint firing on
an in-flight op, and the comments now say so.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): record operations that end on a peer replica
The NATS end event is the only signal a replica gets for an install another
replica ran. Record from applyEnd too, deduped by job ID so the originating
replica does not record its own broadcast twice.
Three start-stamp defects in the same path go with it. applyEnd now drops the
stamp unconditionally, since recordTerminal only cleans up on the path where it
found a cache key and an end event can overtake the local Set. applyStart drops
the stamp of the job whose cache key it replaces, which a peer-driven retry
previously stranded. And recordTerminal reads the stamp once instead of testing
Exists and then reading, so a concurrent record for the same job can no longer
delete the stamp between the two and let the zero time overwrite the
finish-time fallback, which the Activity page would render as a two-millennia
run.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ui): do not guess the outcome of a peer operation with no local status
A replica that restarts mid-operation hydrates its OpCache keys from
PostgreSQL, but gallery statuses are in-memory only and come back empty. The
end broadcast then landed on recordTerminal's nil-status branch, which reads a
missing status as queued-and-removed and filed a successful install as
cancelled. That reading is right locally and wrong on the peer path, where a
missing status means the outcome was never held here.
recordTerminal now takes the source of the terminal event and records nothing
when the peer path finds no status, restoring what the replica did before the
end event started recording. The local path is unchanged.
Also move the ApplyEndForTest seam to the conventional export_test.go, and stop
the dedupe spec from claiming to guard the ring's seen set: the local delete
removes the status keys, so the broadcast that follows returns before reaching
it. An in-package spec that calls recordTerminal twice does the pinning.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(api): add GET and DELETE /api/operations/history
Admin gated like the rest of the operations API. The live /api/operations
payload is unchanged so the one second poll stays small.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): expose operation history through OperationsContext
Fetched on demand and when the live list shrinks, never on the one second
poll interval.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ui): detect operation departure by identity and ignore committing ops in the ETA gate
Refetching history on a shrinking live count missed a completion that
coincided with a start, which is the common case during a batch install.
Track the live job IDs instead, so any departure triggers the refetch
regardless of how the count moved.
An operation that has finished downloading stays live at
currentBytes == totalBytes for the whole commit and install phase and can
never produce an estimate, so counting it in the all-or-nothing gate blanked
every other operation's time remaining for as long as it lasted. Only
operations still moving bytes get a vote.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ui): let only downloading operations gate the time remaining estimate
Verifying pins an operation flat below its total for the whole sha256 pass:
the AfterDownload hook reports completedBytes plus the finished file against
a total summed over every file, then hashes synchronously without emitting
progress. Files download sequentially, so a 15 shard model enters that
window 14 times, and a byte comparison cannot see it because the counter is
genuinely below the total throughout.
Gating on phase closes resolving, verifying, committing and persisting in
one predicate, so a quiet neighbour no longer blanks every other
operation's estimate for minutes at a time. The byte clauses stay: a
producer can report downloading with bytes already at the total.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): collapse the operations bar to a single line
Four concurrent installs used to take four rows above every page. The strip
now shows one operation, failure first, with a counter linking to Activity.
The close button hides the strip and no longer cancels an install.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ui): keep the operations strip from widening the page and from muting a failure
A long install error made the strip report a 1600px minimum width, which sized
main-content to fit and gave every page under it a horizontal scrollbar.
Inline-size containment plus shrinkable detail and bytes cells keep it inside
the viewport.
Hiding is no longer able to swallow the hidden job's own failure, a completed
removal or staging says so instead of claiming an install, a cancelling
operation renders as cancelling, and the live region no longer covers the
per-second percentage.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ui): shrink main-content instead of containing the strip, and expose progress
min-width on .main-content is what actually lets a long install error shrink,
and unlike inline-size containment it has no browser support floor and no
latent collapse if the strip ever lands in a shrink-to-fit context. It matches
what .app-layout-chat .main-content already does, and it clears pre-existing
horizontal overflow on narrow viewports as a side effect.
The progress track is now a labelled progressbar, so assistive tech can read
the value on demand rather than losing it to the aria-hidden that stopped the
live region re-announcing every poll.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): add the live operation card for the Activity page
Carries the detail the one-line strip has to drop: phase, bytes, the per-node
breakdown for cluster installs, and a labelled Cancel button. Cancelling is
destructive, so it gets a labelled button rather than a glyph.
A cancelling operation drops its progress bar and its time estimate, the same
call the strip makes: a percentage still climbing under "Cancelling" reads as
the cancel not having taken.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ui): give the operation card a verb, live node disclosure and its per-node detail
The card carried no verb, so an install, a removal and a staging op rendered as
spinner plus name plus kind tag and were indistinguishable. It now runs the same
verb and icon chain as the one-line strip, which is what stops the page that is
meant to carry more detail from carrying less.
The auto-expand default was evaluated once at mount. An operation is listed as
soon as it is admitted but its nodes are filled in only when the fan-out starts
reporting, so a card mounted at creation latched on the empty list and stayed
collapsed. The default is a live expression now, and state holds only an
explicit choice.
Also: an optional onRetry gates a Retry button, so the page can own the install
reconstruction without the card ever showing a control with nothing behind it;
the disclosure moved above the region it controls and gained aria-controls; the
toggle is gated at more than one node so the count is never "1 nodes"; an
unmapped node status is passed through instead of being relabelled "Queued";
error text is clamped with the full string in the title; and file_name plus the
per-node progress bar are rendered again, reviving three CSS rules that had gone
dead along with the detail they styled.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): add the Activity page
Live operations, unacknowledged failures and the record of what finished, at
/app/activity in the Operate console. Cancelling an install now lives here
behind a labelled button rather than on the strip, and a failed install can be
retried: the retry dismisses the failure first so it still reaches the record,
then reissues the model, backend or node-scoped backend install.
The sidebar Operate entry carries the operation count. The console rail is only
rendered on an Operate route and can be collapsed, so a badge there could
vanish while operations were still running.
Two follow-ups from review fold in here: a failed removal or staging job no
longer reports a failed install on either the card or the strip, and the card's
error text can shrink so one unbroken token cannot widen the card.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ui): dismiss operations by job, and stop the Activity page contradicting itself
Dismissing resolved the job by display id, but /api/operations strips the
"node:<nodeID>:" prefix before emitting, so a local install and a node-scoped
install of one backend arrive as two jobs sharing one id. Dismissing by id
retired whichever came first. That defeated the guarantee retry was built
around: with the wrong job dismissed, the reinstall overwrote the acted-on
failure's opcache entry in place, bypassing recordTerminal, while an unrelated
failure vanished from Needs attention. dismissFailedOp, the card's dismiss
control and the strip now all pass the jobID, which is what the endpoint takes.
A filter matching nothing rendered the "nothing has ever run" empty state while
the header counted the records the filter had hidden. The empty state is now
gated on the All chip and a narrowed view gets its own message plus a way back;
the header counts the instance rather than the chip, so selecting Backends no
longer reports "Nothing running" over running model installs.
Also: the summary drops a zero clause instead of rendering "0 needs attention"
on the happy path and pluralises both counts; a record duration is floored at
"< 1s" and rejected above a day, so a zero-value start stamp cannot render a
span of millennia and a zero span cannot render "installed in" with nothing
after it; a deletion cancelled mid-flight reports the cancellation rather than
claiming it was removed; and the retry variant comment names the fix instead of
calling the gap closed.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: document the Activity page and the operations history endpoints
Adds an Activity page under Operations covering the one-line operations
strip, the /app/activity sections and filters, per-operation cancel,
retry and dismiss, the in-memory 50-entry record, and the sidebar count.
Documents GET and DELETE /api/operations/history, and fills the gap in
the admin-only endpoint list, which also omitted the pre-existing
POST /api/operations/:jobID/dismiss.
Corrects the distributed-mode install-watching section: the per-node
breakdown now lives on the Activity page rather than on the strip, which
rolls a fan-out up into a single phrase.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: correct nine details in the Activity page documentation
The operations strip never renders a file name: its detail line is the
error, the node roll-up, the target node, the phase or the queued note.
Drops the stale clause in the distributed-mode section, where the
per-node bullet is now the only place a file name is described.
Scopes the phase vocabulary to artifact-backed gallery models, since a
plain GGUF install emits no phase. Corrects the per-node list: the
toggle exists for any fan-out of two or more workers and the four-node
threshold only governs whether it starts open, while the N nodes tag
needs more than one node. Notes that a cancelled operation can sit in
the live section reading Cancelling, that cluster staging never reaches
the record, and that Clear history appears only when the record has
something in it.
Names the operations response envelope, with a JSON example, so callers
do not index a bare array, and stops describing the icon-only dismiss
control as a labelled button.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: drop the unreachable Cancelling state and scope the byte claims
An operation can only report isCancelled while it is unprocessed, but
every writer of Cancelled sets Processed in the same breath, on the peer
path as much as the local one, and the cache evicts cancelled entries
before the handler sees them. The state cannot reach the page, so the
live section is described again as running or queued operations.
Byte counts come from the artifact bridge alone, the same producer as
the phase, so a plain GGUF install, a removal and a backend install
report none. Scopes both to artifact-backed gallery models and leaves
the verb, the name and the percentage as what every operation shows. A
worker backend install reports its bytes through fields the operations
payload does not carry, so the distributed section now describes the
percentage and the node roll-up, with per-file counts pointed at the
per-node detail.
Also: staging jobs carry no error, so they never reach Needs attention
and Retry never had a staging case to exclude; an install that involves
workers is no longer called node-scoped, which this page uses for
node-targeted installs; and the record timestamps carry nanoseconds.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: state only the verb and the name as unconditional on the strip
The percentage is as conditional as the bytes were: it renders only for
a running operation that has reported progress, so a queued operation, a
failed one and a removal never carry it. A removal in particular sits at
progress zero for its whole visible life, since the delete path reports
none and its completion is filtered out. Both the strip and the card
paragraphs now lead with what always shows and list the rest as
conditions.
The Cluster chip matches on a node list that finished operations do not
carry, so a fan-out install leaves the chip once it reaches the record.
Scoped that claim to the live sections.
Two more of the same shape, found by re-reading each clause alone: the
strip also appears for a failure, which is not running, and the
four-second hold only applies when nothing replaces the operation that
just finished.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ui): stop reporting a cancelled install as installed, and make queued real
Three defects that all trace to one root cause: `isCancelled: true` is
unreachable from /api/operations. Every writer of Cancelled=true also sets
Processed=true, the handler skips Processed && Cancelled, and OpCache.GetStatus
evicts a cancelled op before the handler iterates it.
Cancelling the last running operation put a green "Installed model X" on the
strip for four seconds: the completion hold was guarded by
`!previous.isCancelled`, which is dead. A cancellation deletes the operation
server side, so the strip sees exactly what it sees on a completion, and
nothing in the payload separates the two. The signal now comes from the side
that issued the cancel: the operations context remembers the job IDs it
cancelled (pruned after a minute) and the strip asks before it holds anything.
A cancelled operation goes as soon as it stops; the record already reports it
as cancelled.
isQueued was set only when the gallery status was missing, but markQueued
publishes a "queued" status at admission, so a queued op has a status for its
whole queued life and the state was unreachable outside a microsecond window.
Every operation waiting behind a running install rendered as "Installing model
X" with a spinner. The queued phase is now the signal, via an exported
PhaseQueued and a nil-safe OpStatus.IsQueued() next to the writer.
With those two fixed, the Cancelling state has no way to be entered: cancelling
is instantaneous from the API's point of view. Its branches, CSS, locale key
and the isCancelled field itself are removed rather than left for a future
reader to assume they work.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ui): keep a removal a removal, and say what an install is doing
OpStatus.Deletion was set once, at admission, and lost on the next status
write: UpdateStatus replaces the whole status and only carried Nodes
forward. Every later writer (the worker's first write, the progress
ticks, the failure path) leaves the field at its zero value, so the flag
survived only the queued window, and both surfaces test isQueued first.
The reachable consequence is that a failed removal reported itself as a
failed install, which is exactly the shape the Activity page offers Retry
for, and Retry installs: pressing it on a removal that failed
re-downloaded the model. A running delete also rendered as "Installing
model X" with a spinner, and a successful one as "Installed model X".
Carry Deletion forward the way Nodes already is. A job is a delete or an
install for its whole life; an unset flag means "no new information", not
"this is an install". Pinned by Go specs on both the service and
/api/operations: the existing Playwright specs were green only because
they stubbed a payload the server could not emit.
Also restore the operation's own status message on the Activity card.
Phases and byte counters exist only on the managed-artifact path, so a
legacy files: gallery model and every backend install rendered a sub-row
with nothing in it but the verb. The strip stays terse on purpose.
And give the strip's name a min-width floor: overflow: hidden zeroes its
automatic minimum, so a long error squeezed the name down to "mod…" and
the identity of the thing that broke was the first thing lost.
primaryOperation is made module-private: its comment claimed the Activity
page selected the same operation, but that page shows all of them,
partitioned into failed and running, and never imported it.
Assisted-by: Claude Code:Opus 5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(activity): read the operations record from PostgreSQL
The Activity page's record of finished installs and removals was a 50-entry
in-memory ring per frontend replica. In distributed mode that is the wrong
place for it: each replica keeps its own copy, a replica added by a scale-out
or a rolling deploy starts empty and never backfills, and "Clear history"
clears only the replica that served the request, so the record reappears on
the next poll routed elsewhere.
The data is already in gallery_operations. Read it from there.
GalleryStore gains ListTerminal and ClearTerminal, sharing a lifted
terminalStatuses set with CleanOld so there is one definition of "finished".
ListTerminal orders by updated_at, when the operation reached its terminal
status, because the record reports what finished and when.
OpCache.History and ClearHistory dispatch on whether a store is wired, so the
HTTP handlers and the OpRecord JSON shape are unchanged and the page needed no
change. A failed store read falls back to the local ring rather than blanking
the page, and ClearHistory empties the ring as well so a database blip cannot
resurrect a record the admin just cleared.
The name derivation in recordTerminal is lifted into operationDisplayName and
used by both paths, so the ring and the store cannot name the same operation
differently.
Also fixes a pre-existing bug the store path made visible: the backend channel
hardcoded op_type "backend_install" even for a removal, while the model channel
derives model_install/model_delete from op.Delete. Both channels carry the same
ManagementOp, whose Delete field the backend handler already branches on, so
the backend channel now derives backend_delete the same way.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(activity): keep a cancelled operation cancelled, and report a failed clear
Review follow-up on the store-backed Activity record.
A cancelled install was recorded as a failure. The cancel handler persists
"cancelled" synchronously, then the handler goroutine unwinds with the context
error and Start hands that to updateError unconditionally, which overwrote the
row with "failed: context canceled". The page rendered a cancelled install as a
red failure card offering Retry, with a raw context error as the reason.
Fixed in GalleryStore rather than in Start, because an operation finishes once
and the paths that retire one are not mutually exclusive: UpdateStatus now
refuses to rewrite a row that already reached a terminal status. That also pins
updated_at to when the operation really finished, which is the key the record is
ordered by, and Create's upsert now freezes the same columns so a worker
dequeuing an operation the admin cancelled while it was queued cannot reopen it
as pending.
ClearHistory returned nothing, so a failed delete logged a warning while the
handler still answered 200. The admin watched the record clear and come back on
the next fetch with nothing said about why. It now returns the error, the DELETE
handler answers 500, and the store is cleared before the local ring so a failure
leaves the fallback record intact rather than faking an empty one.
Hydrate is the only reader that decides from op_type whether an operation is a
removal, and it tested for "model_delete" exactly, so the backend_delete added
in the previous commit hydrated as an install: a replica restarting during a
backend removal rendered "Installing backend X". Both discriminations now go
through IsDeleteOpType/IsBackendOpType so a fifth op_type cannot silently read
as an install in whichever consumer was missed.
Also: the backend channel now persists Cancellable as !op.Delete, matching the
model channel; IsBackend falls back to the op_type prefix, since is_backend_op
is only written by UpsertCacheKey and the rows needing the name fallback were
reporting backend operations as models; and an unrecognized terminal status is
logged rather than quietly filed as a success, which is what the comment already
claimed.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(activity): keep a reaped operation correctable by its real outcome
The terminal-status freeze added in the previous commit was too wide. It froze
"failed" alongside "completed" and "cancelled", and the stale reaper writes
"failed" onto operations that are still going to run.
The gallery worker is a single goroutine consuming both channels serially, so
an operation queued behind a large download sits in "pending" with nothing
bumping updated_at, and ReapStaleOperations gives up on it after 30 minutes.
That used to be self-healing: the worker dequeued it, Create reset the row to
"pending", and the operation reported its real outcome. With the freeze the row
stayed "failed" forever while the install ran and succeeded underneath it: a red
failure card offering Retry for a model that is installed, omitted from
ListActive so no replica hydrates it, and no longer deduped cluster-wide by
FindDuplicate.
Freeze on ("completed", "cancelled") instead. That is all the cancelled-install
fix ever needed, and it leaves a failure correctable by what actually happened.
The set is separate from terminalStatuses, which ListTerminal, ClearTerminal and
CleanOld all still want in full, because the two mean different things: a
failure can be superseded by a real outcome, a completion or a cancellation is
the real outcome.
UpdateStatus now writes the error column unconditionally, so a corrected
outcome drops the previous attempt's reason rather than being recorded as
completed while still carrying "stale operation reaped" as its error.
Also adds the route-level spec for the 500 branch of DELETE
/api/operations/history, and trims a comment that credited the persisted
cancellable column with more than it survives long enough to do.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(activity): offer Cancel in the phase that can honour it
The cancellable flag was set at both ends of an operation's life and was wrong
at both, in opposite directions.
A queued operation is cancellable whatever it is. EnqueueModelOp and
EnqueueBackendOp select on the operation context, so cancelling one that is
still waiting releases the delivery goroutine and abandonQueued retires it: the
worker never sees it, nothing is downloaded, nothing is deleted. markQueued
nevertheless wrote Cancellable: !deletion, so a queued removal reported
cancellable: false and the UI hid the Cancel button in the one window where
pressing it both works and leaves no trace. A removal queued behind a large
install was stuck there until the install finished.
A running removal is not cancellable at all. DeleteModel and DeleteBackend take
no context, and modelHandler only checks the operation context after the call
returns, so a "cancelled" verdict would land after the model was already gone.
Both handlers nevertheless wrote Cancellable: true unconditionally at entry,
ahead of the op.Delete branch, offering a Cancel button the server cannot
honour.
So the queued phase is more cancellable than the running phase, which is the
reverse of the usual shape. markQueued now reports true unconditionally, and
the handler-entry writes report !op.Delete. Both sites carry a comment saying
why, because reading either one alone suggests the other is a bug.
GalleryStore.Create keeps !op.Delete: it runs at dequeue, so its value already
describes the running phase. Its comment now says so.
Specs cover queued removal, queued install, running removal and running install
through the handlers, plus the queued-removal case through /api/operations
where the flag is consumed, plus the behaviour the whole asymmetry rests on: a
removal cancelled while queued never reaches the worker and deletes nothing.
No existing spec asserted the old values.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Write] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(activity): clamp the installer message, and add a real-binary e2e spec
Running the page against a real local-ai showed the legacy installer message
wrapping to three lines and dominating the card: it embeds an absolute file
path, so it is both long and a single unbreakable token. One line, ellipsised,
full text in the title, matching what the error string already does.
The spec that found it runs with no route stubbing at all. Every other spec
here stubs /api/operations, which is how a payload the server cannot emit
(isDeletion true on a live operation) stayed green through a full review while
the UI rendered a removal as an install. It is skipped unless
LOCALAI_REAL_BINARY is set, so CI is unaffected.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Publish the existing Kokoro CPU profile for amd64 and arm64 and use it as the default gallery capability so Vulkan-only and CPU hosts can install the backend.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
sherpa-onnx links onnxruntime's CUDA execution provider, and
libonnxruntime_providers_cuda.so carries cuDNN as a hard DT_NEEDED. The
onnxruntime GPU tarball ships no cuDNN of its own, and Dockerfile.golang
only installs libcudnn9 on the arm64 + CUDA 13 branch, so the amd64 CUDA
builders have none at all.
Since #10946 added the packaging guard, that combination is fatal rather
than silent: package-gpu-libs.sh reports 'cuDNN: venv=absent system=absent
-> bundle=detect', correctly detects the reference, finds nothing to copy
and refuses to emit the package. Both -gpu-nvidia-cuda-12-sherpa-onnx and
-gpu-nvidia-cuda-13-sherpa-onnx have failed to build since 2026-07-19, so
neither image has been published. Before the guard existed they shipped
without cuDNN and failed at load time instead.
Install the runtime package for this backend only. The auto-detection
bundles solely what a package references, so no other backend would grow,
but every Go CUDA builder would pay ~1.1 GB of layer and registry cache
for a library ggml never calls.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
RecordBackendTrace does a non-blocking send onto a 100-slot channel and
drops when it is full, so tracing never stalls inference. The payload
bounding specs pushed all 200 traces in one tight loop, which overruns
that channel on a loaded machine: entries are dropped for good and the
Eventually waiting for 200 can never be satisfied, no matter the timeout.
CI hit this on master at 0a8a7fbb, settling at 158/200.
Feed the traces in chunks of 50, draining after each, so the channel is
never overrun and the count stays exact. Reproduced with 60 busy loops on
a 20-core box at GOMAXPROCS=2: 0/12 runs passed before, 12/12 after.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(tts): forward the OpenAI speed field to the backend (#11097)
/v1/audio/speech accepted the documented OpenAI `speed` field and then
dropped it: schema.TTSRequest had no Speed member, so the value never
reached proto.TTSRequest and the request returned 200 with an unchanged
playback rate.
Accept speed and normalise it into the existing per-request params map,
which core/backend forwards verbatim to the backend. An explicit
params["speed"] still wins, and a value outside the documented 0.25-4.0
range is now rejected with 400 instead of being silently ignored.
Signed-off-by: Anai-Guo <antai12232931@outlook.com>
* fix(tts): distinguish explicit speed=0 from an omitted field
Make TTSRequest.Speed a *float32 so an explicit `"speed": 0` (invalid,
below the documented 0.25 minimum) is rejected with 400 instead of being
treated as unset and silently defaulted. An omitted field stays nil and
leaves the backend default untouched.
Add a request-boundary regression that distinguishes an omitted speed from
an explicit zero, addressing review feedback.
Signed-off-by: Anai-Guo <antai12232931@outlook.com>
* docs: drop the speed field from the TTS docs
Per review: no backend consumes params.speed today, so documenting it
would be misleading. The API-level plumbing and validation stay.
Signed-off-by: Anai-Guo <antai12232931@outlook.com>
---------
Signed-off-by: Anai-Guo <antai12232931@outlook.com>
* fix: CVE-2026-54290 security vulnerability
Automated dependency upgrade by OrbisAI Security
Signed-off-by: orbisai0security <mediratta@gmail.com>
Signed-off-by: Anupam Mediratta <mediratta@gmail.com>
* fix(deps): override hono transitive dep to eliminate CVE-2026-54290
Add package.json `overrides` field to force hono@4.12.25 across the
entire dependency graph, including the transitive copy pulled in by
@modelcontextprotocol/sdk. Previously bun.lock retained a scoped
`@modelcontextprotocol/sdk/hono` entry resolved to the vulnerable
hono@4.12.8; the override removes that entry so only the patched
version ships.
Assisted-by: Claude Code:claude-sonnet-4-6
Signed-off-by: Anupam Mediratta <mediratta@gmail.com>
---------
Signed-off-by: orbisai0security <mediratta@gmail.com>
Signed-off-by: Anupam Mediratta <mediratta@gmail.com>
* fix(ci): build the CUDA 13 image on Ubuntu 24.04
The amd64 `-gpu-nvidia-cuda-13` image is the only runtime image still
built FROM ubuntu:22.04. The Ubuntu 24.04 migration (#7769) bumped its
`ubuntu-version` to 2404 but left `base-image` on jammy, so the image
ships glibc 2.35 while adding the noble CUDA apt repository, and every
backend it unpacks is built on noble.
Backends therefore cannot dlopen the libraries they bundle. The vLLM
backend dies at import time with:
OSError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not
found (required by /backends/cuda13-vllm/lib/libnuma.so.1)
and torchcodec finds no usable libavutil because jammy ships ffmpeg 4.x
(libavutil.so.56) while torchcodec looks for .so.57 through .so.60.
Add a spec over the build matrices that fails when a base image and the
`ubuntu-version`/`ubuntu-codename` it is paired with disagree, or when
the runtime images are split across Ubuntu releases. Entries whose base
image does not name a release (JetPack) are left alone.
The `base-grpc-cuda-13-amd64` builder base stays on jammy: it only
compiles backends, and a lower glibc floor in a builder is safe.
Fixes#11059
Assisted-by: Claude:claude-opus-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ci): drop the build matrix invariant spec
Per review, the CI matrix guard does not belong in the tree. Only the
base image bump remains.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): stop the probe reaper from orphaning busy backends
The reconciler's liveness probe is a 1s gRPC HealthCheck, and a single
failed probe deleted the model's node_models row. A backend that is
merely busy cannot answer it: single-threaded Python backends (video and
avatar generation) block for minutes inside one request, so the reaper
was deleting registry rows for backends that were alive and mid-request.
The model then vanished from the nodes page while it was still
generating, and because the row was gone the in-flight decrement had
nothing to decrement ("DecrementInFlight: no matching row or already
zero"). Every subsequent request re-routed and re-staged the full model
from scratch.
Two guards:
- Replicas with in-flight requests are excluded in SQL. A row that is
actively serving is proof of life, and the running request is
exactly what stops the backend from answering the probe.
- Idle replicas must miss three CONSECUTIVE probes before removal, so
a transient blip cannot orphan a live replica. A successful probe
resets the streak.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(distributed): drop the local model stub when its last replica goes
In distributed mode every routed model leaves an in-process stub in the
frontend's ModelLoader, and DistributedModelStore.Range reports local
stubs UNION the registry rows. Every registry removal path deletes only
the DB row, so the stub outlived the replica and the model was reported
as loaded forever.
That is the "loaded on the home page, absent from every node" ghost:
/system reads the union and still sees the stub, while /api/nodes/models
reads the registry and correctly sees nothing. It never self-healed,
and both frontend replicas showed it independently.
The replica-removed chokepoint could not fix this as it stood, because
it held a SINGLE hook that the prefix cache already owned, and it was
registered only when the prefix cache was enabled. Registering a second
listener would have silently displaced the first.
- Turn replicaRemovedHook into a list (AddReplicaRemovedHook), so
independent subsystems can each register without displacing others.
- Add NewLocalStubInvalidator, which drops the local stub once no
healthy replica of the model remains anywhere in the cluster, and
wire it unconditionally in startup.
The stub is kept while another node still serves the model: the
frontend is right to consider it loaded, and each request re-routes
through SmartRouter to pick a live replica anyway.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(distributed): stop staging checksum sidecars back to workers
The file transfer server writes a "<file>.sha256" sidecar next to every
file it accepts. The sender walked the model directory with no filter,
so it staged those sidecars too, and the receiver duly wrote a sidecar
for each sidecar. Every staging pass multiplied the tree:
config.json -> config.json.sha256 -> config.json.sha256.sha256 -> ...
One LongCat snapshot had grown to 498 files, 466 of them chained, up to
29 levels deep, and the staged file count climbed on every pass. This
inflates each transfer and grows disk without bound on both ends.
Skip hash sidecars in stageDirectory, and mirror the skip in
countStageableFiles so the progress bar still reaches 100%. The check is
"a sidecar sitting next to a real file" rather than a blanket suffix
ban, so a model that genuinely ships a .sha256 payload with no
corresponding base file is still transferred.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(distributed): classify the liveness probe instead of gating on in_flight
The previous commit excluded replicas with in-flight requests from the probe
reaper. That was the wrong guard, and could invert the bug it fixed.
in_flight has no decrement guarantee: track() balances its increment with a
defer, but a frontend killed mid-request never runs it, and the load-time
reservation is released only when the first inference completes. Nothing
resets a leaked counter. Gating the reaper on it therefore meant a leaked
counter would shield a genuinely dead replica from ever being reaped.
Nor was patience alone a fix: three misses at the default interval is ~90s of
silence, while the generation that triggered this blocks for 15+ minutes.
The real conflation was in the probe itself. A gRPC HealthCheck against the
backend's serving port measures "is it idle enough to answer", not "does the
process exist", and probeLoadedModels discarded the error that tells them
apart. Because the gRPC client is lazy, the status code is decisive:
- DeadlineExceeded: transport fine, nothing serviced the RPC. Busy.
- Unavailable: nothing is listening. Gone.
ModelProber now returns a ProbeOutcome, and only ProbeUnreachable counts
toward the reap threshold. ProbeBusy clears the streak: it is evidence of
life. A blackholed network reads as busy too, deliberately, since whole-node
failure is the health monitor's job.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(distributed): reconcile replicas against worker-reported processes
Probing a backend's own serving port cannot distinguish "busy" from "gone"
without inferring it from an error code. The worker can answer directly: it
spawned the process, holds the handle, and its reply is not blocked by
whatever that backend is doing.
Adds a models.running request-reply subject. The worker answers out of its
in-memory process table, reporting each live process as (modelID,
replicaIndex, address) — the supervisor's process keys are `modelID#replica`,
which is isomorphic to a NodeModel row, so the reconciler can diff the two
directly.
reconcileNodeProcesses runs before the port probe and reaps rows for models
the worker is not running. Models the worker vouches for get updated_at
bumped, which takes them out of the port prober's stale set entirely: that is
what keeps a backend deep in a long generation away from the probe in the
first place, rather than relying on classifying its silence after the fact.
A worker that does not answer is skipped, not assumed empty. A messaging
failure says nothing about the processes, and assuming the worst would delete
a node's rows on a transient NATS blip; the port probe stays as the fallback
for those nodes. Rows younger than probeStaleAfter are ignored so a freshly
created row is never judged against a process table that has not caught up.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(distributed): stop in_flight leaking and pin replicas against eviction
A leaked in_flight counter is not cosmetic. FindLRUModel,
FindGlobalLRUModelWithZeroInFlight and the router's eviction query all require
in_flight = 0, so a replica whose counter never came back is pinned and its
VRAM is unreclaimable for the lifetime of the process.
Two halves.
The source: routing reserves in_flight = 1 at load time so a freshly loaded
replica is not evicted out from under the request that caused the load. That
reservation was released ONLY by the first inference completing, so a route
torn down before any inference ran (client disconnect, handler error, failure
between load and the backend call) stranded it. newRouteResult now wires the
reservation to a sync.Once fired by whichever comes first, the first inference
or route teardown, and replaces three copies of the old wiring.
The backstop: a sweeper for counters leaked by paths that cannot run a defer
at all, such as a frontend killed mid-request.
Identifying a leak by elapsed time alone is unsafe. IncrementInFlight stamps
last_used at request START and nothing moves it while the request runs, so a
long generation is indistinguishable from a leak by age, and resetting there
would expose a serving model to eviction. The probe supplies the missing bit:
a backend that answers a health check promptly is not inside a request,
because that is precisely what a busy one cannot do. Requiring the row to also
be idle for 30 minutes covers backends that serve in parallel and can answer
while working, since those keep last_used fresh through each new increment.
Two existing tests asserted the old behaviour ("No decrement on Release").
That assertion was the leak, so both now pin the release instead.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Bump LLAMA_VERSION to 0d47ea7427463093e69128bf2c2f9cd06b3ee5b3 (73 commits
touching common/, src/ and tools/server/). Two upstream changes break the
backend:
* ggml-org/llama.cpp#20834 folded common_params::use_mmap / use_mlock /
use_direct_io into a single `load_mode` enum. LocalAI still exposes the three
as independent settings (`mmap`, `mmlock`, and the `direct_io` option), so
params_parse folds them once all three have been read, keeping the precedence
the separate booleans had: direct I/O bypasses the page cache, mlock implies
mmap, everything off is a plain buffered read. turboquant and bonsai compile
this same grpc-server.cpp against forks that predate the refactor, so
prepare.sh probes the checkout for LLAMA_LOAD_MODE_MMAP and generates
llama_compat.h with LOCALAI_LEGACY_LOAD_MODE set accordingly. Probing beats a
per-fork build flag here because the fork flavor targets disagree on whether
they forward CMAKE_ARGS or EXTRA_CMAKE_ARGS, and it heals itself once a fork
rebases past the refactor.
* The MiniMax M3 patch no longer applies. Upstream merged the model half of
llama.cpp#24523 (LLM_ARCH_MINIMAX_M3, src/models/minimax-m3.cpp, the gguf-py
constants and conversion/minimax.py) but not the chat half, so the patch is
re-cut to carry only the common/chat.cpp template detection and PEG parser,
rebased onto the new pin and onto the thinking_end_tag -> thinking_end_tags
rename. Dropping it wholesale (as #11008 did, reverted in #11136) would have
silently regressed MiniMax M3 tool calling and thinking.
Verified with a CPU docker build of the backend plus LoadModel and Predict
against a real GGUF over gRPC in all four load modes.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(realtime): accept legacy 'modalities' alias for output_modalities
OpenAI's Realtime *beta* used the field name `modalities`; the GA field is
`output_modalities`. LocalAI only binds `output_modalities`, so a client
sending the still-common beta field `modalities: ["text"]` has it silently
dropped by encoding/json and the session falls back to audio: TTS runs and the
client receives large response.output_audio.* frames even though it asked for
text-only.
Accept `modalities` as an alias on both session.update (RealtimeSession) and
response.create (ResponseCreateParams). The GA `output_modalities` wins when
both are present, so GA clients are unaffected. Applied at the two existing
resolution points via a small modalitiesWithAlias helper.
Fixes#11103
Signed-off-by: Anai-Guo <antai12232931@anaiguo.com>
* test(realtime): add JSON-boundary regression for modalities alias
Decode representative session.update and response.create payloads that
carry only the legacy beta `modalities` key and assert the effective
output modality resolves to text (not audio), reproducing the exact
expressions used in updateSession and triggerResponseAtTurn. This guards
against a wrong JSON tag or a missed call site letting encoding/json drop
the alias silently.
Also document output_modalities (and the accepted legacy modalities
alias) for text-only sessions in the realtime feature docs.
Signed-off-by: Tai An <antai12232931@outlook.com>
---------
Signed-off-by: Anai-Guo <antai12232931@anaiguo.com>
Signed-off-by: Tai An <antai12232931@outlook.com>
Co-authored-by: Anai-Guo <antai12232931@anaiguo.com>
* feat(backend): add vllm-cpp text-generation backend (vllm.cpp)
Wrap https://github.com/mudler/vllm.cpp - the LocalAI-team from-scratch C++20
port of vLLM (paged KV cache, continuous batching, prefix caching, safetensors
+ GGUF loading, no Python at inference) - as a Go gRPC backend over its stable
C ABI (ABI v2) via purego.
Backend (backend/go/vllm-cpp):
- Load -> vllm_engine_load: accepts a .gguf file or a config.json model dir
(anything else is refused, satisfying the greedy-probe rule); context_size
maps to max_model_len, options block_size/num_blocks/max_num_seqs size the
KV cache and scheduler admission.
- Predict -> vllm_complete (blocking); PredictStream -> vllm_complete_stream
with the per-delta C callback bridged into the gRPC stream. The backend
embeds base.Base (not SingleThread): concurrent requests batch continuously
in the engine's shared AsyncLLM scheduler.
- PredictOptions.Grammar -> the ABI's structured_grammar (GBNF), giving
grammar-constrained tool calling at parity with llama-cpp; the ABI also
exposes JSON-schema/regex/choice constraints.
- Hand-mirrored POD structs with layout locked by unit tests
(unsafe.Offsetof vs the C offsets) and a runtime vllm_abi_version gate.
- One portable library per platform (vllm.cpp uses per-file SIMD tiers with
runtime dispatch), so no avx/avx2/avx512 variant builds.
Wiring:
- backend-matrix: CPU amd64+arm64 (per-arch + manifest merge), CUDA 12/13
amd64 (120a;121a Blackwell fat binary), L4T arm64 (121a, GB10/DGX Spark -
the runtime-proven GPU target), Vulkan amd64, and Darwin arm64 Metal.
- backend/index.yaml meta + 12 image entries (latest/development x cpu,
cuda12, cuda13, l4t, vulkan, metal); bump_deps registration for the
VLLM_CPP_VERSION pin; root Makefile registration; test-extra runs the unit
specs (pure Go, no engine build).
- Importers: preference-only swaps - llama-cpp (GGUF) and vllm (safetensors)
advertise vllm-cpp via AdditionalBackends and emit backend: vllm-cpp
without tokenizer templating (the C ABI takes the FINAL prompt; templating
and tool parsing stay LocalAI-side). No auto-detect importer.
- Docs: backends list, top-level README maintained-engines table,
compatibility table.
Verified: 20/20 Ginkgo specs against the real pinned engine and
Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU - blocking + streaming parity, greedy
determinism, stop words, GBNF-constrained generation, and 4 concurrent
streams; plus a dlopen/ABI-gate smoke of the built gRPC server binary.
Upstream ABI v2 + production structured-output wiring landed as
mudler/vllm.cpp@86013f3.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): ride the autoparser code path - engine-side chat templating and tool engagement (ABI v3)
The backend now implements AIModelRich (PredictRich / PredictStreamRich) over
vllm.cpp's ABI v3 chat entry points, so chat and tool calling ride the SAME
code path as the llama.cpp autoparser: the ENGINE renders the model's chat
template, decides when a tool call engages, and parses it - LocalAI receives
pre-parsed ChatDelta / ToolCallDelta protos exactly as it does from llama-cpp.
- With use_tokenizer_template + structured Messages, PredictOptions lowers to
ONE OpenAI chat request JSON (messages, tools, tool_choice, sampling,
stream_options.include_usage) for vllm_chat / vllm_chat_stream. tool_choice
auto lowers engine-side to a LAZY structural-tag decode constraint - free
text until the model emits the tool trigger, then the call is
grammar-constrained; required/named force a call. Tool output is parsed by
the engine's streaming Hermes-style parser; each chat.completion.chunk maps
onto ChatDeltas (content / reasoning_content / tool_calls) which the host
already prefers over Go-side tag extraction. Without structured messages the
plain path (LocalAI templating + optional GBNF grammar) applies unchanged.
- The engine resolves the chat template from the GGUF tokenizer.chat_template
metadata (or tokenizer_config.json); templates beyond its minja subset -
e.g. the full Qwen3.5 namespace()/macro template - degrade engine-side to a
Hermes-aware fallback prompt (tools schemas + <tool_call> instruction) with
a stderr witness, so structural-tag engagement keeps working.
- Importers now emit the same config shape as llama-cpp for vllm-cpp
(use_tokenizer_template: true, no-grammar autoparser flow); only the
llama-cpp-specific use_jinja option and the vllm-python parser options are
dropped.
- Pin bumped to mudler/vllm.cpp@aaed7ec (ABI v3 + chat-prompt resolution).
Verified against the real engine and Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU: full
suite green - blocking chat, streaming deltas concatenating byte-equal to the
blocking answer, a REQUIRED tool call returning schema-valid arguments JSON,
and an AUTO run where the engine itself engages get_weather and streams parsed
tool deltas; plus unit specs for the request lowering, chunk->ChatDelta
mapping, and the C struct mirrors (ABI gate now v3).
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): ABI v5 - engine-side parser selection for 30 tool dialects + reasoning
Bump the vllm.cpp pin to the autoparser-parity engine: 30 tool-call dialects
(every pure-text parser in the pinned vLLM registry, each ported 1:1 with its
upstream tests), 7 reasoning parsers, google/minja as the template renderer
(the full Qwen3.5 template now renders engine-side), per-family structural
tags (tool_choice required/named compiles the model's NATIVE syntax where
expressible), and template auto-detection for both parser axes.
Backend changes:
- cModelParams mirrors ABI v5 (tool_parser + reasoning_parser fields,
layout-locked by the offset tests; ABI gate now v5).
- New model options tool_parser:<name> / reasoning_parser:<name> pass through
to the engine; unset means template auto-detection (18-row tool marker
table; [THINK]->mistral, <think>->think_auto for reasoning); "none"
disables the reasoning split; unknown names fail the first chat call.
- Chat chunks parse the `reasoning` field (the pin renamed
reasoning_content), flowing into ChatDelta.ReasoningContent which the host
already prefers.
Live e2e against Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU, full suite green: the
real chat template renders (no more fallback), reasoning auto-detection picks
think_auto so markerless answers stay pure content (the live run caught the
deepseek_r1 content-swallow upstream and drove the think_auto fix), required
tool_choice returns schema-valid arguments, auto tool_choice engages
engine-side and streams parsed deltas, and blocking/streaming stay
byte-identical. Turn latency also dropped (proper template EOS behavior).
Upstream program landed as mudler/vllm.cpp 86013f3..5fffe7e (ABI v2-v5,
minja, parser waves B1/B2/B4, reasoning seam, structural-tag registry,
think_auto).
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(vllm-cpp): bump the engine pin to the ENG-wave close-out
mudler/vllm.cpp@df8909b: the six engine-backed vLLM tool-parser families
(qwen3-coder/xml/mimo, kimi_k2, glm45/47, minimax_m2, gemma4, seed_oss)
text-reimplemented from their wire formats and held to the upstream test
suites - 39 registered dialects; the pinned vLLM registry is now covered
except the three Rust/Harmony-backed families, descoped by decision. kimi_k2
also gains a full native structural-tag builder; four new template
auto-detection rows land with test-pinned ordering.
Full backend e2e re-run green against Qwen3.5-2B-UD-Q8_K_XL.gguf on CPU.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): add the vllm-cpp-development gallery meta
The gallery grew the twelve latest/development image entries but was missing
the separate vllm-cpp-development meta (own capabilities map targeting the
-development image names), which every backend ships so the development
gallery resolves per-platform. Validated: all capability targets in both
metas resolve to existing entries, and every image URI's tag suffix matches
a backend-matrix build.
Also full-stack verified in this change's context (single-node local-ai from
this branch, locally-built backend under --backends-path, Qwen3.5-2B GGUF):
/v1/chat/completions non-stream (clean content + usage), streaming (SSE
deltas), tool_choice auto engaging get_weather engine-side with schema-valid
arguments and finish_reason=tool_calls, and streamed tool-call deltas in the
standard name-first cadence.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): repair the CI backend builds - gcc-14 -Werror + fat-arch Triton
Two distinct failures took down all five vllm-cpp backend builds on the PR:
1. gcc-14 (ubuntu:24.04 CI images; the local toolchain is gcc-13) fails the
engine build with -Werror=maybe-uninitialized in InputBatch::condense - a
false positive through a staging std::optional's raw storage. Fixed
upstream (mudler/vllm.cpp@61f3e85) by moving slot-to-slot directly;
verified BOTH ways under dockerized g++-14.2 (unfixed reproduces CI's two
diagnostics exactly, fixed compiles clean) with the engine's behavior
suites green. Pin bumped to that sha.
2. The amd64 CUDA builds died at CMake configure: the vendored Triton-AOT
cubin trees are per-arch and the engine refuses -DVLLM_CPP_TRITON=ON on a
multi-arch (120a;121a) fat build unless pinned to one tree, which would be
unsound for the other arch. Triton is now enabled only on the single-arch
arm64/GB10 build (where the cubins matter); the fat amd64 binary uses the
engine's non-AOT GDN path.
Backend e2e re-run green at the new pin (Qwen3.5-2B on CPU, full suite).
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): cuda-12 images cannot compile compute_121a - target 120a only
The second CI round surfaced a CUDA-version constraint: the cuda-12 (12.8)
image's nvcc rejects 'compute_121a' (GB10 arch support landed with CUDA 13),
killing the amd64 cuda-12 build at nvcc. Gate the architecture list on
CUDA_MAJOR_VERSION (exported by Dockerfile.golang): cuda-12 builds consumer
Blackwell 120a only, cuda-13 keeps the 120a;121a fat binary, arm64/l4t
(cuda-13) keeps single-arch 121a with the Triton cubins. GB10 is arm64, so
the amd64 cuda-12 image never served it - no capability change.
Verified by Makefile dry-run variable dumps for all three combinations
(cuda12 -> 120a; cuda13 -> 120a;121a; cpu -> CUDA off).
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): drop the cuda-12 variant - the engine needs the CUDA 13 toolchain
Third CI round, third layer: with the arch list already narrowed to 120a,
the cuda-12 (12.8) build still dies in ptxas compiling the sm_120a NVFP4 MMA
kernels ("Vector type too large, exceeds 128 bit limit") - the Blackwell fp4
path genuinely requires the CUDA 13 toolchain, and vllm.cpp supports
Blackwell-family GPUs only. Shipping a cuda-12 image without the fp4 kernels
would be a crippled build of an engine whose whole GPU story is fp4, so the
variant is dropped instead:
- backend-matrix: cuda-12 vllm-cpp entry removed (cuda-13 amd64, l4t arm64,
cpu, vulkan, metal remain).
- gallery: cuda12 image entries removed; the nvidia capability now resolves
to the cuda13 image in both metas; the nvidia-cuda-12 key is dropped so
older-driver hosts fall back to the CPU image instead of an unrunnable one.
- backend Makefile: BUILD_TYPE=cublas under CUDA_MAJOR_VERSION=12 now fails
fast with a clear message; cuda-13 keeps the 120a;121a fat binary and
arm64/l4t keeps 121a with the Triton cubins.
Verified: Makefile branch dumps for all four combinations (cuda12 loud
error, cuda13 fat, arm64 121a+Triton, cpu off), YAML parses, matrix filter
tests green, gallery capability targets all resolve.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): forward multi-turn tool identity and reasoning to the engine
chatRequestJSON dropped Message.ToolCallId and Message.Name on role="tool"
replies and Message.ReasoningContent on assistant history, so a second
turn after tool execution reached the engine's chat template without the
fields that bind a tool result to the call it answers. Forward all three
(present-only, matching the OpenAI wire shape) and pin vllm.cpp to
6a0bd3e7, where ChatMessage parses/round-trips tool_calls, tool_call_id,
name and reasoning and the minja adapter exposes them to the template
context.
Adds the round-trip request-lowering spec (user -> assistant tool_call ->
tool reply -> lowered request) and re-ran the gated e2e suite against the
new engine pin with a real Qwen3.5 GGUF: chat, reasoning split, streaming
parity, required-tool and auto-tool cases all green.
Assisted-by: Claude Code:claude-fable-5 [Bash] [Edit] [Read]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): bump vllm.cpp for the darwin arm64 i8mm build fix
The darwin-metal CI job was the first build to compile the engine's arm
CPU-quant files on macOS and hit their Linux-only <asm/hwcap.h> /
<sys/auxv.h> includes. vllm.cpp 9e1c9025 detects i8mm per-OS (auxv on
Linux, sysctl on Apple Silicon) with kernels untouched. Gated e2e suite
re-run green against the new pin with a real Qwen3.5 GGUF.
Assisted-by: Claude Code:claude-fable-5 [Bash] [Read]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): darwin build - bound cmake parallelism when nproc is absent
The macOS runners have no nproc, so JOBS evaluated empty and
`cmake --build -j$(JOBS)` became bare `-j`: unlimited clang jobs on a
3-core/7GB Mac, which swap-thrashed until the 6h GHA timeout (the log
shows "nproc: Command not found" and 7+ concurrent clang processes being
reaped at the cutoff). Use the same portable fallback chain as the other
darwin backends: nproc, then sysctl hw.ncpu, then 4.
Assisted-by: Claude Code:claude-fable-5 [Bash] [Edit] [Read]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
feat(distributed): cache staged-artifact hashes and publish load lifecycle
Every load request re-hashed every staged artifact on the controller
(probeExisting and the upload path both re-read the full file), which for
a large multi-file model on NAS-backed storage is minutes of pure
re-reading per request even when nothing changed - observed as ~9 minutes
of "Upload skipped (file already exists with matching hash)" before every
avatar generation. Cache the local hash in the same .sha256 sidecar the
worker-side transfer server already maintains, invalidated whenever the
sidecar is older than the file.
The whole staging+loading phase was also invisible: the NodeModel row was
only written after LoadModel succeeded, so /api/nodes and the UI showed
nothing while a cold load spent 10+ minutes staging - indistinguishable
from nothing happening. Publish the lifecycle instead: "staging" as soon
as the node is chosen, "loading" when the checkpoint load starts, and the
existing "loaded" on success, with the row removed on any failure so a
dead load does not leave a phantom replica. The early row also reserves
the replica slot against concurrent schedulers. The nodes view already
renders non-loaded states on model chips; style "staging" like "loading".
Audited every state-filtered registry/router query: eviction, routing,
reconciler and idle-model queries all filter state='loaded' explicitly,
so the new transitional rows are visible to observability surfaces but
inert to scheduling decisions (except slot occupancy, intentionally).
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(gallery): coalesce artifact download progress
Buffer high-frequency downloading events and forward only the latest event on a periodic tick. Flush progress synchronously at phase boundaries and shutdown to preserve ordering and final state.
Assisted-by: Codex:gpt-5
* fix(gallery): wire progress coalescing into model installs
Route artifact progress through the 250 ms coalescer and flush it on every model operation exit. Keep the legacy download callback unchanged.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(backend): add magpie-tts-cpp text-to-speech backend
Add a Go + purego backend wrapping the magpie-tts.cpp ggml port of NVIDIA's
Magpie TTS Multilingual 357M (encoder + autoregressive decoder over NanoCodec
tokens), producing 22.05 kHz mono audio in 5 baked voices (Aria, Jason, John,
Leo, Sofia; case-insensitive names or indices 0-4) across 9+ languages from a
single self-contained GGUF. Mirrors qwen3-tts-cpp / moss-tts-cpp: dlopen the
static-ggml shared library, bind the flat magpie_tts_capi_* C-API via purego
(no local C shim needed, the upstream .so exports it directly), and serve the
gRPC TTS + TTSStream methods behind base.SingleThread (the C context is not
reentrant across synthesize calls).
The backend CMakeLists translates the Makefile's -DGGML_{CUDA,METAL,VULKAN,HIP}
flags into upstream's MAGPIE_GGML_* toggles (upstream FORCE-overwrites the ggml
cache entries from those), pinned to magpie-tts.cpp v0.1.1
(e3f3dd1ebe22b64e7405f93b519f2d1930712568), which statically links ggml into
libmagpie-tts.so (ldd shows only system libs).
Wires the full registration: backend-matrix.yml (CPU amd64/arm64, CUDA 12/13,
Intel SYCL f16/f32, Vulkan amd64/arm64, ROCm, NVIDIA L4T + L4T CUDA 13, and
Darwin metal), backend/index.yaml metas and image entries, the root Makefile
build targets, the changed-backends backend-filter path mapping, the bump_deps
auto-bump matrix, a test-extra per-backend smoke job, the /backends/known
pref-only importer entry, the backend capabilities map (TTS + TTSStream, no
voice cloning), and the README / compatibility-table docs rows.
Verified locally: unit + e2e Ginkgo suites pass against the real q8_0 GGUF
(22.05 kHz mono WAV, RMS > 0.01), a live gRPC LoadModel + TTS round-trip
returns valid non-silent audio, and the pre-commit gates (make lint,
make test-coverage-check) pass, run manually with LOCALAI_TEST_HTTP_PORT
overriding the locally-occupied 9090.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* gallery: add magpie-tts-cpp model entries (q8_0 + f16)
Add the Magpie TTS Multilingual 357M GGUFs from mudler/magpie-tts.cpp-gguf to
the model gallery: q8_0 (~624 MB, near-lossless, fastest decode, recommended)
with an f16 (~784 MB) variant, both served by the magpie-tts-cpp backend.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* magpie-tts-cpp: bump pin to rewritten upstream v0.1.1 SHA
Upstream history was rewritten to purge accidentally committed build
artifacts; v0.1.1 now resolves to 6f7696cf.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Remove the versioned pre-commit hook and its installer while retaining CI coverage and conformance checks.
Assisted-by: Codex:gpt-5
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The stall watchdog arms as soon as the response body exists, but the
downloader then re-hashed the entire existing .partial before reading a
single byte from the network. On slow models storage (a CIFS share
reading at ~117MB/s) hashing a multi-GB partial outlasts the 60s stall
window, so the watchdog aborted every healthy resume with 'download
stalled: no data received for 1m0s'. The partial never grew, so every
retry re-paid the same hash and failed identically, wedging the install
permanently (any partial over ~7GB on such storage).
Open the partial and hash it before the HTTP request instead: the
watchdog now only measures actual network idle time, and the origin no
longer sits on an idle connection while the hash runs.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(trl): disable inline GRPO reward code by default (RCE)
POST /api/fine-tuning/jobs accepts reward_functions[].code, an inline Python
body, and compile_inline_reward() execs it against a restricted-builtins
allowlist (_SAFE_BUILTINS). That allowlist is not a security boundary:
().__class__.__bases__[0].__subclasses__() reaches os._wrap_close and thus
os.system, giving arbitrary code execution. The fine-tuning endpoint is
unauthenticated by default, so any caller could run code on the host.
Hardening the allowlist is a losing game against CPython introspection, so
inline reward code is now refused unless the operator explicitly opts in with
LOCALAI_TRL_ALLOW_INLINE_REWARD=true on the backend. Builtin reward functions
are unaffected. The gate lives in build_reward_functions(), the single point
all inline specs flow through. Docs updated to stop describing the allowlist as
a sandbox and to document the opt-in.
Fixes#11015
Signed-off-by: Isabel Wu <231155141+wuisabel-gif@users.noreply.github.com>
Co-authored-by: Isabel Wu <231155141+wuisabel-gif@users.noreply.github.com>
fix(model-artifacts): persist companion artifacts, not just the primary
A managed model can declare companion artifacts (LongCat-Video-Avatar-1.5
pulls its tokenizer, text encoder and VAE from the separate LongCat-Video
base repo via a target: companion artifact). preloadOne resolves the whole
set in memory, but the binding written back to disk carried only the
primary: persistArtifactBinding marshalled []Spec{result.Spec} and replaced
the entire artifacts: list with it, silently dropping every companion.
In a single process the loss is invisible because the in-memory config keeps
the companion. It bites on the next controller restart: the config reloads
from the mangled file with the primary alone, so withCompanionArtifactOptions
finds no resolved companion and synthesizes no base_model option. The remote
longcat-video backend then never receives base_model, falls back to
BASE_MODEL_ID and downloads the repo itself ("Downloading required files for
meituan-longcat/LongCat-Video"), failing the load with "base_model must point
to a LongCat-Video checkpoint".
This is why an explicit base_model:<path> added to the config options works
where the managed companion does not: an explicit option lives in options:,
which is never rewritten, while the managed companion lives in artifacts:,
which the binding overwrote.
Persist the full resolved set (primary + every companion), and widen
bindingNeedsPersistence to compare the whole artifact list so a companion
resolving for the first time still triggers a write. The single-node path is
unaffected: there the in-memory config already carried the companion, and the
staging/ModelPath resolution for a remote worker (nested per-model staged
root, #10949) is unchanged and already correct once the option is generated.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
2026-07-23 16:50:50 +02:00
788 changed files with 70361 additions and 9103 deletions
A new capability deserves its own page under `docs/content/features/`, plus cross-links from related features and an entry in `docs/content/whats-new.md`. See the pattern used by `face-recognition.md` / `object-detection.md`.
A new capability deserves its own page under `docs/content/features/`, plus cross-links from related features. See the pattern used by `face-recognition.md` / `object-detection.md`.
Announcing it is the release's job, not this page's: the capability gets covered in the release blog post under `website/content/blog/`. See [preparing-a-release.md](preparing-a-release.md). `docs/content/whats-new.md` is only a pointer at the blog and GitHub Releases, so there is nothing to add there.
## Path protection rules
@@ -334,7 +336,7 @@ When adding a new endpoint:
- [ ] Swagger block on the handler: `@Summary`, `@Tags`, `@Param`, `@Success`, `@Router`
- [ ] If new capability area (new swagger tag): entry in `instructionDefs` in `core/http/endpoints/localai/api_instructions.go` + test count bumped in `api_instructions_test.go`
- [ ] If new `FLAG_*` usecase flag: matching `CAP_*` symbol exported from `core/http/react-ui/src/utils/capabilities.js`
- [ ]`docs/content/features/<feature>.md` created; cross-links from related feature pages; entry in `docs/content/whats-new.md`
- [ ]`docs/content/features/<feature>.md` created; cross-links from related feature pages; capability covered in the release blog post (see [preparing-a-release.md](preparing-a-release.md))
**Quality**
- [ ] Error responses use `schema.ErrorResponse` format (or `echo.NewHTTPError` with a mapped gRPC status — see the `mapBackendError` helper in `core/http/endpoints/localai/images.go`)
@@ -28,7 +28,6 @@ The core Go suites (`./pkg`, `./core`, plus the in-process integration suite `./
- **Build tags (`COVERAGE_TAGS`, passed via `GINKGO_TAGS`):** defaults to `debug auth`. The `auth` tag is required to compile the real (sqlite-backed) auth implementation and its ~150 `//go:build auth` tests — without it those files aren't built, the tests don't run, and the gate scores auth against a stub (~3.7% instead of ~38%). If you add new tag-gated tests, extend `COVERAGE_TAGS` or they won't count (and likely won't run in CI at all).
-`make test-coverage-check` — runs `test-coverage`, then `scripts/coverage-check.sh` fails the build if total coverage is **below** the committed baseline in `coverage-baseline.txt`. The Linux job in `.github/workflows/test.yml` runs this instead of `make test`.
-`make test-coverage-baseline` — regenerates and overwrites `coverage-baseline.txt` from the current run.
-`make install-hooks` — sets `core.hooksPath` to the versioned `.githooks/`, whose `pre-commit` runs checks scoped to what's staged: Go changes → `make lint` + `make test-coverage-check`; `core/http/react-ui/` changes → `make test-ui-coverage-check` (Playwright e2e + UI coverage gate). A commit touching neither is skipped; bypass with `git commit --no-verify`. The hook resolves golangci-lint's new-from base to `upstream/master` → `origin/master` → `master`, so it works from a fork clone where `origin/master` is stale (passed to `make lint` via `LINT_NEW_FROM`).
### React UI coverage
@@ -38,12 +37,11 @@ The React UI (`core/http/react-ui/`) has **no component/unit tests** — its onl
- **Browser:** the flake dev shell ships `chromium` and exports `PLAYWRIGHT_CHROMIUM_PATH`; `playwright.config.js` uses it via `launchOptions.executablePath`, and the Makefile skips `playwright install` when it's set. This avoids Playwright's downloaded browser, which can't resolve system libs (`libglib-2.0`, …) on NixOS. In CI (no `PLAYWRIGHT_CHROMIUM_PATH`) the Makefile falls back to `playwright install --with-deps chromium`.
- The app is a React SPA, so coverage accumulates across in-app navigation within a test; a full `page.goto`/reload resets it.
-`.nycrc.json` uses `all: true`, so **every `src/**` file is in the report**, including 0%-coverage ones — that's how you spot features with no test at all (sort the HTML report or `coverage-summary.json` by line% ascending).
- **UI coverage gate:** `make test-ui-coverage-check` runs the suite then `scripts/ui-coverage-check.sh`, failing if total line coverage drops more than `UI_COVERAGE_TOLERANCE` below `core/http/react-ui/coverage-baseline.txt`. `make test-ui-coverage-baseline` regenerates the baseline. Runs in CI (`tests-ui-e2e.yml`) and pre-commit on `core/http/react-ui/` changes.
- **UI coverage gate:** `make test-ui-coverage-check` runs the suite then `scripts/ui-coverage-check.sh`, failing if total line coverage drops more than `UI_COVERAGE_TOLERANCE` below `core/http/react-ui/coverage-baseline.txt`. `make test-ui-coverage-baseline` regenerates the baseline. Runs in CI (`tests-ui-e2e.yml`).
- **Why it has a tolerance (unlike the strict Go gate):** UI e2e coverage is *non-deterministic*. Specs that assert on state and end while async/lazy render work is still in flight collect those lines only when the render beats the coverage teardown — so the total drifts with machine speed/load (a fast local box reads higher than a slow CI runner), diffusely across many specs. The tolerance absorbs that drift, so set the baseline *below* the slow-CI floor, never to a fast-local `make test-ui-coverage-baseline` number, or CI flaps.
- **Raising coverage is cheap:** a *render-smoke* spec (navigate to a route, assert its header renders) mounts a lazy page and runs its full render + initial effects, capturing most of its lines in a few lines of test — see `e2e/page-render-smoke.spec.js`. Auth is disabled in the test server (`isAdmin=true`), so `RequireAdmin`/`RequireFeature` routes render without a mock. The most *deterministic* win is removing a race: make a spec `await` a rendered element before ending (see `e2e/agents.spec.js` → AgentCreate) so its lines count every run.
Rules (both gates):
- **Install the hooks:** `make install-hooks` once per clone so lint + coverage run pre-commit. Don't lean on CI for what the hook catches.
- **Don't work around the gate:** never `git commit --no-verify`, and never hand-lower a baseline or widen a tolerance to turn a red gate green. The ratchet only moves up.
- **Don't weaken the gate:** never hand-lower a baseline or widen a tolerance to turn a red gate green. The ratchet only moves up.
- If a change drops coverage, **add tests** (sort `coverage-summary.json` by line% ascending to find untested code) rather than editing the baseline. When coverage legitimately rises, commit the regenerated baseline (`make test-coverage-baseline` / `test-ui-coverage-baseline`).
- The Go gate is **strict — no tolerance**; `covermode=atomic` keeps it deterministic. The UI gate keeps a small tolerance only because its e2e coverage isn't.
@@ -122,18 +122,89 @@ The per-backend prefix match only sees files under a backend's own directory, so
| Changed path | Rebuilds |
|---|---|
| `backend/backend.proto` | everything (all languages compile or copy it) |
| `backend/backend.proto` | nothing if the edit is additive-only, otherwise everything (see below) |
| `backend/Dockerfile.<x>` | the Linux entries whose `dockerfile:` names it |
| `backend/python/common/` | Python, Linux + Darwin |
| `scripts/build/package-gpu-libs.sh` | Python, Linux only |
| `scripts/build/package-gpu-libs.sh` | every Linux entry (Python, Go and C++ all run it) |
| `scripts/build/<lang>-darwin.sh` | the Darwin entries that build target routes to |
| `.github/workflows/backend_build[_darwin].yml` | everything on that OS |
| anything else under `scripts/build/` (except `*_test.sh`) | everything — conservative default for unclassified packaging inputs |
Deliberately excluded: `backend/index.yaml` (gallery metadata, never enters an image), `.github/backend-matrix.yml` (adding a backend would rebuild all of them), `backend/Dockerfile.base-grpc-builder` (owned by `base-images.yml`), and the root `Makefile` (touched in ~11% of commits, and its backend-relevant edits arrive alongside the backend directory anyway). `make test-ci-scripts` pins all of this.
#### `backend/backend.proto` is content-filtered, not path-filtered
Every language consumes the proto, so a path rule for it can only ever say "rebuild all 473 images". It changes in ~1.3% of commits, and that was enough to make it the single largest CI cost driver in the repo: on 2026-07-29 four runs totalling 935 queued jobs traced to nothing but a proto edit, one of which (#11158) was a six-line diff adding `bool cache_prompt = 8;`.
An additive proto edit cannot change how a backend that never references the new symbol behaves, so `filterMatrix()` suppresses the rule for one. `changed-backends.js` fetches `backend/backend.proto` at the base revision (same contents-API pattern as `.github/backend-matrix.yml`) and hands both texts to `protoChangeIsAdditive()`, which compares them structurally rather than textually:
- **Additive, rebuilds nothing**: a new field with an unused number, a new message, a new enum value, a new RPC. Comment, whitespace and ordering changes also land here.
- **Breaking, rebuilds everything**: a removed, renumbered, retyped or renamed field, a dropped RPC, a changed `option` or `package`. So does an unresolvable base revision, matching the run-all posture used for a truncated diff.
Checked against every proto commit in the preceding six months, all nine resolvable ones classify as additive. Note the tradeoff this accepts: generated stubs do change for an additive edit, so image bytes would differ on a rebuild even though behavior does not. That is the same standard already applied when the filter declines to rebuild on unrelated `pkg/` changes, and the weekly cron remains the backstop.
The Sunday 06:00 UTC cron on `backend.yml` exists specifically because path filtering can leave Python backends frozen on stale wheels. `DEPS_REFRESH` (below) only fires when the build actually runs, so an untouched Python backend would never re-resolve its unpinned deps. The weekly cron is the safety net.
## Content-blind PRs skip the workflows that cannot see them
`backend_pr.yml` and `test-extra.yml` filter themselves (matrix generation and a `detect-changes` job), so a gallery-only or docs-only PR costs them about one job each. The Go and image workflows had no filter of any kind, so a one-line `gallery/index.yaml` edit queued 20 jobs, and a docs-only PR queued the same.
This is worth more than it looks. Measured over the week to 2026-07-30, **97% of CI wall-clock is queueing, 3% is execution** (median queue ~5h against a 4-20min median job). Cutting job count is therefore the only lever that shortens feedback time; making individual jobs faster moves 3%.
The volume is real: 13 gallery-only PRs merged that week with 10 open at once, and 78 of the 137 PRs opened were bot-generated.
`paths-ignore` on the PR trigger of `image-pr.yml` (7 jobs), `build-test.yaml` (3), `lint.yml` (2) and `tests-e2e.yml` (1) drops 13 of those 20. The excluded set:
| Path | Why no image or Go build can see it |
|---|---|
| `gallery/**` | Model-gallery metadata, parsed at runtime, never copied into an image |
| `docs/**`, `examples/**`, `**/*.md` | Never enter an image or a binary. `lint.yml` already excluded these before gallery was added |
### `backend/{cpp,go,python}/**` on `image-pr.yml` and `build-test.yaml` only
Version-pin bumps dominate PR volume: 48 `update/*` PRs in the week to 2026-07-30, from 16 pins, each a two-line diff. Most edit nothing but one `backend/*/<name>/Makefile`.
Neither of those two workflows can observe such a change. `make build` is `go build ./cmd/local-ai`, GoReleaser builds the same plus `./cmd/launcher`, and the core image's final stage ships only `entrypoint.sh`, `healthcheck.sh` and that binary. The per-backend trees are copied into the builder but nothing in them reaches the output.
What still triggers a full run, because none of it lives under those prefixes:
-`backend/backend.proto` — feeds `protogen-go`, so it does change the binary.
-`go.mod` / `go.sum` — the `go mod tidy` before-hook.
-`backend/Dockerfile.*` and anything else directly under `backend/`.
Deliberately **not** applied to:
| Workflow | Why it must keep seeing `backend/**` |
|---|---|
| `test.yml` | `TEST_PATHS` explicitly includes `./backend/go/cloud-proxy/...`, `./backend/go/local-store/...` and `./backend/go/valkey-store/...` |
| `lint.yml` | `.golangci.yml` carries `backend/`-scoped rules, so golangci-lint covers that tree |
| `tests-e2e.yml` | The e2e suite drives real backends over gRPC |
| `backend_pr.yml` | This is the workflow whose entire job is to rebuild the changed backend |
What still runs, and why it has to:
| Workflow | Why it keeps running |
|---|---|
| `test.yml` (`tests`) | `core/gallery/variants_lint_test.go` reads the real `gallery/index.yaml` and asserts the index invariants (no duplicate entry names, no build claimed by two parents). This is the only schema-level check the gallery has. |
| `yaml-check.yml` (`Yamllint`) | Lints `gallery/` for syntax. |
| `backend_pr.yml`, `test-extra.yml` | Already self-filtering; they stop after the detect step. |
Two properties this relies on:
-`paths-ignore` skips a run only when **every** changed file matches, so a PR touching the gallery *and* Go code still runs everything. That is what makes the exclusion safe rather than a hole.
-`master` carries no branch protection and no rulesets, so a skipped workflow reports no status and nothing waits on it. If required status checks are ever introduced, these four entries must be excluded from the required set or PRs will hang on "Expected — Waiting for status to be reported".
### `image.yml` on master push is gated too, by a job rather than a path filter
The same reasoning applies to master pushes, and the volume is larger there: on 2026-07-30, **12 of the 23 queued `image.yml` runs** were commits like "add 1 new model to gallery" or a docs fix, each rebuilding all 18 container images.
`image.yml` now has a `changes` job that decides once whether the push can affect any image; the other 11 jobs carry `needs: changes` plus an `if:` on its output. Verified against the shipped `Dockerfile`: the final stage copies only `entrypoint.sh`, `healthcheck.sh` and the `local-ai` binary, there is no `go:embed` of `gallery/` or `docs/`, and the gallery is fetched at runtime from `github:mudler/LocalAI/gallery/index.yaml@master`. A gallery-only commit therefore produces byte-identical images, and the gallery change reaches users through GitHub immediately whether or not an image is rebuilt.
Two properties to preserve if you touch it:
- **It is a job gate, not `paths-ignore`.** `paths-ignore` on `push` also applies to tag pushes, and a tag created on an existing commit carries an empty commits list, which would silently skip the release image build. The gate short-circuits to "build" for `refs/tags/*`, and for any push whose base commit is missing, zero, or unresolvable.
- **The merge jobs must name the gate explicitly.** They use `if: ${{ !cancelled() && ... }}`, and `!cancelled()` is true when a dependency is *skipped*, so without the extra condition they would run and try to merge manifest lists for images that were never built.
## The `DEPS_REFRESH` cache-buster (Python backends)
Every Python backend goes through the shared `backend/Dockerfile.python`, which ends with:
@@ -169,15 +240,38 @@ RUN --mount=type=cache,target=/root/.ccache,id=<backend>-ccache-${TARGETARCH}-${
bash /usr/local/sbin/compile.sh
```
The compile script exports `CMAKE_C/CXX/CUDA_COMPILER_LAUNCHER=ccache` so CMake threads ccache through gcc/g++/nvcc. `cache-to: type=registry,mode=max` exports the cache mount data into the registry cache, so subsequent builds restore it.
The compile script exports `CMAKE_C/CXX/CUDA_COMPILER_LAUNCHER=ccache` so CMake threads ccache through gcc/g++/nvcc. Cache scope is per `(TARGETARCH, BUILD_TYPE)` so e.g. cublas-12 doesn't share with cublas-13 (their CUDA headers differ; cross-pollination would just be cache misses anyway).
On a `LLAMA_VERSION` bump, most translation units are byte-identical to the previous version's preprocessed source — ccache returns the previous `.o` and skips the real compile. Same for LocalAI source changes that don't actually touch llama.cpp's CMake inputs. Cache scope is per `(TARGETARCH, BUILD_TYPE)` so e.g. cublas-12 doesn't share with cublas-13 (their CUDA headers differ; cross-pollination would just be cache misses anyway).
### ⚠️ This ccache does nothing in CI today
This section previously claimed that `cache-to: type=registry,mode=max` "exports the cache mount data into the registry cache, so subsequent builds restore it". **That is not true.** BuildKit does not export the contents of a `--mount=type=cache` to a registry cache export. A cache mount lives in the builder's local state, and every CI job gets a fresh runner with a fresh builder, so `/root/.ccache` starts empty on every single build.
Measured on 2026-07-30 from the `ccache -s` output the compile script already prints (it runs `ccache -z` first, so the numbers are per-build):
| Job | Commit touched | Build time | ccache |
|---|---|---|---|
| 89766266951 (llama-cpp, cublas 13) | `backend/go/magpie-tts-cpp/Makefile` only | 6369s | **0 / 889 hits**, and 0 / 1778 |
The first two are the decisive control: commit `90355cd44` changed exactly one file, `backend/go/magpie-tts-cpp/Makefile`, nowhere near llama.cpp. The engine source was byte-identical to the previous build, which is precisely the case this section says ccache should serve, and the hit rate was still **0.00%**. A cache that was being restored but merely matching poorly would show partial hits; 0-of-N is the signature of an empty cache.
So the paragraph above about `LLAMA_VERSION` bumps reusing previous `.o` files describes an intended design that is not in effect. `Dockerfile.{llama-cpp,ik-llama-cpp,turboquant,bonsai,ds4,privacy-filter}` pay the ccache wrapper overhead and get nothing back. Multi-hour C++ rebuilds are recompiling identical translation units from scratch.
**Do not "fix" this by adding cache mounts to more Dockerfiles.** Wiring the same mount into `Dockerfile.golang` (215 of the 434 matrix entries) was measured locally at 18% faster on a rebuild after a source edit, with a 71.5% ccache hit rate — but only because the local test reused one builder across both builds. In CI it would be a no-op for exactly the reason above.
Making this actually work needs the cache to live outside the builder. The options, none of them free:
- **ccache `remote_storage`** (ccache ≥ 4.4, HTTP or Redis backend) or **sccache** with an S3/GCS/Redis backend. Genuinely works across runners; needs a cache service to point at. quay.io is a registry, not a blob store, so the existing infra does not cover it.
- **Round-trip the cache dir through `actions/cache` on the runner**: restore it, pass it in, and export it back out via a build stage output. No external infra, but clunky, and the repo already sits at GitHub's 10 GB cache ceiling while the llama-cpp ccache alone is capped at 5 GB.
Until one of those lands, treat C++ backend builds as always-cold and spend the effort on not running them instead (path filtering, see above).
## Composite actions
Two composite actions handle runner-side prep:
- **`.github/actions/free-disk-space/action.yml`** — wraps `jlumbroso/free-disk-space@main` plus an explicit apt purge of dotnet/android/ghc/mono/etc. Reclaims ~6–10 GB on `ubuntu-latest`. No-op on self-hosted runners. Used by `backend_build.yml`, `image_build.yml`, `test.yml`, `tests-aio.yml`, etc.
- **`.github/actions/free-disk-space/action.yml`** — wraps `jlumbroso/free-disk-space@main` plus an explicit apt purge of dotnet/android/ghc/mono/etc. Reclaims ~6–10 GB on `ubuntu-latest`. No-op on self-hosted runners. Used by `backend_build.yml`, `image_build.yml` and `base-images.yml` — the jobs that actually build images. Deliberately **not** used by `test.yml`, which runs no buildx step.
- **`.github/actions/setup-build-disk/action.yml`** — relocates Docker's data-root to `/mnt` on hosted X64 runners. GHA hosted `ubuntu-latest` ships ~75 GB of unused space at `/mnt`; combined with the free-disk-space cleanup this gives ~100 GB working space — enough for ROCm dev image + vLLM torch install + flash-attn intermediate layers. No-op on self-hosted and on non-X64 hosted runners. Used by `backend_build.yml`, `image_build.yml`, `base-images.yml`.
Both actions run before any docker buildx step.
@@ -218,10 +312,20 @@ Eviction is rarely needed in normal operation — `DEPS_REFRESH` handles weekly
## What the cache does **not** cover
- The `free-disk-space` and `setup-build-disk` composite actions run on every job — these reclaim runner-state, not Docker layers, so BuildKit caches don't apply.
- The `free-disk-space` and `setup-build-disk` composite actions run on every job — these reclaim runner-state, not Docker layers, so BuildKit caches don't apply.`test.yml` deliberately does **not** use `free-disk-space`: it runs no buildx step, and the multi-GB fixture downloads that once justified it left `make test` in the test-suite reorg.
- Intermediate artifacts of `Build (PR)` are not pushed anywhere — PRs only build for verification.
- Darwin builds (see below) — macOS runners have no Docker daemon, so the registry-backed BuildKit cache cannot apply.
### The Linux Go workflows set `cache: false` on purpose
`test.yml`, `lint.yml`, `tests-e2e.yml` and friends pass `cache: false` to `actions/setup-go@v5`, unlike the darwin jobs. This looks like an oversight and is not.
Measured over the week to 2026-07-30, the `Set up Go` step has a **median of 11 seconds** on these runners. There is essentially nothing to win: the module download is not where the time goes. The expensive steps are compilation and test execution (`Test (with coverage gate)` at ~18.6min, `Test Backend E2E` at ~14.5min), and Go's build cache would have to survive across runners to touch those.
Enabling it also has a real cost. GitHub caps Actions cache at **10 GB per repo and the repo already sits at that ceiling** (31 entries), so every `setup-go` entry written by a branch with a distinct `go.sum` (222-375 MB on Linux, up to 1.4 GB on macOS) evicts something else. See the darwin cache budget below.
Before re-enabling this, measure `Set up Go` again and confirm it has actually become slow. If room is needed in the 10 GB budget, the cheapest evictions are the `docker.io--tonistiigi--binfmt` entries (~30 MB each, trivially re-fetched).
## Darwin native caches
`backend_build_darwin.yml` runs natively on `macOS-14` GitHub-hosted runners — there is no Docker, no BuildKit, no cross-job registry cache. Instead, the reusable workflow uses `actions/cache@v4` for four native caches that mirror the spirit of the Linux cache (warm by default, weekly refresh for unpinned Python deps, PRs read-only).
@@ -255,6 +359,26 @@ GitHub Actions caches are limited to 10 GB per repo. Steady-state worst case: ~8
One residual self-hosted reference remains in `test-extra.yml` (`tests-vibevoice-cpp-grpc-transcription` uses `bigger-runner` for the 30s JFK-decode timeout headroom). That's a separate concern.
### Small always-on jobs routed to `arc-runner-set`
The hosted pool is shared across the whole *account*, not per repo, so a burst in one repo starves the others. On 2026-07-31 it went to **zero scheduled jobs for 35 consecutive minutes** with 39 jobs queued, while `arc-runner-set` completed 12 jobs without interruption over the same window. Actions was healthy globally at the time (other public repos were scheduling normally), so this is an account-level throttle, not an outage.
`gh-pages.yml` (`build` + `deploy`) is therefore routed to `arc-runner-set` when `github.repository == 'mudler/LocalAI'`. It needs no fork-safety clause because it only triggers on push-to-master and `workflow_dispatch`, so it never executes pull-request code. The repository guard keeps forks (which have no such runner label) from queueing forever. It fetches its own toolchains via `setup-go` / `actions-hugo` and uses no `sudo`/`apt`.
#### What the `arc-runner-set` image actually contains
Measured 2026-07-31 on run `30637392862` by a preflight step, not assumed:
That is why `lint.yml` is **not** on the self-hosted pool. Both of its jobs were routed there and both failed in one second: `golangci-lint` needs `make` (for `make protogen-go`, itself needing `curl`+`unzip` to fetch protoc, and for `make lint`), and `build-scripts` additionally needs a C toolchain because the packaging-script tests compile a throwaway binary and inspect it with `ldd`. Both jobs are back on `ubuntu-latest`.
The preflight steps were deliberately left in place. They cost about a second on the hosted pool and mean that whenever the runner image gains `make` + `gcc`, re-routing is one `runs-on:` line per job and any remaining gap reports itself by name rather than as an opaque mid-build failure.
Note for any future re-route: `lint.yml` also triggers on `pull_request`, and a fork PR runs untrusted contributor code. That must never reach a persistent self-hosted runner, so any re-route has to stay push-only, e.g. `${{ (github.event_name == 'push' && github.repository == 'mudler/LocalAI') && 'arc-runner-set' || 'ubuntu-latest' }}`.
## Touching the cache pipeline
When changing `image_build.yml`, `backend_build.yml`, any of the `backend/Dockerfile.*` files, `Dockerfile.base-grpc-builder`, `.docker/install-base-deps.sh`, `.docker/<backend>-compile.sh`, or `scripts/changed-backends.js`:
@@ -70,3 +70,37 @@ The project documentation is located in `docs/content`. When adding new features
- **Configuration**: If you modify configuration options, update the relevant sections in `docs/content/`.
- **Examples**: providing concrete examples (like YAML configuration blocks) is highly encouraged to help users get started quickly.
- **Shortcodes**: Use `{{% notice note %}}`, `{{% notice tip %}}`, or `{{% notice warning %}}` for callout boxes. Do **not** use `{{% alert %}}` — that shortcode does not exist in this project's Hugo theme and will break the docs build.
## React UI styling
The React UI ships a design system in `core/http/react-ui/src/App.css`: design
tokens, form grids, data tables, stat cards, callouts, plus a small semantic
A release is not finished when the tag is pushed. The GitHub release, the blog post and the demo clips ship together, because the changelog says what moved and the post and the clips are what make anyone care.
## What a release must include
1.**Labels on the merged PRs.** GitHub generates the raw notes from PR labels, so label first, generate second. Wrong labels mean a miscategorised changelog that has to be edited by hand.
2.**`RELEASE_NOTES_vX.Y.Z.md`** at the repository root, in the house style: what changed, why it matters, PR numbers so people can read the diffs.
3.**A blog post under `website/content/blog/`.** One post per release, front matter with `title`, `date`, `author`, `category: "Release"`, `tags`, `summary` and `extracss: ["blog.css"]`. Cover the two or three changes that alter what a user does day to day, not the whole changelog, and link the PR numbers. See `website/content/blog/what-landed-in-localai-4-8.md` for the shape.
4.**Demo clips for the notable features.** Anything visible (a new backend, a UI change, a new endpoint, a measured speedup) gets a short screen recording. Put the file in `website/static/media/`, reference it from the blog post, and reuse it on the marketing pages where it fits.
A release without a post and without clips is incomplete, in the same way a user-facing code change without a docs update is incomplete.
## Clip conventions
- MP4, H.264, no audio track unless the feature is about audio. Keep them short (10 to 30 seconds) and loopable.
- Record the real thing. A clip from the engine's own benchmark suite or a real session, never a mockup.
- Where the change is a speedup, record both sides on the same machine on the same input, so the comparison is honest.
- Name the file after the feature, not the release (`vllm-race.mp4`, not `v4-8-demo.mp4`), so it stays reusable once the release is old.
- The marketing site plays clips with `muted loop playsinline preload="none"` and a `data-lazy` attribute, which the site's IntersectionObserver uses to play and pause them on scroll. Follow that pattern for anything you add.
## Order of work
Label the PRs, generate and edit the release notes, cut the draft release, record the clips while the branch is still fresh in your head, then write the post against the notes and the clips. Publishing the release and merging the post should happen on the same day.
The `creating-localai-releases` skill drives steps 1 to 3 and captures the React UI screenshots that go into the notes.
Beyond the parser names above, `Options[]` carries `--` prefixed engine flags (`--enable-prefix-caching`, `--kv-cache-dtype:fp8_e5m2`). `apply_options_to_engine_args` in `backend/python/common/vllm_utils.py` maps them onto `AsyncEngineArgs` fields, and it must run **before**`AsyncLLMEngine.from_engine_args()` - applying them afterwards is a silent no-op, which is exactly what issue #11130 was.
Things to keep straight when touching this:
- Precedence is typed proto fields → `options:` → `engine_args:`. `applyEngineArgDefaults` in `core/config/hooks_vllm.go` therefore skips seeding a production default whose key the user already set as an option, otherwise the later `engine_args:` pass would silently override them.
- Only `--` prefixed entries are engine flags; `tool_parser:`/`reasoning_parser:` and friends keep their meaning. Parser lookups accept both spellings via `normalize_option_key`.
- Unknown or uncoercible flags warn and are skipped, unlike `engine_args:` which is strict - `Options[]` is a shared bag and knows entries this mapping doesn't.
- Field types come from the annotation's *base* (`Literal["auto","float16"]` is not a float). The helper's tests are stdlib-only: `make test-python-helpers`.
Auto-defaults for known model families live in `core/config/parser_defaults.json` and are applied:
- at gallery import time by `core/gallery/importers/vllm.go`
- at model load time by the `vllm` / `vllm-omni` backend hook in `core/config/hooks_vllm.go`
if echo "$staged" | grep -qE '\.go$'; then go_changed=1; fi
if echo "$staged" | grep -qE '^core/http/react-ui/'; then ui_changed=1; fi
if echo "$staged" | grep -qE '^(core/http/endpoints/openai/(coordinator|respcoord|turncoord|conncoord|compactcoord|ttscoord)/|formal-verification/)'; then rt_changed=1; fi
if [ "$go_changed" -eq 0 ] && [ "$ui_changed" -eq 0 ] && [ "$rt_changed" -eq 0 ]; then
echo "pre-commit: no Go, React UI, or realtime-spec changes staged — skipping."
exit 0
fi
if [ "$go_changed" -eq 1 ]; then
# Resolve the ref golangci-lint's new-from-merge-base should compare
# against. .golangci.yml pins origin/master, which is correct in CI
# (origin == the canonical repo) but wrong from a fork clone, where
# origin/master lags behind and lint would report the whole upstream
# backlog. Prefer upstream/master, then origin/master, then master.
lint_base=""
for ref in upstream/master origin/master master; do
if git rev-parse --verify --quiet "${ref}^{commit}" >/dev/null 2>&1; then
| [.agents/preparing-a-release.md](.agents/preparing-a-release.md) | Cutting a release: PR labels, `RELEASE_NOTES_vX.Y.Z.md`, the blog post under `website/content/blog/`, and the demo clips under `website/static/media/` |
## Quick Reference
- **Git hooks & coverage gates**: Run `make install-hooks` once per clone so the pre-commit lint + coverage gates run. **Never bypass them with `git commit --no-verify`, and never lower a coverage baseline or widen a gate's tolerance to turn a red gate green** — the coverage ratchet only moves up. If a change drops coverage, add tests to raise it (e.g. render-smoke specs). See [.agents/building-and-testing.md](.agents/building-and-testing.md).
- **Coverage gates**: Never lower a coverage baseline or widen a gate's tolerance to turn a red gate green — the coverage ratchet only moves up. If a change drops coverage, add tests to raise it (e.g. render-smoke specs). See [.agents/building-and-testing.md](.agents/building-and-testing.md).
- **Logging**: Use `github.com/mudler/xlog` (same API as slog)
- **Go style**: Prefer `any` over `interface{}`
- **Comments**: Explain *why*, not *what*
- **Docs (docs-with-code rule)**: When you change user-facing behavior (API endpoints, CLI flags, config keys, or features), update the corresponding page under `docs/content/` in the SAME change, not as a follow-up. A user-facing change without a matching docs update is incomplete. See also the documentation conventions in [.agents/coding-style.md](.agents/coding-style.md).
- **New API endpoints**: LocalAI advertises its capability surface in several independent places — swagger `@Tags`, `/api/instructions` registry, auth `RouteFeatureRegistry`, React UI `capabilities.js`, docs. Read [.agents/api-endpoints-and-auth.md](.agents/api-endpoints-and-auth.md) and follow its checklist — missing any surface means clients, admins, and the UI won't know the endpoint exists.
- **Admin endpoints → MCP tool**: every admin endpoint that an admin would manage conversationally (install/list/edit/toggle/upgrade) MUST also be exposed as an MCP tool in `pkg/mcp/localaitools/`. The LocalAI Assistant chat modality and the standalone `local-ai mcp-server` consume that package; drift between REST and MCP is a real risk. Read [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) — the `TestToolHTTPRouteMappingComplete` test fails until you wire the new tool and update the route map.
- **Releases ship with a post and clips**: a release is not done at the tag. It needs labelled PRs, `RELEASE_NOTES_vX.Y.Z.md`, a blog post under `website/content/blog/`, and a short demo clip in `website/static/media/` for each notable feature. See [.agents/preparing-a-release.md](.agents/preparing-a-release.md).
- **Build**: Inspect `Makefile` and `.github/workflows/` — ask the user before running long builds
- **Backend OS coverage**: a new backend must target every OS it can build for, not just Linux. `.github/backend-matrix.yml` has two matrices — `include:` (Linux) and `includeDarwin:` (macOS / Apple Silicon). Most C/C++/GGML and many Python backends build on Darwin too — wire the `includeDarwin` entry + `backend/index.yaml``metal:` entries, or say in the PR why an OS is unsupported. See the darwin checklist in [.agents/adding-backends.md](.agents/adding-backends.md).
- **Gallery variant ranking**: a gallery entry can declare `variants` (alternative builds of the same weights), and LocalAI ranks the ones a host can run by engine preference first, size second. A new backend that should be preferred on some hardware must be listed in `engineNamePreferenceRules` in `pkg/system/capabilities.go`; the sibling `backendBuildTagPreferenceRules` speaks build tags rather than engine names, and using the wrong table matches nothing without erroring. See [.agents/adding-backends.md](.agents/adding-backends.md).
@@ -198,7 +198,6 @@ For AI-assisted development, see [`AGENTS.md`](AGENTS.md) (or the equivalent [`C
- Prefer modern Go idioms — for example, use `any` instead of `interface{}`.
- Use [`golangci-lint`](https://golangci-lint.run) to catch common issues before submitting a PR.
- Run `make install-hooks` once per clone to enable the pre-commit hook: Go changes run `make lint` + the coverage gate (`make test-coverage-check`); `core/http/react-ui/` changes run the Playwright e2e suite (`make test-ui`). Bypass a single commit with `git commit --no-verify`.
- Use [`github.com/mudler/xlog`](https://github.com/mudler/xlog) for logging (same API as `slog`). Do not use `fmt.Println` or the standard `log` package for operational logging.
- Use tab indentation for Go files (as defined in `.editorconfig`).
@@ -268,7 +267,7 @@ make test-e2e
### React UI tests and coverage
The React UI (`core/http/react-ui/`) is covered by Playwright e2e specs, gated by a **monotonic line-coverage ratchet** (`make test-ui-coverage-check`, run in CI and pre-commit). The metric is non-deterministic — a fast local box reads higher than a slow CI runner for the same code — so a small tolerance is unavoidable.
The React UI (`core/http/react-ui/`) is covered by Playwright e2e specs, gated by a **monotonic line-coverage ratchet** (`make test-ui-coverage-check`, run in CI). The metric is non-deterministic — a fast local box reads higher than a slow CI runner for the same code — so a small tolerance is unavoidable.
**If your change lowers UI coverage, raise it back by adding specs — do not widen the tolerance or hand-lower the baseline.** A *render-smoke* spec (navigate to a page, assert its header is visible) cheaply covers an entire lazy page. See `core/http/react-ui/e2e/page-render-smoke.spec.js` and the full policy in [.agents/building-and-testing.md](.agents/building-and-testing.md#react-ui-coverage).
@@ -161,7 +161,7 @@ local-ai run https://gist.githubusercontent.com/.../phi-2.yaml
local-ai run oci://localai/phi-2:latest
```
To test a running LocalAI server from the terminal, open an interactive chat session from another shell. Inside the prompt, `/models` lists installed models and `/model <name>` switches between them.
To work with a running LocalAI server from the terminal, start the built-in agent from another shell. It answers questions, reads your files and runs commands on your machine, asking you to approve anything that changes state. Inside a session, `/models` lists installed models and `/model <name>` switches between them. See the [Terminal agent](https://localai.io/docs/features/terminal-agent/) docs.
```bash
# Terminal 1
@@ -195,7 +195,7 @@ For more details, see the [Getting Started guide](https://localai.io/basics/gett
- **August 2025**: MLX, MLX-VLM, Diffusers, llama.cpp now supported on Apple Silicon
- **July 2025**: All backends migrated outside the main binary — [lightweight, modular architecture](https://github.com/mudler/LocalAI/releases/tag/v3.2.0)
For older news and full release notes, see [GitHub Releases](https://github.com/mudler/LocalAI/releases) and the [News page](https://localai.io/basics/news/).
For older news and full release notes, see [GitHub Releases](https://github.com/mudler/LocalAI/releases) and the [blog](https://localai.io/blog/).
## Features
@@ -231,18 +231,21 @@ Most backends wrap a best-in-class upstream engine. A handful of them are native
| Backend | What it does |
|---------|-------------|
| [vllm.cpp](https://github.com/mudler/vllm.cpp) | From-scratch C++20 port of vLLM for text generation: paged KV cache, continuous batching, prefix caching, safetensors + GGUF loading, engine-enforced structured output, on CPU, CUDA, Metal and Vulkan |
| [parakeet.cpp](https://github.com/mudler/parakeet.cpp) | C++/GGML port of NVIDIA NeMo Parakeet ASR (tdt/ctc/rnnt/hybrid), with cache-aware streaming transcription |
| [moss-transcribe.cpp](https://github.com/localai-org/moss-transcribe.cpp) | C++/GGML port of OpenMOSS MOSS-Transcribe-Diarize: joint long-form transcription, speaker diarization and timestamping in a single pass |
| [moss-tts.cpp](https://github.com/mudler/moss-tts.cpp) | C++/GGML port of the OpenMOSS MOSS-TTS family: text-to-speech (MOSS-TTS-Local v1.5, 48 kHz stereo) with reference-audio voice cloning, through the MOSS-Audio-Tokenizer neural codec |
| [magpie-tts.cpp](https://github.com/mudler/magpie-tts.cpp) | C++/GGML port of NVIDIA's Magpie TTS Multilingual 357M: 22.05 kHz mono text-to-speech in 5 voices and 9+ languages, with the NanoCodec neural codec and tokenizer/G2P embedded in a single GGUF |
| [ced.cpp](https://github.com/localai-org/ced.cpp) | C++/GGML port of the CED audio-tagging models: sound-event classification (527-class AudioSet) over REST and the realtime API for live recognition |
| [voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp) | Speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion), replacing the Python speaker-recognition backend |
| [voxtral-tts.c](https://github.com/mudler/voxtral-tts.c) | Voxtral Realtime 4B speech-to-text in pure C |
| [voxtral-tts.c](https://github.com/mudler/voxtral-tts.c) | Mistral Voxtral-4B-TTS text-to-speech in pure C: 20 preset voices across 9 languages, 24 kHz WAV output, no dependencies beyond libc |
| [vibevoice.cpp](https://github.com/mudler/vibevoice.cpp) | Native port of Microsoft VibeVoice for TTS (voice cloning) and long-form ASR with speaker diarization |
| [face-detect.cpp](https://github.com/mudler/face-detect.cpp) | Face detection, recognition, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace), replacing the Python insightface backend |
| [free-splatter.cpp](https://github.com/localai-org/free-splatter.cpp) | Pose-free 3D reconstruction (FreeSplatter): turns a handful of plain photos into 3D Gaussians, no camera poses or GPU required |
| [trellis2.cpp](https://github.com/localai-org/trellis2cpp) | C++/GGML port of Microsoft TRELLIS.2: single-image to textured 3D mesh (GLB with PBR materials) |
- [Integrations & community projects](https://localai.io/docs/integrations/)
- [Installation video walkthrough](https://www.youtube.com/watch?v=cMVNnlqwfw4)
- [Media & blog posts](https://localai.io/basics/news/#media-blogs-social)
- [Blog: release write-ups, benchmarks and engineering notes](https://localai.io/blog/)
- [Examples](https://github.com/mudler/LocalAI-examples) — including the [realtime voice assistant demo](https://github.com/localai-org/localai-realtime-demo) (Go client for the Realtime API with tool calling)
# Gated on cublas because the variable means nothing to any other build, so a
# stray CUDA_MAJOR_VERSION in the environment must not break `make clean` or
# a CPU build. It does still error for `BUILD_TYPE=cublas make clean`, which
# is the right trade: that invocation is asking about a CUDA build tree.
$(error CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) has no architecture list here (12 and 13do). Leave it empty for a native build, or pass CUDA_ARCHITECTURES explicitly.)
$(warning audio-cpp: libomp not found at '$(LIBOMP_PREFIX)'; building without OpenMP (single-threaded host DSP). Install it with `brew install libomp`, or set LIBOMP_PREFIX.)
CMAKE_ARGS+= -DENGINE_ENABLE_OPENMP=OFF
endif
else
# Portable Linux CPU. Upstream wires this to GGML_BACKEND_DL +
# GGML_CPU_ALL_VARIANTS + $ORIGIN rpath, so one build serves every CPU
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
