Add the official Q4_K_M and Q8_0 GGUF builds with their matching vision projectors so the compact agentic model can be installed through LocalAI.
Assisted-by: Codex:gpt-5 [web]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Cosign v2.4.1 does not select the Sigstore bundle format by default, while LocalAI's verifier only consumes OCI bundle referrers. Request the format explicitly for both registries and guard the producer contract with a shell regression test.
Document strict backend integrity configuration and release-tag identities for operators.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Register JSON API exchanges before their handlers run so the traces dashboard can surface active work. Replace the live entry with the completed persisted record under the same ID, and clean it up if a handler panics.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The runtime section described the small core and on-demand backends but
never stated the simple fact readers look for: one model can run on
llama.cpp while the next loads on vLLM, SGLang or MLX, behind the same
endpoint, and switching is one line in the model's config.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* docs(blog): new DeepSeek and Laguna numbers, visuals, humanizer pass
vllm.cpp master moved 26 commits past what the post was written against,
and two results changed enough to matter. Both came from the same lever:
staging weights device-resident at load instead of reading them from the
GGUF mmap over unified memory, which the GB10 reads about 20% slower per
GEMV than device memory.
- DeepSeek-V4-Flash against DwarfStar: 0.997x parity becomes 1.144x
ahead, 18.69 vs 16.33 tok/s decode, same generated tokens.
- Laguna-XS-2.1 against vLLM: 87% becomes 1.03x, 44.46 vs 43.10 tok/s.
New row in the scoreboard.
Adds three visuals. A chart of throughput against every reference engine,
which is worth having now that the spread is 0.976 to 1.144 rather than a
flat line at parity. The Activity page with four installs running, and the
model detail pane with all four pocket-35b variants. Both screenshots were
recaptured on 2026-08-04 because #11288, #11305, #11307 and #11222 had all
changed those pages since the earlier set.
llama.cpp is deliberately absent from the chart: its 1.18x is a prefill
ratio, and putting it on the same axis as throughput ratios would be
comparing two different measurements.
Also carries the media the release notes embed, since a GitHub release
body needs URLs that survive publishing and drag-and-drop has no CLI.
Supersedes #11364.
Humanizer pass on the prose. The post had collected five exactness idioms
in one section (token-for-token, byte-exact twice, byte-identical,
token-identical). One is precision, five is a tic, so the 27B row keeps
its "token-for-token identical" where identical output is the actual
claim and the rest say what they mean. That also fixed a hyphen in
predicate position ("is token-identical").
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
* docs(blog): redraw the benchmark chart as a branded card
The Flint bar chart was generic: default palette, no brand, and drawn
from zero, which made five ratios between 0.976 and 1.144 look like five
bars of roughly equal length.
Redrawn in the style of recorder-for-agents' render-card.sh cards, the
same shape as the vllm.cpp README GIF. Palette taken from the two logos
rather than invented (LocalAI navy #0E2632 and teal #469AAF, vllm.cpp
teal #3AB4CA), SVG generated by a small JS loop so the geometry is exact
at any scale, headless Chrome to PNG at 2x.
The substantive change is that bars now run from the 1.00 parity line
instead of from zero. Deviation is what the data is about, so DeepSeek's
+14.4% and MLX-LM's -2.4% are both legible, and the one row that is
behind is the one row in amber. Each bar carries its ratio and the raw
measurement under it.
Keeps the .html source next to the .png so the chart is editable later:
change a number, re-run render-card.sh.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This is the text users read in the backends list and the gallery, and it
was the last place still describing vllm.cpp as "a from-scratch C++20
port of vLLM created and maintained by the LocalAI team" with no
indication of maturity.
Three corrections, matching the v4.8 release notes and blog post:
- It leads with ALPHA. These are alpha development builds and llama-cpp
stays the recommendation for production, which is the single most
useful thing to know before clicking install.
- It is maintained by the LocalAI team but developed in its own
repository and usable without LocalAI. vLLM is named for what it
actually is, the reference implementation that output is checked
against and benchmarked against, rather than just the thing that was
ported.
- It records the featureset that has grown past vLLM: GGUF loading,
speculative decoding and KV offload, alongside the architecture and
hardware coverage that were already listed.
Also notes that the project is expected to be renamed, with the new name
still to be decided, so anyone who installs it now is not surprised
later.
vllm-cpp-development inherits all of this through the YAML anchor, so
both entries are covered by the one edit. Verified the file still parses
and that both entries carry the new text.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* docs(blog): final figures for the 4.8 post, and the MLX provider
The cycle closed at 374 PRs over twenty-one days, not the 321 over
eighteen the post was written against. Corrects the summary, the opening
line, the contributor count and the gallery total, and moves the date to
the day the release is cut.
Adds the MLX GEMM provider (#11137), which merged after the post was
written and is the one number an Apple Silicon reader wants: 1.54x to
2.19x on an M4 with time to first token roughly halving, both arms
toggled on one binary. The +/-10% caveat travels with the table rather
than being left in the PR.
Two lines edited against the no-ai-slop skill while I was in the file,
the same pass #11324 ran over the engines post:
- The opener balanced two clauses across a colon and closed on "without
lying to you", which is the built-to-be-quoted shape readers picked
out of the HN thread. It is a flat statement now.
- "This is a new modality rather than a new backend under an existing
one" is a binary contrast that says nothing the next clause does not.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
* docs(blog): call vllm.cpp alpha, and finish the no-ai-slop pass
vllm.cpp is not a released backend and the post read like it was. The
old wording buried the caveat in a block quote at the end of the section
and still said "first release of a young engine". It now says plainly,
before the caveat can be skipped, that these are alpha development
builds, that shipping them in 4.8 is about letting people try the thing
rather than recommending it, and that llama-cpp stays the default.
Also completes the no-ai-slop pass I had only half run. Counting the
lines built to be quoted, headings and section endings included, the post
is in reasonable shape: long flat informational stretches, tables
followed by a plain finding, headings that are labels rather than
epigram-verdicts. Three patterns survived, each one an item in eval.md:
- "and inverts that:" set the usual shape against ours across a colon.
The sentence works without the frame.
- "Two things were conflated there: a signal, which needs one line, and
the detail, which needs somewhere to put it" is a role-assignment pair.
Says what happens instead.
- "The maturity statement from the release notes is worth repeating in
full" is throat-clearing in front of a quote, and the quote is gone.
Left the rest alone. Minimum effective edit, not a rewrite.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
* docs(blog): present vllm.cpp as a community project, with its own numbers
The post described vllm.cpp as "a from-scratch port of vLLM, written and
maintained by the LocalAI team". Two things wrong with that. It is a
community project, and it has stopped being only a port: it loads GGUF,
runs on CPU, Metal and Vulkan, ships speculative decoding and KV offload,
and its benchmark page measures against llama.cpp, MLX-LM and DwarfStar
as well as vLLM, because those are the engines it competes with on that
hardware.
vLLM's role is now stated for what it is, the reference implementation.
Correctness is checked against it and the scoreboard is kept against it.
Also flags that the name will probably change, since it is drifting far
enough that vllm.cpp will eventually mislead.
Adds real numbers from the project's own docs/BENCHMARKS.md rather than
adjectives: 1.045x vLLM at concurrency 1 on Qwen3.6-27B NVFP4 with
token-for-token identical output, 1.010x and 1.013x at c16 and c32 on the
35B MoE and behind below that, prefill 1.18x over llama.cpp on CPU
aarch64, 97.6% of MLX-LM warm total on an M4. Upstream's own caution
travels with them: it treats c2 through c32 as ties because its noise
band is 0.5% and those margins are 0.7% to 1.7%.
Every figure was checked against ~/_git/vllm.cpp/docs/BENCHMARKS.md
rather than restated from memory. The heading is marked alpha to match
the section body.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
* docs(blog): say who maintains vllm.cpp, and add the DeepSeek Flash result
Two corrections to the previous commit.
"A community project" says nothing and was not quite true either. The
LocalAI team maintains vllm.cpp. Community-first is the intent, not a
description, so it now says that and says what backs it: its own
repository, its own docs, benchmark record and issue tracker, and it runs
without LocalAI anywhere in the picture.
Adds the DeepSeek-V4-Flash result, which makes the divergence point
better than any of the prose around it. That model does not run on vLLM
on a single GB10: every vLLM-loadable checkpoint is 156 GB or more
against a 119 GiB unified pool, and the only quant that fits is an
extreme-low-bit GGUF that vLLM cannot load. vllm.cpp reads GGUF and runs
it at 16.28 tok/s against ds4's 16.33, a parity result. Also notes MTP
speculative decoding, token-identical to vLLM's and about 4% faster at
concurrency 1.
Both figures checked against ~/_git/vllm.cpp/docs/BENCHMARKS.md.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
* docs(blog): lead the DeepSeek result with what we run, not with what vLLM cannot
The previous version opened on "that model does not run on vLLM on a
single GB10 at all". Wrong emphasis twice over: it makes a strong
negative claim about another project the headline, and it buries the
actual result, which is that vllm.cpp runs DeepSeek-V4-Flash at roughly
2-bit (IQ2_XXS mixed, about 80 GB) on a single DGX Spark and decodes at
16.28 tok/s against DwarfStar's 16.33.
The size constraint is still there, stated as the reason the quant is
what it is rather than as a point about vLLM: at 300B+ total parameters
even a 4-bit checkpoint is 156 GB or more, so a 2-bit GGUF is what fits
the Spark's 119 GiB unified pool.
The table row now names the quant and the box (IQ2_XXS, one DGX Spark)
instead of just "GGUF, GB10", since that is the part a reader with a
Spark wants.
Figures unchanged and still from ~/_git/vllm.cpp/docs/BENCHMARKS.md.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
* docs(blog): say the new name is undecided
"The name will probably change at some point" invited the obvious
question. It now says the rename is expected and the name is still to be
decided, which is the actual state.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Replace copied HauhauCS base-model text with metadata for the actual Genesis Hermes V6 artifact and link its upstream base model.
Assisted-by: Codex:gpt-5 [Hugging Face]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
No -gpu-rocm-hipblas-llama-cpp image has been published since 2026-08-01.
Every build since has been killed by GitHub at exactly its 6h job limit:
job 91830652349 cancelled 6h00m (2026-08-04)
job 91763226161 cancelled 6h00m (2026-08-03)
job 91466626154 cancelled 6h00m (2026-08-02 full matrix)
The registry shows the damage: master-gpu-rocm-hipblas-llama-cpp last
built 2026-08-01 05:53, latest-gpu-rocm-hipblas-llama-cpp 2026-07-15,
against master-cpu-llama-cpp which is current.
Same cause as #11321, different mechanism. Since #11255 every x86 GPU
image also builds ggml's CPU_ALL_VARIANTS matrix. SYCL died because icpx
stalls on one translation unit; ROCm dies on volume. hipcc compiles the
HIP kernels once per entry in AMDGPU_TARGETS, and that list is eleven
architectures (gfx908, gfx90a, gfx942, gfx950, gfx1030, gfx1100, gfx1101,
gfx1102, gfx1151, gfx1200, gfx1201). The CPU matrix lands on top of that.
The numbers are unambiguous. The same job took 2h27m in the 2026-07-26
full matrix, before #11255. #11255 merged 2026-08-01 07:26, an hour and a
half after the last image was published, and it has been 6h00m ever since.
The tail of the last run shows it 61% through ggml-hip at the 83 minute
mark, still building HIP template instances.
Route hipblas to the portable fallback, exactly as #11321 did for SYCL and
for the same practical reason: it is what these images shipped before
#11255, and run.sh already prefers *-cpu-all when present and falls back
otherwise. Expected to restore the 2h27m build with room to spare.
Not fixed here: the CPU variant matrix is genuinely wanted on ROCm for
partial offload. Getting it needs the build to fit in 6h, which means
trimming AMDGPU_TARGETS or splitting the job per architecture. Both are
larger changes than unbreaking the image, and neither should ride along
with a build that is currently not shipping at all.
Verified: make test-build-scripts passes, including the extended
llama-cpp-build-target_test.sh. bonsai is unaffected (own compile script,
ROCm builds in 1h52m) and turboquant has no hipblas variant.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): enable and vendor the MLX GEMM provider on darwin/metal
The darwin vllm-cpp image built the Metal backend with vllm.cpp's native MSL
GEMM only. vllm.cpp also ships an optional MLX provider for the dense GEMM,
kept OFF upstream because it costs a ~19 MB libmlx.dylib plus a ~105 MB
mlx.metallib, on the stated position that it must earn that cost by
measurement.
Measured on an Apple M4 (16 GiB, macOS 26.5.2) it does. One binary, arms
toggled with VT_OP_PROVIDER_DISABLE=mlx so there is no build-difference
confound, Qwen3-1.7B-bf16 p=512 g=128, 2 reps, arm order alternated per rep:
B=1 5.79 vs 3.08 agg tok/s (1.88x) TTFT 3.32 s vs 7.68 s
B=8 25.70 vs 13.69 (1.88x) TTFT 13.95 s vs 34.38 s
B=16 38.65 vs 17.69 (2.19x) TTFT 18.33 s vs 54.48 s
Peak RSS is unchanged (6.65 to 7.50 GB in both arms) and the output is
bit-identical: vllm.cpp's three-way parity test measures mlx-vs-msl NMSE of 0
on all six shapes, and mlx-vs-cpu equal to msl-vs-cpu, against a 5e-4 bar. MLX
serves the dense GEMM alone; paged attention stays vllm.cpp's own kernel
because MLX has no paged-KV primitive. Full disposition, including the
INDICATIVE status and the isolation actually achieved, is in vllm.cpp
docs/BENCHMARKS.md "MLX GEMM provider A/B on Apple M4".
Build: MLX comes from the pinned prebuilt pip wheel (MLX_VERSION, default
0.29.3) into a venv under the backend dir. Building MLX from source needs
`xcrun metal`, i.e. a full Xcode the macOS runners do not have, while the wheel
ships include/, lib/libmlx.dylib and the compiled metallib ready to link. The
install is a stamp FILE rather than a phony target, because a phony
prerequisite is always newer than libvllm and would re-link it every
invocation. VLLM_CPP_MLX=off restores the previous Metal build.
Packaging vendors libmlx.dylib, mlx.metallib and MLX's MIT license into
package/lib/. Three things this had to get right, each verified on the M4
before it was written rather than after:
1. libvllm.dylib links @rpath/libmlx.dylib and its build-time LC_RPATH points
inside the build venv, a path no user has. Every build rpath is deleted
and replaced with @loader_path/lib.
2. MLX loads its metallib from beside its OWN dylib, so both files must land
in the same directory or every Metal op fails with "Failed to load the
default metallib".
3. install_name_tool invalidates the code signature and macOS refuses to load
an arm64 image with a stale one, so the patched library is re-signed
ad-hoc.
Verified end to end on the M4 by building through this Makefile and running the
packaged artifact: `DYLD_PRINT_LIBRARIES` resolves libmlx from package/lib/,
`codesign -v` passes, no build-venv path survives in the load commands, and a
real generation runs with the provider selected (op=65 selected=mlx) and zero
metallib failures. A missing rpath now fails the build instead of the user's
first inference.
Cost: the darwin vllm-cpp image grows by about 124 MB.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
* fix(vllm-cpp): default the MLX GEMM provider OFF on darwin
This branch opened with VLLM_CPP_MLX=on, justified by an A/B that measured the
MLX provider at 1.88x to 2.19x against the native MSL GEMM. That measurement was
correct when taken and is now stale: vllm.cpp's own Metal kernels have improved
several-fold since, through mma prefill attention, a vectorised decode V
accumulation, vectorised attention staging, a fused qk-norm-RoPE preamble and a
simdgroup-per-row softmax. The native path MLX was compared against no longer
exists.
Re-measured on the same Apple M4, in the same binary, with the arms toggled by
VT_OP_PROVIDER_DISABLE=mlx, on Qwen3-1.7B-bf16 warm at p=512 g=128:
MLX provider ON prefill TTFT 1370 ms warm throughput 11.98 tok/s
MLX provider OFF prefill TTFT 1400 ms warm throughput 22.06 tok/s
Shipping the previous default would have halved Apple Silicon throughput.
MLX's steel GEMM is still about 20% faster than ours in isolation, but the
provider pays a per-op mx::eval synchronisation plus an output memcpy, because it
cannot write into our buffer. Across prefill's roughly 112 GEMMs that overhead
leaves a 2% gain; on decode, where the same synchronisation is paid once per
matmul per token, it costs 46%. The option is kept for prefill-dominated
workloads, where the margin is small but real.
The README section is rewritten rather than patched: it previously presented the
stale table as the reason for the default, so leaving it in place would have made
the new default look arbitrary.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): bump vllm.cpp and default MLX ON, gated to prefill
Bumps VLLM_CPP_VERSION from 9e1c9025 to eec09bed and turns VLLM_CPP_MLX back on.
These two must move together, which is why they are one commit.
Upstream now shape-gates the MLX provider to prefill: it declines m < 2, which is
exactly the decode GEMV. MLX's steel GEMM wins prefill, 524.5 ms of TTFT against
602 for the native path, but loses decode badly because the provider pays an
mx::eval synchronisation and an output memcpy on every call while decode makes
about 112 calls per token. Ungated it does both; gated it does only the good half.
Measured on an Apple M4 with Qwen3-1.7B-bf16 warm at p=512 g=128:
MLX gated to prefill (pin >= 89c46aeb) TTFT 524.5 ms 24.40 tok/s, 99.1% of MLX-LM
MLX ungated (older pins) TTFT 537 ms 12.7 tok/s
MLX off TTFT 602 ms 23.9 tok/s
This branch briefly defaulted the provider off, which was the correct call for an
ungated provider at the old pin. The gate is what makes on correct again, so the
pin and the flag are coupled: rolling VLLM_CPP_VERSION back before 89c46aeb while
leaving MLX on would select the middle row and roughly halve throughput. Both the
Makefile comment and the README state that dependency explicitly.
The bump also brings six Metal kernels landed upstream since the old pin — mma
prefill attention, a vectorised decode V accumulation, vectorised attention
staging, a fused qk-norm-RoPE preamble, a simdgroup-per-row softmax and a
simdgroup-per-head preamble — which take the non-MLX Metal path from 89.4% to
96.4% of MLX-LM on their own.
One caveat, recorded in the README: MLX's GEMM is not bit-identical to the native
kernel, so an MLX build produces a different greedy sequence than a non-MLX build.
That is a property of the provider rather than of the gate and predates this
packaging.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(vllm-cpp): correct the MLX-gated figure to 97.6%, from 99.1%
The previous commit quoted 99.1% of MLX-LM for the prefill-gated MLX build. That
figure divided by a two-run MLX-LM baseline, 27.135 and 27.744 generation tok/s
averaged to 27.44. Re-measured interleaved with ours over four ABBA blocks,
MLX-LM's decode is 27.848 with a 0.34% spread across six runs, so the 27.135 was
an outlier and averaging it in overstated us by roughly 1.5 points.
Corrected: the gated configuration is 24.37 tok/s, or 97.6% of MLX-LM, and the
MLX-off build is 23.9 tok/s or 95.9%. Prefill TTFT is unchanged at 524.5 ms
against MLX-LM's 532.6, so we remain about 1.5% faster there.
Nothing else changes. MLX still wins prefill and loses decode, the shape gate is
still the right disposition, and the pin and the flag are still coupled. The gate
is worth about 1.7 points over the MLX-off build rather than 2.7.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): pin MLX gate from mainline
The previous pin was a merge commit from the experimental C ABI v9 branch. Pin the same MLX prefill gate on upstream main so the backend build does not pull unrelated ABI v9 work into every platform variant.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): restore backend build portability
Keep the current master pin when enabling MLX so every backend variant builds against the known-good vllm.cpp revision. Suppress Apple clang’s GNU constant-folding diagnostic for Objective-C++ Metal compilation only, since upstream treats warnings as errors.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): demote MLX header VLA warning
MLX 0.29.3 headers trigger Apple clang's gnu-folding-constant diagnostic in the Objective-C++ provider. Keep the diagnostic visible while exempting only it from vllm.cpp's global warnings-as-errors policy.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): suppress MLX header VLA warning
Target-level Objective-C++ -Werror is appended after the directory flags, so a no-error demotion is re-promoted. Disable this single warning for the MLX header while keeping every other warning fatal.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): pin source-scoped MLX warning fix
Move the AppleClang warning exception into vllm.cpp where its target warning policy is defined, and pin LocalAI to that source-scoped fix.
Assisted-by: Codex:gpt-5
* fix(vllm-cpp): pin effective MLX warning suppression
The source-scoped no-error flag was overridden by the target warning policy. Pin the companion vllm.cpp change that disables only the MLX header diagnostic for its Objective-C++ translation unit.
Assisted-by: Codex:gpt-5
* fix(vllm-cpp): pin diagnostic pragma fix
Pin the companion vllm.cpp correction that scopes the AppleClang folding warning suppression inside the MLX translation unit, after command-line warning policy.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): pin remaining Darwin build fixes
Advance the MLX-enabled backend to the vllm.cpp revision already validated by the dependency update branch. This includes the feature guards and AppleClang pragma boundary needed by the Darwin build.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): pin MLX system dependency boundary
Pin the companion vllm.cpp change that models MLX as an imported system dependency, keeping third-party header diagnostics out of the project's warnings-as-errors policy while retaining fatal warnings for project sources.
Assisted-by: Codex:gpt-5 [Codex]
* fix(vllm-cpp): pin scoped MLX warning guard
Advance vllm.cpp to the companion fix that keeps MLX headers on a SYSTEM dependency and scopes AppleClang folding-constant suppression to the external includes.
Assisted-by: Codex:gpt-5 [systematic-debugging] [test-driven-development]
* fix(vllm-cpp): use available MLX wheel
MLX 0.29.3 is no longer available to the Darwin runner, so the backend build stopped before CMake. Pin the first available compatible wheel and keep the documented default in sync.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update antirez/ds4
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(ds4): link upstream CUDA MMQ objects
The updated ds4 CUDA object now calls into the vendored MMQ implementation. Build and link those objects into both the gRPC server and distributed worker.
Assisted-by: Codex:gpt-5 [Codex]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Wait for observable loop events instead of budgeting hundreds of milliseconds for scheduler timing. Keep a short bounded overlap observation for the two-leader exclusion check.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The blog has a deep post for 4.8 and a history post that covers the earlier
releases at summary altitude, but nothing in between. These five fill that
gap in the same shape as what-landed-in-localai-4-8: what the release was
for, runnable examples, and the limits that apply.
Every endpoint, CLI flag, env var and gallery entry is verified against the
matching release tag rather than taken from the release notes. That caught
two paths the published 3.10.0 notes got wrong: tracing is /api/traces, not
/api/v1/trace, and a stored response is fetched from /v1/responses/:id, not
/api/v1/responses/{response_id}.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* blog: rewrite the engines post without the AI tells
The HN thread on this post (item 49125065) spent most of its comments on the
writing rather than the engines. Readers quoted specific lines back as tells.
This is the same post with the same numbers, edited against the updated
no-ai-slop skill.
Every figure, table and link is unchanged, except that "27% of the memory"
is now the underlying 363 MB against 1328 MB from the table.
Two substantive framing fixes, both from the reply draft in
hn-reply-engines-post.md:
- vllm.cpp is no longer implied to be a speed win. The table is a tie, the
result is the install size, and the post now says so before a reader has to
work it out and post about it.
- Added one line on the language mix. Readers took the C++/Python/Go tree as
incoherence rather than as a Go core with per-ecosystem backends.
Cut throughout: the ledger metaphor ("what those ports buy", "not paid for in
throughput"), unearned framing ("the honest reading is", "has nothing to do
with"), the shape summary ("that is the general shape of these wins"),
confident deference ("people who are better at those models than we are"),
self-grading numbers ("a good result for a 66 MiB binary"), verbless
comparisons, three of the four exactness idioms, and the aphoristic headings
and verdicts. The double-tricolon summary is one plain clause now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* blog, website: same anti-slop sweep over the rest of the site
One-by-one pass over the other four posts and the site templates, with the
same rules used on the engines post. All figures, tables, links and PR
numbers are unchanged everywhere; the edits are to prose only.
apex-moe-quantization: ledger metaphors were the main issue, eight uses of
buy/cost/pay/spend for things that are not money. Also "the honest reading
is", "that is the comparison that matters", and two section-ending aphorisms
("Size is a speed knob as much as a memory knob", "Q6_K is the ceiling worth
paying for").
localai-since-march-2023: light touch, this one already reads like a person.
Removed "the curve is not the point", a "not the feature list, but the four
decisions" contrast, and two "X is what made / is the piece that" forms.
parakeet-cpp-asr-on-cpu: six exactness idioms across one post, "byte for
byte" twice, "character for character" twice, "byte-identical" twice and
"bit-identical" once, including in the title. Down to one, kept where the
precision is load-bearing. Also the "what end-of-utterance detection buys
you" heading and the "we say so rather than averaging it away" flex.
what-landed-in-localai-4-8: no changes. It is dense, flat and ends every
section on a PR number or a plain fact, which is the shape the other posts
should look like.
Site templates: "Most backends wrap somebody else's engine. These do not."
was the same contrast the engines post opened with. Also "Not a degraded mode
that technically runs", "A port only ships once it matches the original",
"Speed is the part we then go and win ... not a marketing run", and the last
"byte for byte" on the landing page.
Hugo builds clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* website: it is eighteen engines, not nineteen
Three places said nineteen: the /engines/ page description, the JUL 2026
timeline entry on the landing page, and the header comment in
data/engines.yaml.
Eighteen is right, confirmed two ways. The "Backends built by us" table in
the README has exactly 18 rows, and data/engines.yaml has 19 entries of which
one is apex-quant, which is a quantization recipe rather than an engine. The
two lists otherwise match name for name.
The yaml comment is the likely origin: it read "the nineteen native engines
the LocalAI team wrote, and the one quantization recipe that feeds them",
which counts apex-quant twice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
master does not compile:
vet: core/http/endpoints/mcp/localai_assistant_test.go:157:19:
method stubClient.ListScheduling already declared at
core/http/endpoints/mcp/localai_assistant_test.go:87:19
Two fixes for the same breakage landed. The four Scheduling methods were
already present at lines 87-99, in interface order after ListNodes, by
the time #11318 merged; #11318 appended its own copy after
GetRouterDecisions. The two blocks sit in different parts of the file, so
git merged both without a conflict and nothing flagged it.
Remove the appended copy and keep the one in interface order. Pure
deletion, no behaviour change.
Verified: go vet clean on ./core/http/endpoints/mcp/, and
go test ./core/http/endpoints/mcp/ passes.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The News page kept a hand-maintained "Highlights" list that had drifted:
it was missing all of 2025, duplicated the README's own news list, and
linked /features/middleware/ for a page that lives at operations/.
Both of its jobs already have owners. website/content/blog/ carries the
release write-ups and engineering notes, and GitHub Releases carries the
full changelog. Replace the list with a pointer at those two, so there is
one place to update instead of three.
The page keeps its url and front matter, so /docs/basics/news/ and the
root /basics/news/ redirect that .github/ci/gen-redirects.sh generates
both keep resolving.
Also drop the two contributor instructions in .agents that told authors
to add a whats-new.md bullet per feature: announcing a capability is the
release blog post's job, per .agents/preparing-a-release.md.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Write] [Bash]
Deploy site to GitHub Pages failed on five of the last eight master
pushes, always in the build job before Hugo runs:
Setup go version spec 1.22
...
go: downloading go1.26.0 (linux/amd64)
go: download go1.26.0: golang.org/toolchain@v0.0.1-go1.26.0.linux-amd64:
Get "https://proxy.golang.org/...": connect: network is unreachable
##[error]Command failed: go env GOPATH
The workflow pinned setup-go to 1.22 while go.mod declares go 1.26.0, so
the `go run ./.github/ci/modelslist.go` step that generates the gallery
page had to fetch the real toolchain from proxy.golang.org first. That
fetch is not reliably reachable from the runner, which is why the deploy
alternated between passing and failing rather than failing outright.
Track go.mod instead of a literal. The version the module needs is then
installed directly and there is no toolchain download to fail.
This matters beyond CI noise: the docs and the site, including the
release blog post, ship through this workflow.
Scoped deliberately to gh-pages, the workflow with the observed failure.
test-extra.yml pins 1.25.4 in a dozen places and is below go.mod for the
same reason, so those jobs also download a toolchain, but they are
currently green and rewriting twelve pins on a hunch risks more than it
fixes. Worth a follow-up.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
#11228 added ListScheduling, GetScheduling, SetScheduling and
DeleteScheduling to localaitools.LocalAIClient but did not update
stubClient, the hand-written test double in the mcp endpoints package.
The package therefore fails to typecheck, which takes out both lint and
tests on master:
cannot use stubClient{} as localaitools.LocalAIClient value in
argument to h.Initialize: stubClient does not implement
localaitools.LocalAIClient (missing method DeleteScheduling)
Red on 8f74f74b, fd4ec083 and 8a68f357; green on cd62e8ff, the commit
before.
Add the four methods with the same inert bodies the rest of the stub
uses. The real implementations are covered in the localaitools suites;
this double only exists so the holder can be constructed.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
chrome-audit.spec.js walks 25 routes in a single test, and has been the
UI E2E suite's failure on 5 of the last 6 master runs. It always dies the
same way, at the 30s per-test default:
Test timeout of 30000ms exceeded.
Error: page.waitForTimeout: Test timeout of 30000ms exceeded.
19 | await page.goto(route)
> 20 | await page.waitForTimeout(400)
The spec is new in 5cb0c1a8; the commit before it was green, and every
run since has been red on this file.
The failure is cumulative rather than one bad route. Across those runs
the clock runs out at line 19, 20 or 21 depending on where the loop
happens to be, and the timeout lands on waitForTimeout rather than on
goto, which is what running out of budget looks like as opposed to a
navigation that hangs. 30s over 25 routes is ~1.2s each, including a
deliberate 400ms settle, so there is very little headroom to begin with.
Give the test a budget proportional to its work: six seconds a route.
That absorbs a slow runner and still fails promptly if a route genuinely
hangs.
Verified: the spec passes on the current UI in 12.2s solo, and the full
suite passes 418 at 8 workers locally. What I could NOT do is reproduce
the CI timeout on this machine, which has 20 cores against the runner's
2 to 4; under synthetic CPU load it still finished in 13.5s. So the fix
is argued from the CI signature and the arithmetic, not from a local
repro, and the proof is this spec going green on the hosted runner.
Note test.setTimeout() has to be called inside the test body. At module
scope Playwright rejects it with "test.setTimeout() can only be called
from a test".
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Every CUDA sglang image failed in the 2026-08-02 full-matrix rebuild:
-gpu-nvidia-cuda-12-sglang, -gpu-nvidia-cuda-13-sglang and
-nvidia-l4t-cuda-13-arm64-sglang, all with the same build error.
Building cuda-tile==1.6.0rc3
x Failed to build `cuda-tile==1.6.0rc3`
ModuleNotFoundError: No module named 'wheel_stub'
hint: `cuda-tile` (v1.6.0rc3) was included because `sglang` (v0.5.16)
depends on `flashinfer-python` (v0.6.14) which depends on `cuda-tile`
This is the failure mode requirements-cublas1{2,3}-after.txt already
carries an nvidia-modelopt bound for, arriving through a different
package. install.sh passes a global --prerelease=allow, which is
load-bearing for flash-attn-4, so an unbounded dependency resolves to a
prerelease; cuda-tile 1.6.0rc3's build backend imports wheel_stub without
declaring it in build-system.requires; --no-build-isolation means nothing
provides it, and the build dies.
Nothing in this repo changed. cuda-tile published 1.6.0rc1 and rc3 and
the weekly cron picked them up, which is the drift that job exists to
catch.
Bound the one package rather than dropping the global flag, matching the
existing precedent. 1.5.0 is the newest stable release, so <1.6 takes the
last good one. l4t13 gets the same bound: it installs plain sglang rather
than sglang[all], but flashinfer-python is a dependency of both.
NOT VERIFIED LOCALLY: reproducing this needs a CUDA docker build, which
this machine cannot run. The diagnosis is from the CI log and the
resolver's own hint, and the change follows a fix already proven in these
same files. CI on this PR is the check that matters.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Since #11255 and #11276 every GPU image also builds ggml's CPU_ALL_VARIANTS
matrix, so a partial offload uses the host's SIMD kernels. That works
everywhere except SYCL, where the Makefile compiles the whole tree with
icpx -fsycl: icpx never finishes ggml-cpu/arch/x86/repack.cpp at
-march=sapphirerapids. In run 30765516644 both sycl_f16 and sycl_f32 stopped
at that translation unit and sat there for 5h30m with a single compile in
flight until GitHub killed the job at its 6h limit, and turboquant's f16 job
lost its runner outright. gcc compiles the same file in seconds in the vulkan
and CPU jobs of the same run, so the CPU variant matrix is only unbuildable
under icpx.
Route SYCL back to the portable fallback binary, which is what these images
shipped before #11255. run.sh already prefers *-cpu-all when present and falls
back otherwise, so nothing else has to change.
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(api): add text moderation endpoint
Add an OpenAI-compatible /v1/moderations endpoint backed by constrained local text generation. Register its auth and discovery surfaces, document the text-only MVP, and cover response shaping and access control.
Assisted-by: Codex:gpt-5
* test(mcp): update assistant client stub
Keep the LocalAI Assistant holder test stub aligned with the scheduling methods added to LocalAIClient so repository-wide type checking succeeds.\n\nAssisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(api): add POST /v1/images/upscale endpoint
Add a new image upscaling endpoint that accepts a source image and
returns an upscaled version. Supports selectable upscaler models
(e.g. realesrgan) and a configurable scale factor (2x or 4x).
- backend.proto: add UpscaleImage RPC and UpscaleImageRequest message
- pkg/grpc: implement UpscaleImage in Backend interface, client, server
and embed shim
- core/backend/upscale.go: new backend helper (mirrors ImageGeneration)
- core/http/endpoints/openai/upscale.go: new multipart/form-data handler
- core/http/routes/openai.go: register POST /v1/images/upscale
- core/http/auth/features.go: gate upscale routes under FeatureImages
- backend/python/diffusers/backend.py: implement UpscaleImage — uses
diffusers upscale pipeline when loaded, falls back to Lanczos resize
* fix(grpc): add UpscaleImage stub to Base backend
All Go backends embedding Base now satisfy the AIModel interface
without needing to implement UpscaleImage explicitly.
* fix(images): complete upscale endpoint integration
Store generated upscales under the served images directory, validate scale factors, document and advertise the endpoint, and add a functional Stable Diffusion x4 gallery model.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Give gallery operations distinct pause and cancel paths. Pause preserves partial download data so reinstalling the same model or backend resumes through HTTP Range, while cancel keeps its destructive semantics. Surface the action in the Activity UI and document the API behavior.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add multilingual 1B and 8B Q4_K_M GGUF embedding entries and link them as variants for automatic memory-aware selection.
Assisted-by: Codex:gpt-5 [web]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
fix(gallery): repair DeepSeek V4 0731 entry
Use the official single-file ggml-org MXFP4 artifact with its verified SHA256 and route it through llama.cpp instead of treating an unsloth repository page as a ds4 model file.
Assisted-by: Codex:gpt-5 [Hugging Face API]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add Q4_K_M and Q8_0 GGUF builds for the trending Instella-MoE-16B-A3B-Think model, with host-selectable variant metadata and verified Hugging Face LFS hashes.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(ui): give the Operate overview real numbers and traces a latency shape
First two items from a component-by-component comparison against the mocks.
The pattern that audit found: everything newly built matched, everything
pre-existing got the palette but not the layout, and an "absent rather than
empty" rule hid most of the overview exactly when someone was looking at an
idle installation.
**The headline grid is always rendered**, including at zero, with a fourth cell
for host memory. Hiding it removed the page's structure precisely when it was
most likely to be read, and "0 failed" is information — an absent panel is not.
The quiet case is now said in a line underneath instead of by showing nothing.
**The sections state counts** rather than listing their destinations: backends,
models, updates and running operations instead of the words "Usage and traces".
That needed installed backend and model counts in the summary context, which
are two more cheap reads on the poll that was already running.
**Traces rows carry latency as a bar as well as a figure**, scaled against the
slowest request currently in view and turning amber past two seconds. The table
had no latency column at all — the number was buried in the expanded detail, so
the shape of the tail was invisible while scanning. Scaling against the view
rather than an absolute ceiling is deliberate: what matters when reading a page
of traces is which of these are the outliers, and an absolute scale flattens
every row on a fast installation into nothing.
Full e2e suite: 409 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): name the engine on Home's resident models, and add jump-back-in
Third item from the mock comparison.
The mock showed each resident model with the engine serving it. /system carried
only the id, so the audit recorded this as blocked on a server field — but the
config loader is already in scope where that response is built, so it is one
lookup. SysInfoModel gains an optional `backend`, resolved from the model's
config and omitted rather than guessed when there is none (a loose file, or a
config since removed). Home renders the column blank in that case; the test
pins both halves of that.
Memory per model stays out. It is not one lookup — it would mean asking each
backend process — and inventing a number beside a real one is worse than
leaving the column off.
"Jump back in" is the block the mock had and Home did not. The quick-links row
above it is a set of first-run actions; these are the three places someone
returns to, each stated with what it currently holds rather than as a bare
label.
Go: routes suite passes. Full e2e suite: 412 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): rank the recommended models as lanes instead of equal cards
The hardware recommendations were a grid of equally-weighted cards. The list is
already sorted by fit, and a grid throws that order away: three cards side by
side say "pick one", when the page has actually formed an opinion about which
one.
They are lanes now, read top to bottom in fit order, with the leader carrying
the single amber "Best fit" label and the rest marked "Also fits". One opinion
per page — the alternatives are alternatives, not runners-up each worth their
own colour, which is how a strip of coloured badges ends up meaning nothing.
Below 720px the size and VRAM columns drop and the lane keeps the name and the
install action, which are the two things a narrow screen needs.
The existing panel spec moves off .rec-models-item onto .lane rather than being
deleted; dismissal, collapse, keyboard operation and install all still pass
unchanged, and there is a new assertion that exactly one row is called out.
Full e2e suite: 413 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): drop capsule chips app-wide, and un-break the empty voice library
**Pills are gone.** A capsule radius reads as a tag floating on the surface,
which fights a system whose structure is hairlines and square corners — and
with chips on Discover, Host, Activity and the biometrics pages, "some pages
have pills" was the real inconsistency rather than any one page.
Sixteen selectors move to the small radius: filter buttons, tab pills, activity
and biometrics chips, file and count badges, the jump-to-latest control, the
nav badge. Round *buttons* keep their circle — .lightbox__nav and
.home-send-btn are circles, not capsules — as do every progress track, status
dot and avatar, which are round because they are round, not because they are
tags.
**The empty voice library was unusable.** `.voice-library-empty` sets
min-height: 430px, border: 0 and background: transparent — a description of the
empty PANEL — and it had been attached to the action instead. The create button
was therefore a 430px transparent box that pushed itself out of the panel and
could not be seen. Moved onto the container it describes, which now centres its
action rather than letting it fall off the bottom. Same class-mangling shape as
the Agents header fixed earlier.
Full e2e suite: 416 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): put Host's headline figures on the shared hairline strip
Host had shadowed, clickable StatCards above a page that already has a rail, a
pane and a tab bar — a second dashboard language on one screen, and a different
one again from the figures inside its own detail pane.
The Operate overview's figure grid is generalised into a shared `.stat-strip`
and Host adopts it, so the two pages read as one system: same cell, same figure
scale, same tone vocabulary, and the same hairline grid the split-view StatGrid
already uses. The cells stay clickable and still route into the tab and filter
they describe, because a count is worth more when it is also the way to the
thing counted.
Tone is spent only where the number means something — running and updates when
non-zero — since a strip where every cell is coloured has no emphasis left.
Two bugs made on the way, both now covered:
- The first version put `<button>` elements inside a `<dl>` with `<dt>`/`<dd>`
inside the buttons. Neither is valid, the browser re-parents both, and the
cells collapsed. These cells are a set of controls, so a plain container of
buttons is also the honest markup.
- Even correct, the strip rendered 2px tall: `.page--app` is a flex column
whose split view takes flex:1, so a child with no intrinsic minimum is shrunk
away. The old cards survived only because `.stat-card` carried
min-height:96px. The strip now declines to shrink, with a test pinning it.
The stat-card specs are retargeted rather than deleted: they were written to
guard a class collision on a page that no longer uses cards, so they now guard
the strip's labels and its height.
Full e2e suite: 417 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): make Backends notices an edge rather than a filled card
The install and upgrade banners were tinted cards with a full border. A filled
panel makes every notice shout at the weight of an error, which is how notices
stop being read — and Backends shows one on most visits, so it was shouting
routinely.
They are now a hairline with a coloured left edge, the same treatment the
Operate overview gives rows that want a decision, so "this needs you" looks the
same wherever it appears. Counts in the notice take the monospace tabular
figures the rest of the console uses.
Also drops the last inline style on the page, and refreshes the inline-style
baseline, which has read 624 against a real count since #11288 landed. The gate
exits 0 either way, so nothing was failing — but a baseline 86 above the truth
would have let that many inline styles back in unnoticed. Now at 538, which
tightens the ratchet rather than loosening it.
The spec creates the upgrade it asserts on rather than skipping when the mock
has no notice: a test that skips is a test that proves nothing.
Full e2e suite: 418 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): finish the mock parity list, and stop hiding the recommendations
The last two items from the audit, plus a correction.
**Discover's use-case shelf is lanes.** These are a list of ways in, read in
order; a grid of equal cards asks the reader to compare them, which is not the
choice on offer.
**The request panel reaches every generator.** Video, 3D, Sound and Audio FX
join Images and Speech, so each one teaches its own endpoint rather than two of
six doing it. Audio FX records the fields that shape the request rather than
the bytes, since its payload is multipart.
**Recommendations no longer collapse themselves.** They were folded away by
default once anything was installed. That is the page's one opinion about this
host, and an opinion hidden by default is one the reader never gets. Someone
who disagrees can still collapse it and that choice is remembered — the
difference is that we no longer make it for them. Three specs asserted the old
default and now assert the new one.
The use-case heading also sat a line's width from the text it introduces, so
the two read as one paragraph. It has air under it now, and the shelf is
separated from the recommendations above it.
Two tests removed rather than kept: a generator loop whose only real assertion
was `expect(endpoint.length).toBeGreaterThan(0)`, and an earlier card-gap guard
that could only skip. A test that cannot fail is worse than no test, because it
reads as coverage.
Full e2e suite: 418 passed, 4 skipped. Inline styles at baseline.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): make the Host figures legible and give the strip its spacing back
Three defects introduced by the Host redesign, all found by looking at the
running app rather than by the suite.
**The figures were invisible.** "Running now" and "Updates available" rendered
pure black on the dark ground. Two causes compounding: the `--muted` tone alias
never landed, because the source rule has extra spaces before its brace and the
exact-match edit missed it silently; and a `<button>` does not inherit colour,
so with no tone rule the value fell back to the user agent's `buttontext`.
Both fixed, and a test now fails on any figure computing to pure black.
**The strip sat flush against the resources panel.** `.stat-strip` declares
`margin: 0 0 ...` and is declared later in the file than `.manage-summary`, so
the shorthand quietly won and the top margin became zero. Raised to
`.stat-strip.manage-summary` so it beats the shorthand on specificity rather
than on declaration order, which is the kind of thing that breaks again the
next time a rule moves.
**Discover's use-case heading had a doubled gap.** `.zero-pane` is a flex
column that already separates its children; adding a margin on top of the gap
stacked the two. The margin is gone and the heading keeps only its own breathing
room.
Full e2e suite: 420 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): make Studio's tabs path segments rather than a query parameter
`/app/studio?tab=images` reads like a filter applied to a page. It is
navigation: a different generator, with its own state and its own deep link. It
is now `/app/studio/images`, with the overview at `/app/studio`.
Legacy `?tab=` links are redirected once to the path form, replacing the
history entry so Back does not bounce between two spellings of the same place.
Bookmarks and older links keep working and land on the canonical URL rather
than a second version of it, which is the part worth having a test for.
The nine `?tab=` references were all in specs, none in docs, so the migration
is contained. They move to paths, and a new spec pins the redirect.
Full e2e suite: 421 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): make the hardware recommendation a section, not a dismissable card
It was a bordered card with a collapse control and a close button, sitting
inside a pane that is otherwise hairline sections. Two problems: it read as
something bolted onto the page rather than part of it, and treating it as an
interruption to be shut is the wrong frame for the one thing the page has to
say about the machine it is running on.
It is now a plain section with the same heading treatment as the shelves below
it. The collapse state, the dismissal, their storage keys and the legacy key
read for backwards compatibility all go with it, along with the installedCount
prop that existed only to pick a default collapse.
Five specs described behaviour that no longer exists and are removed rather
than adjusted — collapsing, dismissing, persistence of both, and the toggle's
keyboard handling. One new spec asserts the replacement contract: no control
with aria-expanded, no dismiss, and no card border.
Full e2e suite: 416 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): restore every stripped icon and every default-chrome control
You reported two broken icons. They were not two: an earlier automated edit had
stripped the `fa-*` class from twenty `<i>` elements across eleven pages, and an
`<i>` with no icon class renders nothing at all. Settings' save button, the
voice-profile back link, and eighteen others — agent row actions, task and job
buttons, import and create actions — were all drawing empty space.
Each is restored from its own context rather than a blanket icon: the agent row
gets pause/play, pen, comments, file-export and trash; the fine-tune toggle
swaps plus for xmark as it opens; the P2P documentation link gets the
external-link glyph.
The same edit left controls without their classes. Fine-tune's "Import config"
was rendering in the browser's own chrome, and `.p2p-cmd__copy` set a border
but no background, so it fell back to `buttonface` — a pale grey chip on a dark
command block. FineTune's "New job" also had its icon classes folded into the
button's className, the same mangling already fixed on the Agents header.
Rather than fix the reported two and wait for the next report, this adds a
standing audit: twenty-five routes are walked and the test fails on any visible
control rendering with user-agent chrome, or any `<i>` without an `fa-*` class.
It found the three remaining cases after the first sweep, and it is the reason
the next one cannot ship quietly.
Full e2e suite: 417 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): give Operate a front door and fold six rail groups into four
Opening Operate ran firstVisiblePath() and landed on Backends, because
Backends happens to be written first in operateConsole.groups. The section
that should answer "is anything wrong" opened on a package manager, and
nothing was reported until you visited it.
Adds /app/operate. Its one irreplaceable block is "Needs attention", which
is empty when nothing is wrong and says so in a line rather than rendering a
reassuring green panel. It collects stale backends, failed operations and
unhealthy nodes. Everything else on the page is a summary you could already
assemble by visiting four others.
The rail regroups from six headings to four: Inference and Activity were both
"the runtime right now", Access and System were both administration. No
destination is removed and no gate changes, so isConsoleItemVisible and
consolePaths are untouched. Overview leads the first group, which is what
makes firstVisiblePath() return it without knowing it exists.
Rail entries now carry a signal beside the label. This does not replace the
sidebar badge and is not built as if it does: the badge stays on the
always-visible sidebar entry for the reason recorded in Sidebar.jsx, that the
rail exists only on Operate routes and can be collapsed. The signals are
orientation while inside Operate, so they are aria-hidden and nothing urgent
depends on them alone.
OperateSummaryContext polls once for the whole console, following
OperationsContext, which exists because per-consumer setInterval against one
endpoint was the defect it fixed. It is mounted by ConsoleLayout for the
Operate console only, so "poll only while in Operate" needs no route check.
Built on usePolling, so it pauses on a hidden tab. Operations are read from
OperationsContext rather than polled a second time, and each source degrades
to no-signal on its own so one dead endpoint cannot blank the rest. It reads
the cached GET /api/backends/upgrades and never the POST that forces a real
registry check.
Traces and Usage get no signal yet: /api/traces returns the list, so a count
would mean fetching every trace to render one number. A counts endpoint is
the honest fix and is scoped separately.
Full e2e suite green (369 passed, 4 skipped), including a render-smoke entry
for the new route.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): open Studio on what this machine can actually make
Studio was a tab strip over six generators that opened on Images, which was
never a decision, only the first entry in BASE_TABS. Nothing said which
modalities this installation could run, so the way to learn that video had no
model was to pick the tab and find an empty select.
Adds an overview tab and makes it the fallback. Explicit tabs still win, so
existing deep links keep working; anything unrecognised or gated now lands on
the overview rather than Images.
Each tab carries a dot: filled when an installed model advertises that
modality, hollow when nothing serves it. That is the feature in one detail,
turning the strip from navigation into a report of what the machine can do
before anything is clicked. The dot is aria-hidden because the overview states
the same facts in words and the dots change as models load.
Two kinds of unavailable, which had to stop looking alike:
- switched off, via a permission: no tab and no lane, unchanged
- available with no model: a lane, and a route to installing one
Studio now owns one MODALITIES table so the tab strip and the overview cannot
disagree about what exists, and calls useModels() once, unfiltered, grouping in
the browser. useModels(capability) fetches the whole list and filters locally,
so a hook per modality would have been six identical requests to
/api/models/capabilities on every mount. There is a test for that.
Recent outputs read every localStorage store through a new
readAllMediaHistory(), which avoids mounting five hooks that carry save timers
the overview has no use for. 3D is read separately through use3DHistory rather
than folded in: its entries are GLB blobs in IndexedDB, so they cannot come
from the same synchronous read.
Typical cost is the median of this machine's own history, not a guess, and
renders as a dash when there is nothing to go on.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): stop the stat cards and the console rail breaking on small screens
Two unrelated causes behind one report that /app/manage looks wrong when the
window is narrow.
The stat cards were being laid out by the wrong rule at every width. Two
different components both claimed `.stat-grid`: the dashboard card strip that
holds .stat-card children, and the detail-pane StatGrid the split views
introduced further down App.css. Being later, the second won every shared
property, so the cards got its 120px columns and its 1px hairline gap in place
of their own 180px columns and spacing-md. Four cards were packed onto a row
that fits two, labels wrapped to three lines and clipped, and the icon crowded
the value. Renamed the strip to `.stat-cards`, after the children it actually
holds, which also removes the mismatch of a `.stat-grid` container full of
`.stat-card`s. The split-view component keeps `.stat-grid` and its BEM parts.
The expanded console rail had no bounded height. Thirteen destinations stacked
in one column is taller than a phone, so opening the menu pushed the page's own
heading past the fold: the menu replaced the page rather than annotating it.
Capped at 55vh with internal scrolling below 768px, so the content behind stays
reachable.
Both are asserted on behaviour rather than markup: no stat-card label may be
clipped, the card gap must not be the detail pane's hairline, and expanding the
rail must leave the page heading on screen.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): retemper the palette to localai.io and add the lane primitive
The token half of the style transfer, plus the shared list idiom the two
overviews had each grown their own copy of.
theme.css moves from Nord to the website's palette, variable names preserved so
every consumer moves with it: ground #13171f -> #0d1117, accent frost cyan
#88c0d0 -> action blue #4f8cff, success sage -> mint #56d6a4, warning -> the
amber #f1b95d the site spends only on the thing asking for a decision. Eyebrows
go mint. Dividers become an opaque #29384a hairline rather than alpha over a
varying surface, which is what makes stacked surfaces read crisply on the site.
Light is derived, not inverted. The site ships one theme and never had to
answer this, but the app does: blue darkens to #2f62d8, mint to #0d8b60 and
amber to #8a5d0b, all clearing 4.5:1 on a cool paper ground, where the
dark-mode values sit near 2:1. Same three roles, different values.
Three files restate the palette because CSS variables cannot reach them:
cmTheme.js (the whole CodeMirror theme), VoiceVisualizer and WaveformPlayer
(canvas). Left alone they would have quietly kept the app half-Nord.
The `.lane` primitive replaces the near-identical row CSS that OperateOverview
and StudioOverview had each written: a full-bleed row on a hairline that insets
on hover, with no card and no shadow. Callers supply only the column template.
Both pages now use it, along with `.lane-head` for section rhythm and a
`.page-pad` container for top-level pages outside a console shell — without
which Studio sat flush against the sidebar with its eyebrow clipped.
Studio's tab strip wraps rather than running off the edge at narrow widths.
Full e2e suite: 386 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): put Home's resident models on lanes and give the footer one line
Home's status line was three chips saying a thing was true. It now reports
figures: how many models are resident, how many nodes are healthy, what share
of memory is in use, set in tabular monospace so the digits line up. A chip
answers whether; a figure answers how much, which is what someone opening the
page at a glance is after.
Resident models move from status chips to lanes, with the id set in a new
`.lane__name--id` because an id is something you might type or paste and the UI
face makes it read as a label. /api/system-information carries only the id, so
there is deliberately no backend or memory column: inventing one would mean a
server change this does not make.
The footer was three centred rows and cost the bottom sixth of every page for
chrome. It is one line now, version left and links right, wrapping to centred
when the viewport is too narrow to hold both. Every link it had, it keeps.
Full e2e suite: 392 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): correct three contrast failures and stop a guaranteed-404 poll
A contrast audit of the new palette found three values below WCAG AA, one of
which the previous commit message claimed was fine:
- White on the #4f8cff button is 3.22:1, which is large-text only. The website
does exactly this, but a button label in an app is not large text, so the
label goes to dark ink at 5.88:1. Light mode keeps white, which is 5.44:1 on
its darker blue.
- Light-mode success was 4.08:1 on paper, not the 4.5 claimed. Darkened to
#0a734f, 5.56:1.
- Nord red was already 4.28:1 on raised surfaces, a pre-existing miss carried
over unexamined. Lifted to #c96f78, 5.02:1.
Lanes gain the two states they were missing: a 44px target on coarse pointers,
matching what EntityRail already does so the two list idioms feel the same
under a thumb, and a reduced-motion variant that keeps the background feedback
while dropping the hover inset, which is a position change.
The Operate summary no longer asks for /api/nodes on a single-node install. The
cluster API answers 503 when distributed mode is off, so it was a guaranteed
miss every fifteen seconds; it is now gated on useDistributedMode, the same
condition the rail already uses for the Nodes entry.
Full e2e suite: 392 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): restore the gap between overview blocks, and stop claiming zero nodes
Two defects a design review surfaced.
`.lane-head:first-child { margin-top: 0 }` was meant to stop the first block on
a page carrying a top margin. But every <section> makes its lane-head a first
child, so the reset applied to all of them and the gap between blocks vanished:
"Sections" sat flush against the attention row above it. The header supplies its
own bottom margin, so a uniform top margin is correct everywhere.
The Cluster summary read "0 nodes" on a single-node install, which looks like a
fault when the cluster API is simply switched off. It now says "Single node".
Full e2e suite: 392 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): open dark by default, and stop clipping the collapsed sidebar footer
Dark is the identity rather than a preference: localai.io ships one theme and
it is this one, so an install should look like LocalAI before anyone has chosen
anything. The OS setting no longer selects light on first load. The toggle
still does, and a stored choice wins forever after, which the tests assert
both ways.
The collapsed sidebar footer stacked its controls but kept the expanded row's
inline padding, so their edges were clipped against the 51px rail.
Full e2e suite: 394 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(api): count traces server-side and give the Operate overview real totals
The overview's headline block had no source. /api/traces returns the trace
list, so "37 errors in 24h" meant fetching every buffered exchange to count it
in the browser — waste that grows with the buffer, to produce three integers.
Adds GET /api/traces/summary: totals, failures, p95 and a bucketed series for
sparklines, over a window that defaults to 24 hours and is capped at a week.
Deliberate calls, each with a spec:
- A 4xx is the caller getting it wrong, not the installation being unhealthy,
so only 5xx and transport errors count as failures.
- p95 is a nearest-rank percentile rather than the slowest request, which is
what a max would report and what makes latency panels lie.
- Buckets are oldest-first so a sparkline reads left to right, and the slice is
never nil: nil serialises as null and breaks .map() on the other side, which
is a silent runtime error rather than an empty chart.
- Exchanges outside the window are not counted at all.
The route is registered before /api/traces/:id so "summary" is not captured as
a trace ID.
On the client, Traces and Usage gain the rail signals they were shipped
without, the Observability section summary now states counts instead of listing
its destinations, and an installation that has served nothing says so rather
than showing three zeroes dressed as telemetry.
Sparkline is a bare stroke with an emphasised endpoint and no axes: the figure
above it already states the value, so its only job is the shape.
Go: 185 middleware specs pass. Full e2e suite: 396 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): stop the memory chart calling a trade-off an error
The VRAM-by-context chart rendered any build over the limit in error red, and
escalated the verdict to the error tone as soon as two context sizes crossed
it. But an over-limit build still installs — #11288 keeps a test on exactly
that — so red overstates what is happening. A model that fits at 32k and not
64k is a trade-off, not a fault.
Over-limit bars and the limit line now use the warning tone, which is the
constraint colour used everywhere else in this branch: know what you are doing,
not you may not. The error tone is reserved for "fits nowhere", where the model
genuinely cannot run on this host.
Full e2e suite: 397 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): give the new surfaces orchestrated motion
Uses the reveal system already in the codebase rather than adding a library:
pageReveal, .reveal-stagger and staggerStyle() were built for exactly this, and
anime.js would be ~17KB duplicating four lines of CSS for list reveals.
The overview's headline figures, attention rows and section lanes stagger in,
as do Studio's modality lanes and recent outputs, so a page assembles in the
order it is read instead of appearing all at once.
Two additions beyond stagger. Rail signals transition on opacity when a poll
lands, so a number changing reads as an update rather than a jump cut, and it
stays on the compositor so it cannot reflow the rail. The attention block
animates its left edge in — the one thing on the page that should announce
itself, and on the border rather than the text so nothing moves under a reader.
Both are dropped entirely under prefers-reduced-motion, alongside the lane
hover inset already handled.
Full e2e suite: 397 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): put the generators on a hairline field stack and record the request
The workbench treatment from the mocks, applied where it costs least: both
changes land on shared surfaces, so all six generators get them at once rather
than drifting apart page by page.
The control column stops being a shadowed card of boxed groups and becomes a
hairline field stack — the panel is the page's left half, not an object
floating on it — with uppercase micro-labels matching the eyebrow treatment
used elsewhere. Because .media-controls is shared, Images, Video, 3D, Speech,
Sound and Audio FX all move together.
RequestPanel shows the request the form actually built, with a copy-as-curl.
LocalAI is API-first and Studio is the best place in the app to teach its own
endpoints: the form stops being a black box, and a result worth keeping can be
reproduced from a shell without reverse-engineering which fields the page sent.
It records what was sent rather than what the form currently holds, and renders
nothing until a request has been made — a panel describing a request nobody
made is a tutorial, not a record. Wired into Images and Speech.
Full e2e suite: 401 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): make Chat a transcript instead of a bubble thread
Rounded, filled, asymmetric bubbles fight a system built on hairlines, and they
carry the speaker in shape and side rather than in words. The assistant side
had already given up its bubble; this finishes the job.
Both roles now run full width down one column, separated by a rule, each with a
mono role label. The user turn keeps a left edge in the action tone so the two
are still told apart at a glance, without a fill or a corner radius. The
avatars go: the accent and the label carry the speaker, so the glyph was
decoration once neither side had a bubble.
Saying who is speaking in words rather than in geometry is also what survives
being read aloud, printed, or looked at by someone who cannot pick the sides
apart by colour.
Full e2e suite: 404 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): dress the API reference in LocalAI's palette
The Swagger page was the last surface still shipping in someone else's colours,
which is conspicuous now that everything it links from is dark.
Swagger UI has no theming hook, so rather than fork it we serve our own index
ahead of the library's wildcard and restate the palette over its stylesheet.
The library's own bundle and assets are still what load, so a swagger-ui
upgrade cannot silently break the page — this is a skin, not a fork.
Two things needed real care. Swagger tints the entire operation row per method
via .opblock.opblock-post and friends, so the palette had to match that
specificity rather than reach for !important; the method now lives on one edge
instead of washing across the row, because a page where every row is a status
colour has no status colour left. And the filled method chip put white on pale
green, which was the least readable thing on the page — it is an outlined mono
chip now, carrying the method in its border and text.
Palette values are copied from theme.css rather than referenced: this page is
served by Go and never sees the app's CSS. The comment says so, and says to
keep them in step.
Go: routes and middleware suites pass. Full e2e suite: 405 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): make tall split-view pages reachable, repair the Agents header, scale titles
Three things found by actually using the app rather than measuring it.
**Host was unusable.** The shell above a split view is overflow:hidden so the
document cannot grow, which left anything taller than the viewport simply
unreachable — and Host stacks a resources card, four stat cards and a tab bar
above its split, so the bottom of the pane fell off at every window height with
nothing to scroll. Every sweep I ran for this was horizontal, which is why it
kept coming back clean.
The page now scrolls inside the pinned shell. The pane keeps its own scroller:
letting it grow instead pushes the document taller and stretches the rail to
match, which is the regression e2e/discover-height.spec.js exists to catch, and
which the first version of this fix duly caused.
**The Agents header controls were unstyled** — "Create Agent" was rendering
with the browser's default chrome. The markup had been mangled at some point:
six unrelated classes merged into one string on the link, and the label and
button left with none at all and empty icons. Repaired, with the inline flex
replaced by a shared .header-actions class.
**Page titles take the editorial scale from the site**: larger, tracked at
-0.04em, on a line height near 1, so a two-word title reads as a statement
rather than a label. The typeface is unchanged — DESIGN.md keeps the existing
type system — so the whole difference is scale, tracking and leading, which is
where the site gets its voice from. This was the biggest reason the running app
still did not look like the mocks.
Full e2e suite: 404 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Follow-up to #11288, which warmed the VRAM estimate caches at startup and left
the variant picker paying its own way.
Describing an entry's variants probes the weight files of every build it
offers, so the first time a model is opened costs 1.2-1.9s against a cold
cache. That is the same cost as an estimate wearing a different hat, and it
lands in the same caches underneath, so it belongs in the same pass rather than
in a second mechanism.
The warm-up now describes variants for the entries it walks. Entries that
declare none cost nothing: the call is gated on HasVariants rather than
attempted and discarded. The host resolve env is derived once for the run,
since it describes the machine rather than the entry.
Failure handling matches the estimate half. An entry whose variants cannot be
described is logged at debug and skipped, and the estimate for that same entry
is unaffected, because neither half is allowed to fail the other.
Measured against a live instance with 1,595 models, first ever call to
/api/models/variants/:id after a cold boot:
before 1.2-1.9s
after 2ms
The warm-up's own cost barely moves: 3m0s to 3m19s for 300 entries, of which
40 declared variants. It stays bounded by the same knobs, and
LOCALAI_VRAM_WARM_LIMIT=0 still turns the whole thing off.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): rename the Install Models nav entry to Discover
"Install Models" named the action rather than the destination, and it was
the only multi-word entry in a rail of one-word ones (Home, Chat, Studio,
Talk, Build, Operate). A bare "Models" was the obvious fix but it collides
with the installed-models view under Host, which is a different page for a
different job.
"Discover" keeps the rhythm and says what the page is for. The icon moves
from a download arrow to a compass for the same reason: the page is browsed
before it is installed from.
Translated in all seven locales rather than left to fall back, so a locale
switch does not leave the entry in English.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(ui): replace the gallery table with a rail and a detail pane
The eight-column table was not the real problem; the click-to-expand row
underneath it was. Variants, files and a VRAM estimate never fitted inside a
<tr>, so they were pushed into a drawer that could hold one model at a time,
could not be linked to, and had no room to say anything useful.
The gallery is now a rail to scan and a pane that answers. The pane has two
states and no third: with nothing selected it is the discovery page, and with
a model selected it is that model's detail. Selection lives in the URL, so a
model is linkable and Back steps out of the detail instead of off the page.
The rail groups by capability while browsing and flattens to results the
moment a term is typed. That is a rule rather than a toggle: once someone has
said what they are looking for, the buckets are between them and the answer,
and making the user choose would be handing them our problem.
The detail pane plots VRAM against context length with the host's own limit
drawn across it. This is new information, not a restyle. A single number
invites "so will it run?", and the honest answer is usually "yes, up to a 32k
context", which is a shape rather than a number. The estimates were already
fetched for every context size, so it costs no new request. Backends that
take no context length say so instead of being given a meaningless chart, and
a host with no GPU gets no chart at all rather than bars with nothing to
compare against.
The split-button variant menu goes with the actions column. The pane lists
every build with its backend, quantization, size, fit and a details
disclosure, each installable, which is what the dropdown was a cramped
substitute for. Its tests move onto that list; the three contracts it alone
carried (fetch-once caching, the loading state, an unfit build staying
installable) are backfilled against the pane.
RecommendedModels moves inside the pane, where it has the width to argue for
a model instead of listing one, and keeps its own dismissal and collapse.
Rail entries carry no description. Two lines is the budget and the second is
better spent on whether the thing will run; the stripped-Markdown contract
moves to the pane's lede, tooltip included.
e2e: 123 passing across models-gallery, navigation, recommended-panel,
model-artifact-operation, operations-strip and page-render-smoke. Inline
styles in Models.jsx drop from 82 to 41.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* refactor(ui): extract the split view into shared components
Discover shipped its rail, pane and detail header as private functions inside
Models.jsx. Backends and Host have the same defect and want the same shape, so
leaving them there guarantees three rails that drift.
SplitView, EntityRail, DetailHeader and StatGrid now live under
components/split/. EntityRail is deliberately data-driven: a surface maps its
own entity onto { id, name, icon, meta, stripe, groupId } and keeps its
vocabulary to itself, which is what stops the rail learning about models,
backends and loaded state all at once.
The CSS moves with it. What was .discover__rail is .entity-rail, .discover__
pane is .split-view__pane and so on, because a class named after one page is a
lie on the next two. Only what is genuinely Discover's stays behind the old
prefix: the shelves, the hero and the VRAM-by-context chart.
Two additions the shared rail needs and Discover did not: a state stripe, for
surfaces read by condition before they are read by name, and an empty label.
Discover passes neither.
No behaviour change. e2e 100 passing across models-gallery, navigation and
models-recommended-panel.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(ui): put the backend gallery on the split view
Same defect as the model gallery, so the same shape: a seven-column table over
a click-to-expand row that was the only place the repository, licence, tags and
links could go.
The rail groups backends by the use case they serve, sharing Discover's
taxonomy on purpose: a backend is the runtime a use case needs, so "vision"
ought to mean the same thing one level down. It flattens on a query for the
same reason it does on Discover.
The zero state is the one real departure. A backend's fitness is not free
memory, it is the accelerator and platform it was built for, so the pane leads
with what this host is, then what is not installed yet, then whether anything
installed has gone stale. The table listed 37 runtimes and left "which of these
can even run here" entirely to the reader.
Distribution moves into the pane, which is the one thing a row could never
carry: which nodes hold a copy and which do not, with the install-on-more
control next to it rather than squeezed against a chip.
The distributed and target-node action logic is unchanged, including the guard
that keeps a hardware-specific build off the fan-out path. The split-button
popover loses its per-row anchoring because there are no rows; one pane, one
anchor.
Selection lives in ?backend=, preserving the ?target= scope rather than
clobbering it.
e2e: 139 passing across models-gallery, navigation, backends-management,
models-recommended-panel, nodes-per-node-backend-actions, page-render-smoke,
operations-strip and model-artifact-operation. The backends spec gains six
split-view tests; its three description-cell tests move onto the pane lede.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(ui): put the Host inventory on the split view
The last of the three surfaces, and the one that is not a catalog. Both tabs
had the same click-to-expand row, so the shell transfers; what does not
transfer is the zero state, because there is nothing to discover in your own
inventory.
With nothing selected the pane reports what is happening: how many models are
loaded, what failed, what has an update, and which models are holding VRAM
right now. Every number was already on the page. None of them had been
assembled into one statement, so "what is going on" was a question the tabs
could not answer however long you looked at them.
The rail buckets by state rather than capability - Running, Idle, Disabled for
models; Update available, Installed for backends - which is the opposite of the
galleries and deliberately so: nobody opens Host wondering which of their
models does vision. Entries carry a state stripe for the same reason.
Load and Stop are promoted out of the kebab, because that is what an operator
came for; the rest stays behind the menu rather than diluting it. Adopted,
pinned and alias badges follow the model into the pane: they are facts about
the thing, not about its state, and the rail line is spent on state.
Deliberately NOT done: folding the two tabs into one rail, as the mock had it.
It costs five URL parameters, the manage-tab localStorage key and the
stat-card shortcuts, all of which are live deep-links today. The tabs stay as
the group selector; merging them is a follow-up with its own migration.
e2e: full suite 355 passing. New host-split-view spec; alias-template,
manage-logs-link, manage-action-menu-position and model-editor-back-nav move
off `.table` and the row kebab onto the rail and the pane.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* polish(ui): accessibility and consistency pass over the three split views
Findings from a pass over what the previous four commits actually shipped,
rather than what they were supposed to.
The rail was not a listbox. ARIA lets a listbox contain options and groups,
and nothing else, but each group's collapse control is a button that has to
sit inside the scroller with the entries it folds. It is now a labelled group
of buttons, which is the honest description; selection is announced with
aria-current and the arrow keys are unaffected.
Every entry was its own tab stop, so tabbing past a forty-entry rail to reach
the pane took forty keystrokes. Roving tabindex makes the rail one stop, and
arrowing now moves focus with the selection instead of leaving it behind on an
entry Tab can no longer reach.
The rail rounds its corners with overflow:hidden, which was clipping the focus
ring off the first and last entries entirely. Inset outlines fix it.
A 30px row is fine under a mouse and too small under a thumb, so coarse
pointers get a 44px target without costing density on a desktop.
One slot said three different things: "9 models loaded" on Discover, "12
loaded" on Backends, "3 of 9" on Host. All three lists are a page of a larger
set, so all three now say so the same way.
Also removed: an emptyLabel prop on EntityRail that nothing passed, its dead
CSS rule, and MODELS_COLSPAN and ResourceRowDesc, which died with the tables.
e2e: full suite 355 passing.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(ui): correct three defects only a real gallery exposed
Running the branch against a live instance with 1,595 models and 1,017
backends, rather than against mocked fixtures, surfaced three things the e2e
suite could not.
Grouping did nothing. The rails matched on the use-case keys the filter chips
send (`chat`, `tts`, `transcript`), but those are a server-side vocabulary the
handler maps onto entries. What entries actually carry is free-form and
inconsistent: models come back tagged `llm`, `gguf`, `vision`, `coding`, and
backends `LLM`, `text-to-text`, `audio-transcription`. Nothing matched, so
every model landed in "Everything else" and the feature was decorative.
Grouping now lives in utils/entityGroups.js, shared by both galleries, matching
case-insensitively against the vocabulary the API really uses, with the entry's
backend as a fallback signal - a backend named `whisper` is a speech backend
whatever its tags say. Order is specific before general and that is
load-bearing: a vision model is tagged `llm` too, so testing text first would
swallow it.
The zero state claimed GPU memory on a machine with no GPU. The resources
endpoint reports system RAM in the same field when gpu_count is 0, so the hero
read "84.4 GB of GPU memory" next to the recommendations panel correctly
saying "No GPU detected". The number was never wrong, only its label; it now
says system memory unless a GPU is actually present.
The page title still said "Install Models" under a nav entry saying Discover.
Also: the keyboard test named the model it expected to arrive at, which made it
a hostage of the grouping table and broke the moment the buckets were fixed. It
now asserts that the selection moves and returns.
e2e: full suite 355 passing.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(ui): the filters and the rail were fighting over the same job
Four things you find odd on Discover, and they turn out to be one mistake seen
from four sides.
The rail grouped the current page. The listing is paginated at nine rows, so
those bucket headers described nine entries out of 1,595, and turning a page
reshuffled the sections under the reader. The structure was never stable
because it was computed over the wrong set.
The chips were redundant for the same reason, seen from the other side. They
send tag= and filter all 1,595 server-side. The rail grouped nine of them
client-side by the same axis. Two controls for one job, and the weaker one was
the one this branch added, so it goes. Grouping stays only on Host, where the
list is complete, local, and bucketed by state rather than capability.
The search bar felt odd because it sat in a full-width band while the thing it
narrowed was a 290px rail below and to the left. The whole band now lives in
the rail column: search, backend, use cases, refinements, then the list it
narrows. One column to say what you want, one to show what you got. Nineteen
chips do not fit at that width, so they fold into a disclosure that states the
selection. A disclosure and not a popover, deliberately: picking use cases is
multi-select and interleaves with the backend select and the toggles below,
and a popover dismisses itself the moment you touch either.
The header held two counts and two buttons at arm's length from all of it. The
counts were the third statement of the same number on one screen, after the
rail's "9 of 1,247" and the pane's own headline, so they go. The buttons move
into the pane's zero state, which is the surface that answers "what do I do
here".
Also: the two first-run empty states wore .loading-center, which is
display:flex in the default row direction because it exists to centre one
spinner. With four children that put the icon, the heading, the sentence and
the buttons on a single line with no gap. They are now a proper full-height
empty state.
e2e: full suite 353 passing. Grouping tests are replaced by ones asserting the
rail stays flat; chip tests open the disclosure first; two filter-layout tests
that asserted the old three-band arrangement now assert the column.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* polish(ui): make Discover a full-height view, group the chips, name the refinements
Four things, all of them the same complaint: the page read as a document with
controls scattered on it rather than as one view.
The header is fused. A title block with its own padding, a subtitle and two
counts made the split view look like an attachment to a document that happened
to sit below it. It is now a slim bar carrying the title, the count and the two
page-level actions, and the split fills the rest of the window. Rail and pane
scroll independently, so the filters and the pane's headline stay put while a
long list moves under them.
The chips group. Nineteen in a flat row is a lot to scan even behind a
disclosure, and they already belong to the four families the rest of the UI
speaks, so they are bucketed by those. "All" sits on its own above them without
a heading, because it is a reset rather than a use case.
The refinements stop looking dumped. When the band became a column they were
three controls left where they landed; they now read as a named section with
one control per row.
The zero state suggests again. It had decayed into a "Browsing / 9 of 1,247 /
select a model" line that restated the count for the third time on one screen.
It now offers the four use cases as tiles that set the filter, which is the
shelf idea from the mock without inventing curation or paying for a second
fetch.
Two bugs found by looking at it rather than at the tests: the disclosure was
clamped to 190px, which cut it off partway through its third section so two of
the five never appeared at all; and the creation actions rendered twice, once
in the new bar and once in the pane hero a few pixels away.
e2e: full suite 353 passing. The chip-row test now holds its contract across
the per-family rows rather than a single one, and additionally asserts every
family is present and non-empty.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(ui): pin the split view's height so a long detail scrolls the pane
Selecting a model with a long description grew the whole page and dragged the
rail down with it, which is the opposite of what "full height" was supposed to
buy.
The flex chain was right and the ceiling was missing. .app-layout and
.main-content are min-height:100dvh, which is a floor: flex distributes free
space but nothing caps growth, so a pane taller than the viewport expanded the
column, the document scrolled, and the rail stretched to match. height:100% on
the pane then resolved against an auto-height parent and did nothing.
The chat route already solves this by pinning .main-content to 100dvh. The
same treatment now applies to any route containing a .page--app, selected with
:has() so the shell does not have to learn which pages happen to be split
views. Below the stacking breakpoint the pin is lifted, because two stacked
halves in two short scrollers is worse than a page that scrolls.
Measured on a live instance: document height stays at the viewport across
selection (950px either side) and the pane overflows internally instead.
Adds discover-height.spec.js, which asserts the page height and the rail height
are unchanged by selection and that the pane is the thing that scrolls. The
existing specs could not have caught this: they mock short descriptions, and
the bug only appears when the pane has more content than the viewport holds.
e2e: full suite 355 passing.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(ui): give Backends and Host the full-height view, and fix the Update button
Backends now matches Discover: the header fuses into a slim bar carrying the
title, the count and the page-level actions, the filters move into the rail
column where they narrow the rail and nothing else, and the split fills the
window. Its seven chips fit at rail width, so unlike Discover's nineteen they
need no disclosure. Host gets the bar and the height; its resource monitor,
summary cards and tabs stay above the split, because those are read once while
the rail and the pane are worked in.
Two things the height change surfaced.
The console layout is a flex row with align-items:flex-start, so its body sizes
to content. Right for the pages it was built for, wrong for a split view, which
needs a ceiling to scroll inside: without it the Backends rail ran past the
viewport and over the footer. Pinned with :has() so only split-view routes are
affected.
The filters vanished when nothing matched. Both galleries swapped the whole
shell for an empty state, which took the search box and the chips with it, so
the page said "try adjusting your search or filters" while offering neither.
The shell now stays and the empty state moves into the pane.
Also fixes the Update control on Host, which had no className at all and
rendered as bare text, next to a status span that had picked up btn classes and
two copies of `fas` and so rendered as a button you cannot press. They have
swapped appearances back.
e2e: full suite 355 passing. The render-smoke selector learns .view-bar__title,
since the pages it checks no longer all use PageHeader.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(ui): keep the view mounted while searching, and bring rail grouping back
Searching replaced the whole view with a loader. The search box lives in the
rail column, so every debounced refetch unmounted the field being typed into
and dropped its focus with it. The list, the filters and the pane went too.
The shell now stays and the rail says it is busy: a sweep bar under its header
and the stale list dimmed, so the eye knows the answer is being replaced
without losing its place. A cold start still gets the skeleton, because there
is nothing to keep.
The condition for that is "nothing has loaded yet", not "the list is empty".
Those differ exactly when someone is editing a query that matched nothing, and
getting it wrong there would unmount the view on the keystroke after a
no-results search - the worst possible moment.
Grouping comes back on both galleries. It was removed because nine rows could
not fill five buckets, so a page turn rebuilt the rail's whole structure. That
was a symptom of the page size rather than of grouping: the rail now asks for
30 rows instead of 9 (Backends 60 instead of 21), which is enough for the
sections to read as structure and turns five times fewer pages. The order of
the sections is fixed, so what changes between pages is membership, not
arrangement.
Grouped while browsing, flat while searching, as before: once a term is typed
the buckets stand between the reader and the answer.
Also gives GalleryLoader a class and a testid instead of six inline style
declarations on a bare div, which is why nothing could select it.
e2e: full suite 359 passing, including a new spec asserting the search box
keeps its focus and its value across a refetch, and that a cold start still
shows the skeleton.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* perf(gallery): stop invalidating the VRAM estimate caches on every request
Searching or turning a page felt slow. It was not the search and not the
listing: /api/models answers in 3-9ms. It was the VRAM estimate, which the
gallery asks for once per row, and which took ~2.3s every single time however
often the same model was asked about.
pkg/vram already caches what makes that expensive - the remote content-length
probes, the GGUF metadata reads and the HF repo sizes. Those caches key on a
gallery generation counter, and AvailableGalleryModelsCached triggered a
background refresh on every call, with each refresh bumping the counter. One
page view is one listing request plus thirty estimate requests, each of which
re-read the gallery and started another refresh, so the generation moved
constantly and every cache entry was stale before it could ever be read. The
caches were dead in production.
Three changes, each doing one thing:
A refresh interval. The cached list is still served immediately; this only
decides how often re-fetching from upstream is worth starting. Five minutes,
as a package variable so tests can drive it without waiting.
A generation bump only when the gallery actually changed. An unchanged gallery
re-fetched on schedule must not throw away work that is still valid, which is
the difference between an estimate costing nothing and costing a network round
trip.
A separate "loaded" flag. The cache engaged on `cached != nil`, so a gallery
that legitimately holds nothing read as never-loaded and took the blocking path
on every call, bumping the generation each time. Found by the test for the
interval, which could not pass while this was true.
Measured against a live instance with 1,595 models:
one estimate, repeated 2.3s -> 2ms
a page of 30, in parallel 10s -> 0.04s
A first, genuinely unseen model still costs its remote probe. That is inherent;
what changed is that it is now paid once per model per gallery version rather
than once per request.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* perf(gallery): warm VRAM estimates at startup, and stop the UI waiting on them
Two halves of the same complaint: the gallery stalls on VRAM estimation.
Server side, the estimates are now warmed in the background at startup.
Estimating an entry nobody has asked about costs a remote probe of its weight
files, and the gallery needs one per row, so the first visitor was paying for
the whole page. The warm-up walks the gallery in the order the UI lists it, so
the first page is ready before anyone reaches it.
It is bounded and it never blocks: 300 entries at 4 at a time by default, on
its own goroutine, stopping with the server's context. Warming the whole
gallery would be thousands of probes on every boot, which is rude to the
upstream and slow to finish; warming nothing leaves the first page paying two
seconds a row. Anything past the limit still warms itself on first view.
LOCALAI_VRAM_WARM_LIMIT=0 turns it off for an air-gapped host,
LOCALAI_VRAM_WARM_CONCURRENCY=1 slows it for a metered link.
Client side, the page no longer waits on estimates it does not need yet. It
fired one request per row at once; a browser allows about six connections per
host, so thirty estimates took every slot and the request behind a click - the
variant list, an install - queued behind work nobody asked for. That is the
freeze: the list was already usable, and the UI was busy fetching sizes. Four
at a time leaves room for the interactive request to overtake, and a row whose
estimate is still in flight says "sizing…" rather than leaving a blank where a
number will appear.
buildEstimateInput moves to core/gallery as EstimateInput, since the handler
and the warmer both need it.
Measured against 1,595 models, from a cold boot:
page 1, 30 estimates in parallel 10s -> 0.04s
full warm-up (299 of 300 entries) 3m, in the background
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* chore: untrack data/.local_user_id and ignore the runtime data dir
`local-ai run` writes its instance state under ./data when started from the
repo root, which is exactly what a contributor testing a build does. The
identity file ended up committed on this branch by a `git add -A` while
verifying the gallery changes against a live instance.
Anchored, so it matches the runtime directory at the repo root and not a
`data` directory nested inside some package.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Add the recommended Q4_K_M build and an MTP-enabled variant with the shared vision projector. Tag the existing Qwythos 9B MTP entry so serving-feature ranking recognizes it.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* chore(deps): bump cogito to v0.11 ahead of the nib harness
nib is the agent harness that becomes 'local-ai chat'. It requires cogito
v0.11, so pull that bump forward on its own: minimal version selection would
apply it to LocalAI anyway, and both repos use cogito and cogito/clients.
Landing it separately keeps the harness change reviewable.
nib itself is not pinned yet. Nothing in LocalAI imports it, and 'go mod
tidy' runs as a goreleaser before-hook in CI, so an unimported require line
does not survive. It lands with its first importer.
No LocalAI call site needed a change. Both cogito.WithMaxAttempts callers
guard the argument above zero, so v0.11's new clamp is unreachable, and
LocalAI's Multimedia values implement only URL(), so v0.11's new
TypedMultimedia routing treats them as images exactly as v0.10 did.
Binary size (cmd/local-ai): 200,301,381 -> 200,336,045 bytes (+34,664).
A throwaway probe that links nib measured 201,042,243 bytes (+740,862 over
the pre-change baseline).
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): resolve and seed the agent state directory
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): write the agent config atomically and tighten its modes
Replacing config.yaml in place truncated it first, so an interrupted write
would have destroyed the api_key nib keeps in the same file. Stage through a
sibling temp file and rename over the target instead, and match nib's 0700
directory mode.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): probe the endpoint and classify failures
Probe lists what a LocalAI endpoint advertises and separates the two
failures that need different advice: nothing listening, and rejected
credentials.
go-openai reports a rejected key as one of two concrete types depending
on the error body, and both occur against a real LocalAI. The normal
error handler sends an OpenAI error envelope, which arrives as
*openai.APIError; the opaque-errors handler replies with a bare status
and no body, which arrives as *openai.RequestError. Classifying on only
one of them misses half the cases, so the status is read from either.
A cancelled probe is not reported as an unreachable server, because it
learned nothing about the endpoint, and neither is a reply that could
not be parsed, because something did answer. Both would otherwise send
the user off to start a server that may already be running.
The model list is returned verbatim and in server order. LocalAI lists
whatever it finds in the models directory, including stray archives and
dotfiles, and deciding which advertised ids are real belongs to whoever
presents them.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): resolve the model from flag, config, or the server
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(chat): pin that model resolution sorts a copy of the caller's slice
The sort spec asserted only on what the chooser was offered, so replacing the
defensive copy with an in-place sort of req.Available still passed all 37
specs. Assert the input slice's order after the call, so the guarantee cannot
be dropped silently by a later refactor.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): offer to start a server when none is reachable
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): bound the server wait and pin readiness and stop semantics
Set cmd.WaitDelay so a backend subprocess holding the child's stderr pipe
cannot block cmd.Wait forever, which would leave exited unclosed, burn the
whole shutdown grace on a clean exit, and leak the waiter goroutine.
Two test gaps closed alongside it: the readiness spec now counts polls, so
treating 503 as ready is observable, and Stop's single-interrupt contract is
pinned by giving StartedServer interrupt/kill hooks that a spec can count.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactor(chat): drive Stop through one process interface, hide exec plumbing
Two independent interrupt/kill func fields plus a nil check admitted wirings
no test could distinguish: the pair swapped, so a SIGKILL would strand the
backends SIGINT exists to let local-ai run clean up, or kill left nil, so a
wedged server never escalates. One two-method interface that *os.Process
already satisfies leaves nothing to swap and nothing to nil.
Also translate exec.ErrWaitDelay, whose text names an os/exec struct field,
into what the user can act on. os/exec only substitutes that sentinel when the
process exited without an error of its own, so no exit status is swallowed.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): replace the REPL with the built-in agent
local-ai chat is now the nib agent harness compiled into the binary: tool
use behind an approval gate, sub-agents, MCP, plugins, and skills, all
auto-configured against the local server.
The REPL goes with it. Its model listing and its 401 classifier were
duplicates of the ones Probe now owns, and the classifier was the version
that misreads a bare 401 with no OpenAI error envelope, so keeping either
would leave the package with two divergent answers to the same question.
github.com/mudler/nib lands in go.mod in this commit rather than earlier:
go mod tidy runs as a goreleaser before-hook on every PR, so a require
line with no importer is stripped before it reaches CI.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactor(chat): split the pre-agent phase out of Run and pin it
Everything before the handoff is testable and nothing after it is: once
app.Run owns the terminal there is no seam left. prepare draws that line,
takes interactivity as a parameter so the prompts can be driven over a
pipe, and hands Run the state dir, the model, and any server it started.
The questions move onto one prompter that owns its buffered reader. A
fresh bufio.Reader per question reads ahead and discards what it buffered,
so the model choice typed behind an answer to "start a server?" was lost
and the next question saw EOF.
choose answers with a list index and refuses an empty offer, so a value
that was never on the list cannot reach ResolveModel, which persists it
and starts every later run against it.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): bound each server check with a deadline
Nothing bounded the model listing, so pointing chat at an address that
accepts the connection and then never replies left the user with no
output and no offer to start a server.
The budget is context.WithTimeout rather than a cancel plus a timer.
Probe deliberately refuses to call an endpoint unreachable on a
context.Canceled, since a caller who gave up learned nothing about the
server, and only honours a deadline. A cancel-based budget therefore
expires as the one error that suppresses ErrUnreachable, exactly for the
hung servers the offer exists to rescue.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): tell the user when their model choice cannot be saved
The choice is meant to be asked for once. When saving it fails the user
is silently asked again on the next run, and the only trace was an
xlog.Warn: the agent runs at log level error, and a --log-level=error run
swallows it entirely.
ModelRequest gains Notify for exactly this class of problem, one that is
worth telling the user about but not worth failing over, and the chat
wiring points it at the same writer the question was asked on.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): stop a session's server when the process is signalled
A server started for the session is stopped by a deferred call, and a
signal skips deferred calls: a SIGTERM between the spawn and the exit
left 'local-ai run' reparented to init with nothing left that knew to
shut it down. Ctrl+C was already safe, but only incidentally, because the
child shares this process' foreground process group.
A signal handler rather than Pdeathsig on the child. Pdeathsig is
Linux-only and, in Go, is delivered when the OS thread that forked exits
rather than when the process does, so it can fire on a healthy parent.
Setpgid would break the Ctrl+C that works today by taking the child out
of the foreground group.
SIGHUP joins SIGINT and SIGTERM: a terminal program whose terminal is
gone has nobody left to talk to. The same context is what cancels the
agent, which nib leaves to its embedder.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): only skip the server checks for work that stays local
Two argument shapes were classified wrongly. Every 'mcp ...' invocation
counted as management, so 'local-ai chat mcp --stdio', which serves the
agent over MCP and needs a model like any other session, was handed an
empty one. And --init, whose shell snippet a user pastes into an rc file
long before any server exists, went the other way: it demanded a running
server to print a static string.
The mcp split is asked of nib's own IsMCPManageSubcommand rather than
restated here, so a verb added upstream cannot drift out of this list.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): exit with the agent's status instead of reporting it twice
nib writes what went wrong to stderr and returns nothing but an exit
code, so returning that error unchanged had main log "Error running the
application error=exit status 1" underneath the message the user had just
read. The refusal to render the full-screen interface into a pipe is the
one they meet in practice: it names --cli, and burying that hides the fix.
ExitCodeError says "already reported, exit with this status". main
honours it and prints nothing more, so a piped or redirected chat still
fails a script the way it should.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* style(chat): route interactive chatter through one writer helper
The prompts and notices all write to a terminal, where a failed write is
not worth failing the session over and the read that follows the question
reports the real problem. say says that once instead of five discarded
error returns.
The command's one-line help comes along: chat is no longer "an
interactive chat session".
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(chat): record why the agent gets this process' streams
Injecting them is what makes nib refuse to draw its full-screen interface
into a pipe and name --cli, instead of rendering onto a terminal the
caller may not own. The tradeoff is worth stating where the wiring is.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): stop the session's server on cancellation, not on the way out
The deferred Stop is reached only if the agent returns, and cancelling
the context does not make it: nib hands the TUI to bubbletea without the
context, so what actually unwinds a running session today is bubbletea's
own SIGINT and SIGTERM handler. SIGHUP has no such backstop, and
registering for it removed the default disposition that used to end the
process outright, so kill -HUP left a live TUI with a cancelled context
and the started server still running.
runSession watches the context alongside the agent and stops the server
the moment it is cancelled, so the guarantee no longer depends on what
the agent does with cancellation. Stop is idempotent, so the deferred
call stays correct and free.
The doc comment on shutdownContext described the mechanism it was
supposed to work by rather than the one that does. Corrected, bubbletea's
handler included.
ResolveModel now checks the chooser's answer against what it offered.
The shipped chooser answers by list index and cannot be wrong, but
ModelChooser is exported, the answer is persisted, and every later run
starts against it, so the invariant belongs at the consumer.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(chat): bump nib to v0.5.1
v0.5.1 carries four fixes that matter to 'local-ai chat':
- --init now names the embedder's command, so the emitted widget invokes
'local-ai chat' rather than a bare 'nib' the user does not have.
- A piped CLI session that succeeds exits 0 instead of failing with EOF.
- EOF at a tool-approval prompt denies the call rather than approving it,
and the session exits 3 (app.ExitCodeApprovalNoInput) so a script can tell
"answered" from "refused to act" without reading stdout. Read-only tools
are unaffected and still run. ExitStatus already unwraps app.ExitError,
so the code propagates with no change here.
- RunTUI passes the context to bubbletea and gives up bubbletea's own signal
handler, which makes shutdownContext the single owner of the signal and
stops a SIGHUP leaving a wedged TUI behind.
Verified against a live server on 127.0.0.1:8080: the three --init shells,
a piped prompt exiting 0, a denied 'touch' that left no file and exited 3,
a read-only 'ls' that still ran and exited 0, and a SIGHUP that unwound a
TUI running under a pty.
Two comment blocks in run.go described the old TUI behavior and are now
wrong, so they are corrected in the same change. No behavior change: both
shutdownContext and runSession are untouched, and stopping the server on
cancellation is still worth keeping independent of how promptly nib unwinds.
One known gap, not addressed here. The widget --init now emits runs
'output=$(local-ai chat --height 50%)', and runAgent injects Stdout
unconditionally, so under $(...) nib refuses the TUI for a non-terminal
stream. This is the cost the runAgent comment already anticipated, now that
the snippets no longer hardcode standalone nib. Ctrl+Space should not be
documented until that is decided.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): let nib own stdout, so the Ctrl+Space widget works
The widget 'local-ai chat --init' emits runs
'output=$(local-ai chat --height 50%)', which puts a pipe on stdout by
construction. runAgent injected os.Stdout unconditionally, and nib refuses
every mode but --cli when a stream it was handed is not a terminal, so
Ctrl+Space printed "Re-run with --cli to use the injected streams" and
inserted nothing. Verified against a pty before and after.
nib reads a nil stream as "not injected" and falls back to the process
stream, which is how an embedder asks for nib's own behavior. That is what
stdout needs: the interface renders on /dev/tty but writes the chosen
command to stdout even when stdout is a pipe, and that write is the whole
of the shell-capture idiom.
Stdin is deliberately left injected. A piped or redirected stdin really is
ignored by the interface, so the refusal is the honest answer there, and it
is the one users meet: 'echo q | local-ai chat' still says to re-run with
--cli, once, exit 1. Nilling stdin the way stdout is nilled would delete
that silently. Stderr is not gated by nib at all and is unchanged.
One case does change and cannot be kept: 'local-ai chat > out.txt' from a
terminal no longer refuses, because it is indistinguishable from the
widget. It renders on /dev/tty and writes the capture line to the file,
which is what standalone nib does.
The app.Options literal moves into agentOptions so the decision is
reachable from a spec rather than being a detail of a function that takes
the terminal. Both sides of the asymmetry are pinned: reinstating
'Stdout: opts.Out' fails "hands nib nothing for the process stdout", and
nilling stdin fails "hands the process stdin over".
Also rewrites the last comments describing the pre-v0.5.1 behavior.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(chat): say what the stream refusal actually keys on
Two comments still called it the refusal to render the interface "into a
pipe". That was true when both stdin and stdout were injected, but a pipe on
stdout no longer refuses, so the wording now points at precisely the case
that was un-refused to make Ctrl+Space work. Only a stdin that cannot be
read triggers it, and both comments now say so and name the command a user
meets it with, 'echo q | local-ai chat'.
The agentOptions doc also said a "file a caller chose" stays injected and
refused, which reads as though 'local-ai chat > out.txt' still refuses. It
does not: a shell redirect arrives as os.Stdout and is nil-ed like the
widget's pipe, because the two differ only in being a regular file rather
than a FIFO and nib's gate does not look at that. What stays injected is a
writer an in-process caller chose for itself. Says that now, in the doc and
in the spec comment that had the same ambiguity.
Comments only. No behavior change.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: local-ai chat is now the built-in terminal agent
`local-ai chat` was a plain chat prompt and is now an agent that runs
shell commands behind an approval gate, so the pages that described a
REPL were wrong rather than merely thin.
Adds a Terminal agent feature page at /features/terminal-agent covering
the approval gate, piped runs and their exit codes, Ctrl+Space, model
resolution, state directory, and the pass-through management commands
(including the `--yes` caveat that leaves a plugin installed but
disabled in a script).
The three-way "looking for something else" notice becomes four-way and
moves into an agentic-routing shortcode. Four hand-kept copies of the
same paragraph is what produced the drift the new page would otherwise
have added to; the shortcode takes `current=` so each page still marks
itself, and errors the build on a name that is not one of the four.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* website: the agent is in the binary, not a second install
The nib section sold a separate tool you also install, with a GitHub
link as the only way in, which is now the wrong order: the agent ships
compiled into local-ai, and the standalone binary is the second reason
to care rather than the first.
Leads with `local-ai chat`, keeps nib as the SSH-anywhere story, and
adds a docs CTA pointing at the new Terminal agent page. id="nib" is
left alone because localai.io/#nib is linked from outside.
The two credits on the demo clip named nib as the thing that drove the
machine; they now credit the agent in LocalAI, which is the same agent.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* website: fix the exit keys, the plugin warning, and the redirect gap
Three claims on the chat-agent pages that the code does not back.
try-it-out told readers to press Ctrl+D. nib has no Ctrl+D handler: the
full-screen interface quits on Esc or Ctrl+C, and Ctrl+D is only an exit
in --cli, where it arrives as ordinary tty EOF. That sentence had
replaced the removed /exit and /quit text, so the page was left with no
working way to leave a session. Document both modes, since they differ.
The plugin warning said nothing tells you the install stopped short. It
does: the command prints that the plugin was left disabled. What it does
not do is say so in its exit code, which is 0 either way. That is the
part a script cannot work around, and it is the reason to pass --yes.
Overstating it in the paragraph that gives the advice only makes the
advice easier to dismiss.
Redirecting stdout no longer refuses; the interface goes to /dev/tty and
only the yanked command reaches the file. It is what lets the Ctrl+Space
widget capture a command at all, since a redirect and out=$(...) are the
same thing to the stream gate. It was documented nowhere. A non-terminal
stdin is still refused, and the new text says which of the two it is.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): make the CLI flags outrank the agent config file
local-ai chat routed --endpoint, --model, --api-key, --trace-dir and --yolo
through nib's app.Options.Defaults. Defaults are seeds: they sit beneath the
config file, so the file silently undoes them. That made the flags accepted and
inert, and not in an edge case, since EnsureStateDir writes base_url on the
first run and the interactive picker writes model, so from the second run on
the file carried a value for both.
Observed against a live server: with base_url: http://127.0.0.1:9999/v1 in the
config and --endpoint http://127.0.0.1:8080 on the command line, the probe hit
8080 and every agent turn posted to 9999. With model: gemma-4-e2b-it-qat-q4_0
in the config, --model lfm2.5-8b-a1b was ignored on the wire.
nib v0.6.0 adds app.Options.Overrides, applied above the config file and above
the bare environment block. Move the whole block there: all five values are
decisions this invocation already made on the user's behalf, and a flag the
config file can undo is not a flag. Nothing is left in Defaults, because
LocalAI's one genuine seed, the initial base_url, is written into the config
file by EnsureStateDir rather than handed to nib.
Two limits come with the channel and are documented on agentOptions rather than
worked around. An override can only raise a field, since nib cannot tell "set
to the zero value" from "not set", so --yolo can turn approval off but nothing
on the command line turns it back on over an approval_mode: auto in the file.
And nib's own NIB_TRACE_DIR and NIB_YOLO are resolved after the config load and
still outrank these, deliberately, upstream.
The existing spec pinned that the right values reach app.Options, which they
always did, which is exactly why it could not see nib discarding them. The new
specs resolve the config the way app.Run resolves it, against a real config
file that disagrees with every flag, and one asserts Defaults stays empty.
docs/content/features/terminal-agent.md already documented --model as winning
over the saved model; that was false before this change and is true now, so no
docs edit was needed.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): document intentional config file read
Assisted-by: Codex:gpt-5 [gosec]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The post was written against the first draft of the release notes, when the
cycle stood at 214 PRs over thirteen days. It closed at 321 PRs over eighteen
days, and three of the larger user-facing changes landed after it was written.
- Correct the counts throughout: 321 PRs, eighteen days, 24 contributors
(11 first-time), gallery 1,221 to 1,505.
- Add sections for the three new capabilities: 3D generation as a modality
(Generate3D, FLAG_3D, /v1/3d/generations, trellis2cpp), audio.cpp serving
six audio endpoints from one process, and the operations bar becoming the
Activity page.
- Cover the two further hardening fixes (tar hardlink escape, cyclic $ref
stack overflow) alongside the TRL one.
- Note the Valkey store, systemd socket activation, persistent trace history,
in-place chat edits, the self-contained SYCL backend and the site split.
- Group the new-engine sections together rather than splitting them across
the operational ones.
Embeds the existing vllm-race and magpie clips, and adds a 3D generation clip
cut from the demo recording to the conventions in .agents/preparing-a-release.md
(no audio track, 14s, named for the feature). blog.css styled figure img but
not figure video, so a clip in a post rendered outside the card; both selectors
now share the rule.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
tests-qwen3-tts-cpp has been failing on master since 2026-07-31. The suite
loads every component fine and then stops: TTS() never returns from the
native call, so a job that takes ~5 minutes runs into the 20 minute Go test
timeout instead.
goroutine 74 [syscall, 19 minutes]:
github.com/ebitengine/purego.RegisterFunc.func4
qwen3-tts-cpp.(*Qwen3TtsCpp).TTS goqwen3ttscpp.go:154
qwen3-tts-cpp.init.func2.4 e2e_test.go:90
Not a flake: reproduced on master and again on an explicit re-run.
Bisected across this cycle's eight qwentts.cpp bumps by their own check:
10832, 10850, 10902, 10964, 11006, 11039 and 11127 all pass in ~5 minutes;
11241 (abab6b3) fails at 1h58m. That PR was merged with this check already
red, which is how the hang reached master.
35ebe537..abab6b3 is three upstream commits, and the only functional one is
26dd8adb, "predictor: unroll the frame into one cgraph and sample in standard
ops", which is consistent with a generation loop that never reaches its stop
condition.
Hold the pin at the last known-good commit. The bump entry is commented out
rather than left in place, because it tracks upstream master and would put
the hang straight back on the next nightly run. Both spots carry a pointer to
the other so the hold is discoverable, and restoring it is uncommenting four
lines once upstream is fixed.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Select the CPU_ALL_VARIANTS target for x86 GPU images so partial offload uses runtime-selected host kernels. Keep GPU arm64 builds on the portable fallback until their toolchains consistently provide gcc-14.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): drop merged MiniMax-M3 patch
The bumped llama.cpp revision includes the MiniMax-M3 parser and template detection, so the carried patch now rejects during backend preparation. Remove the obsolete patch while retaining the independent score-task patch.
Assisted-by: Codex:gpt-5 [systematic-debugging]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add the new 9B Fara computer-use model alongside its existing 27B sibling, with Q4_K_M and Q8_0 llama.cpp variants plus the required vision projector.
Assisted-by: Codex:gpt-5 [web]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Build the runtime CPU variant set alongside x86 GPU backends so partial offload uses the host's SIMD kernels instead of the scalar fallback. Keep arm64 GPU images on the portable binary until their builders consistently provide gcc-14.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(utils): reject tar hardlinks that escape the extraction root
ExtractArchive pre-scans archive members and rejects symlinks, but tar
hardlink entries carry a regular file mode and so pass that check.
Header.Linkname was never validated, so an archive could create a link
to a path outside the destination directory.
Validate Linkname with the same path check already applied to member
names. Hardlinks that resolve inside the extraction root still extract,
so ordinary archives are unaffected.
pkg/oci/image.go already resolves tar.TypeLink targets before using
them; this brings the archive extraction path in line with it.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Zelys-DFKH <zelys@dfkhelper.com>
* test(utils): cover hardlink overwrite and in-root hardlinks
The existing hardlink test names a link target two levels above the
extraction root, so its final assertion checked a path the link never
resolved to and could not fail. Point the target one level up instead,
at the path that assertion already names.
Add two cases. The first uses a .tar.gz, where ExtractArchive binds a
Tar config with OverwriteExisting set, and follows the link entry with a
regular entry of the same name. Before the fix that pair linked to a
file outside the root and then truncated it through the link, which the
plain .tar case does not reach. The second extracts a hardlink whose
target is an earlier member of the same archive, covering the claim that
ordinary archives are unaffected.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Zelys-DFKH <zelys@dfkhelper.com>
---------
Signed-off-by: Zelys-DFKH <zelys@dfkhelper.com>
Use the case-sensitive Hugging Face filenames and refresh the linked SHA256 values for both gallery variants.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Detect text-to-speech MLX repositories during model import and emit a TTS-ready mlx-audio configuration. Expose mlx-audio in the backend preference dropdown for repositories without complete metadata.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(sycl): make the intel llama.cpp backend self-contained on any host
The SYCL backend shipped an incomplete oneAPI runtime AND relied on a
host-provided GPU driver, so it only ran inside the build container. On a
bare host it died with "libze_loader.so.1 / libdnnl.so.3: cannot open
shared object file", and even with the host's Intel driver installed it
SIGSEGV'd during SYCL init when the host driver was built against a newer
glibc than the backend's bundled loader (rolling-release distros).
package_intel_libs now bundles the complete, coherent oneAPI runtime
(the missing MKL ILP64 / sycl_blas / tbb_thread + oneDNN + the dlopen'd
UR adapters, plus a sweep of the backend binaries' own direct deps) and
the Intel GPU userspace driver (libze_intel_gpu + libigdrcl + IGC + gmm)
with its OpenCL ICD manifest, mirroring how package_vulkan_libs bundles
Mesa. run.sh points the Level Zero and OpenCL loaders at the bundled
driver, and install-base-deps.sh installs it in the SYCL build image.
Bundling the driver is safe across kernels because it talks to the host
i915/xe via the stable DRM UAPI (unlike NVIDIA's kernel-locked
userspace).
Validated on Arch (glibc 2.43, i915): the backend loads and runs on an
Iris Xe with no host Intel packages installed.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
* fix(sycl): install a driver that exists, and let the user choose their own
The driver install added earlier in this branch asked apt for
intel-level-zero-gpu, which is not a package in Ubuntu 24.04. apt fails
outright on an unknown name, so neither driver was installed, nothing was there
to copy, and the images carried no driver at all.
It now comes from Intel's own repository, which has 25.18 for this Ubuntu
release, against 23.43 from late 2023 in the Ubuntu archive. The archive driver
does not know any card released since, so a machine with a recent Intel GPU
would end up carrying a driver that cannot drive it. Anything that goes wrong
during that install fails the build on purpose: an unreachable repository is a
passing problem that a retry fixes, while quietly carrying a different driver,
or none, is a difference nobody would notice until a user reports an idle GPU.
run.sh used to overwrite whatever driver the user had chosen. Level Zero uses
only the driver it is given, so on a machine with a card too new for the
carried driver, the GPU would go unused with no way back. Both that setting and
the OpenCL one are now left alone when already set, and the docs say how to
point a backend at the machine's own driver.
The OpenCL setting also used to be applied whenever the backend held a driver
list, even when the driver it named had not been copied, which leaves OpenCL
with nothing instead of falling back to the machine's own driver. It now
requires the copied driver to be present, and the packaging leaves out the list
entry of any driver it did not copy. The oneAPI images list a processor-only
OpenCL library, which was being carried with nothing behind it.
Two more corrections in the packaging. The scan for libraries a program is
linked against only looked at files named llama-cpp-*, so turboquant and bonsai,
which are also built for Intel GPUs, were left with the incomplete set of
libraries this branch set out to fix; it now looks at every program in the
directory. And a build that should carry a driver but ends up without one now
says so, which is what a stale prebuilt base image looks like: such a backend
still runs on a machine that has its own driver, so nothing fails and the only
other symptom is a user reporting an idle GPU.
Backends now also ask the driver to report how much graphics memory is free,
without which llama.cpp reads zero on an integrated GPU, since such a chip
shares the system memory instead of having its own. turboquant and bonsai get
the same run.sh handling as llama.cpp.
The driver is only carried by the builds that start through run.sh, because
run.sh is what points Level Zero and OpenCL at it. The Python backends for
Intel GPUs start differently and would never load it, so they keep using the
machine's own driver rather than carrying several hundred megabytes they cannot
use.
Checked in a container on Ubuntu 24.04: the install brings driver 25.18 with
the files where the packaging expects them, an unreachable repository fails the
build, and the copied set resolves on its own once the machine's Intel packages
are moved away.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
* fix(ci): rebuild every Linux backend when the GPU packaging script changes
scripts/build/package-gpu-libs.sh decides which GPU libraries end up inside an
image. The filter that builds the backend matrix listed it as an input of the
Python images only, so changing it rebuilt no Go and no C++ backend, even
though those run it from their own package.sh. A packaging fix aimed at the
Intel llama.cpp backend could merge and reach no image, which is the same
failure this rule was written to prevent.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
* fix(sycl): carry only the driver Level Zero uses, not the OpenCL one
llama.cpp reaches an Intel GPU through Level Zero, which hands the driver
programs that are already compiled and so needs only the back end of the
graphics compiler. The OpenCL driver can be handed source code instead, so it
needs the compiler's front end as well, and that arrives with its own copy of
clang. Carrying it cost about 139 MB in every backend built for Intel GPUs, and
took the carried set from 123 MB to 261 MB.
Nothing here takes that path. No LocalAI code selects an OpenCL device, each
backend image holds one backend, and the documentation never described OpenCL
as a way to run models: the only mentions are a stale clblas row in the
BUILD_TYPE table, for a llama.cpp backend that no longer exists and that no
build matrix entry uses, and the sycl-ls troubleshooting hint. Before this
branch the packaging carried the OpenCL loader and adapter but no driver, so
the path could not work in a released image either. There is nobody to keep
working.
The driver list that OpenCL reads is no longer carried, and run.sh no longer
sets OCL_ICD_VENDORS, so OpenCL inside a container keeps using whatever the
image provides rather than being pointed at a directory with no driver in it.
Checked in a container against the real 25.18 driver: the carried set is 123 MB
with nothing unresolved, and Level Zero still reports the GPU with the
machine's own Intel packages moved out of the way. Neither the Level Zero
driver nor the compiler back end names the front end or clang among the
libraries it opens by name, so the leaner set is complete for this path.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
---------
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
* feat(chat): edit saved conversation messages
Add inline edit, save, and cancel controls for stored user and assistant messages without triggering inference. Preserve structured message attachments and cancel edits when streaming starts.
Assisted-by: Codex:gpt-5
* test(chat): preserve seeded conversation on reload
The saved-message edit test reloads the page to verify persistence, but its init script was replacing localStorage with the original fixture on every navigation. Seed only an empty store so reloads exercise the data written by the application.
Assisted-by: Codex:gpt-5 [Codex]
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Direct repository-root users to the supported make docs target and document the equivalent direct Hugo invocation from docs/.
Fixes#10062
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The page reads as a list of features with nothing joining them, so two
things did not land.
The engines section never said these are the backends LocalAI loads. The
runtime section describes a core that pulls each engine in on demand, and
the engines section describes engines written from scratch, and nothing on
the page connected the two sentences. Readers were taking parakeet.cpp and
the rest for unrelated side projects by the same people. The lede now says
whose backends they are before it says anything else.
APEX was used as a known term on first appearance, in a section that opened
onto a benchmark table. Nothing said what it is or why it follows the
engines. It now opens by placing itself in the stack: the engine decides how
fast a model runs, the weights decide whether it runs at all, and APEX is
the second of those. Then the numbers.
Also drops "Most backends wrap somebody else's engine. These do not", which
is the machine-written antithesis shape, and fixes a list that broke its own
parallel halfway through.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m]
The star, fork, contributor and release counts were typed into the templates
by hand, so they only moved when somebody remembered. They had already
drifted: stars read 48,042 against 48,067, forks 4,314 against 4,320, and
contributors 224 against 225.
They move to website/data/stats.yaml, which .github/ci/refresh-site-counters.sh
rewrites from the GitHub API, run weekly by a new workflow. The contributors
and releases endpoints never report a total, so the script asks for one item
per page and reads the count out of the Link header. It refuses to write a
zero or a non-number, which is what a rate-limited or failed call looks like,
and the workflow commits only when a number actually moved. The Discord count
has no API behind it, so the script reads the existing value back and carries
it through.
The engine count was wrong in a second way. The hero said 18, the section
heading said "Eighteen engines", the timeline said "Nineteen engines of our
own", and the /engines/ page derived 19 from the data file. All of them now
derive from that same file, so they cannot disagree again.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m]
Add Q4_K_M and Q8_0 llama.cpp entries for the newly released Qwopus3.6-27B Fusion reasoning and coding merge, with MTP enabled.
Assisted-by: Codex:gpt-5 [Hugging Face API]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The timeline set six 15rem columns in a flex row with overflow-x:auto, which
needs 90rem and so scrolled sideways on any normal laptop. It is a wrapping
grid now, and the rule that carries the dots moves from the container onto
each item so a wrapped row still gets a line above it. Column gap is zero and
the items carry their own right padding, so the rule stays continuous.
Integrations move from a card grid to a reel. Any single integration is a weak
signal and the whole moving line is the strong one, so the count is doing the
argument. It pauses on hover and on keyboard focus, since the names are links.
The list grows from 8 to 26: Open WebUI, Dify, LibreChat, RAGFlow, Continue,
big-AGI, Nextcloud, Frigate, promptfoo, Mods, TypingMind, baibot, k8sgpt-
operator and others. Each was admitted only after opening that project's own
repository or docs and reading the line that names LocalAI. The ones that
failed that test are listed in the data file so nobody re-adds them.
The blog cards were hand-written, which is how one of them came to advertise
"Porting vLLM to C++", a post that does not exist, and how all three linked to
the blog index instead of an article. They range over the posts now.
The section intro used the "a changelog tells you what moved, these posts show
you what it does" shape, which is the standard machine-written antithesis. It
states what the posts contain instead, including the perplexity regression
that APEX costs, because publishing the price is the actual claim.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m]
The band led with three sentences of hedging and printed a commit count
next to each employer, so a one-commit entry beside a large name read as
weakness rather than as the modest, true claim it was. It now opens on the
contributor count, sets the employers as a sentence instead of a pill wall,
and keeps the caveat to one line. The counts stay in ecosystem.yaml, since
they are the provenance for the list and anyone re-checking it needs them.
Two "coverage" cards were not about this project. The modelslab.com piece
reviews Frikallo/parakeet.cpp, an unrelated project of the same name, and
the snailtext.app benchmark measures Parakeet through ONNX Runtime without
mentioning LocalAI at all. Both are removed, along with the contributor
card that duplicated the band's opening line.
Press was four posts from one vendor, which read as the whole of the
coverage rather than one enthusiastic outlet. SUSE collapses to a single
series entry, and Pulumi, Semaphore and Spectro Cloud join it. Each was
opened and checked against the project before being added. K8sGPT and
LlamaIndex join the integrations; both document LocalAI as a backend.
The quotes move above the lists so the section opens on its strongest
line, which is somebody else's. The hero gains a GitHub call to action,
the APEX collection link was returning 404 and is corrected, and the
footer no longer describes the site as a design mock.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m]
Follow-up to 42541dd4f, which routed lint to arc-runner-set. Both of its
jobs failed there in one second (run 30637392862): the runner image has
git, curl, unzip, tar, ldd and python3, but not make, and build-scripts
additionally needs gcc because the packaging-script tests compile a
throwaway binary and inspect it with ldd.
golangci-lint needs make twice over: `make protogen-go` (which also wants
curl + unzip to fetch protoc) and `make lint` itself. So both jobs go back
to ubuntu-latest.
gh-pages.yml stays on arc-runner-set and is unaffected: it uses no make and
no C toolchain, and setup-go / actions-hugo fetch their own toolchains.
The preflight steps stay. They cost about a second on the hosted pool, and
they are what turned this into a one-second named failure instead of an
opaque one midway through a build. When the runner image gains make + gcc,
re-routing is one runs-on line per job. Any such re-route must stay
push-only: lint also runs on pull_request, and fork PRs execute untrusted
code that must not reach a persistent self-hosted runner.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
The GitHub-hosted runner pool is shared per ACCOUNT, not per repo, so a
burst in one repo starves every other. On 2026-07-31 it reached zero
scheduled jobs for 35 consecutive minutes with 39 jobs queued, while
arc-runner-set completed 12 jobs without interruption across the same
window. Actions was healthy globally (other public repos were scheduling
normally), so this is an account-level throttle we cannot fix from inside
the workflows, only route around.
Site publishing and lint are small, run on nearly every commit, and gain
nothing from waiting behind a saturated hosted queue, so both move to
arc-runner-set, the label already proven in generate_intel_image.yaml.
lint.yml is routed for PUSH ONLY, and this is the important part: that
workflow also triggers on pull_request, and a fork PR executes untrusted
contributor code. Running that on a persistent self-hosted runner would be
a real compromise vector, so anything that is not a push to mudler/LocalAI
stays on the ephemeral hosted pool. gh-pages.yml needs no such clause: it
triggers only on push-to-master and workflow_dispatch, so it never runs
pull-request code. Both carry a repository guard so forks, which have no
such runner label, fall back to hosted instead of queueing forever.
Neither workflow uses sudo or apt, and both fetch their own toolchains via
setup-go / actions-hugo. A self-hosted image can still be leaner than the
hosted one, so each lint job opens with a preflight that names the missing
tool (curl/unzip/make for protoc and lint; gcc/ldd/python3 for the
packaging-script tests) rather than failing opaquely mid-build. Reverting
is one runs-on expression per job.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed backend verification performs separate registry requests for manifests and referrers. Reuse LocalAI's version-aware User-Agent there so the full install flow is attributable to LocalAI.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Adds the "who turns up around this project" section, split into three lists
because the evidence behind each one is a different strength and collapsing
them into a single logo wall would overclaim.
Contributors 21 companies whose engineers have commits here. Evidence is
the commit history plus the employer on that person's public
GitHub profile, so it is a claim about the person. Commit
counts are shown next to each name, including the ones that
are a single patch, because hiding that would be the whole
problem.
Integrations six projects that reference LocalAI in their own repository
or documentation, which anyone can verify without asking us.
Press four SUSE Communities articles about running LocalAI.
Names are set in type rather than fetched as logos. A logo reads as
endorsement, and a one-line typo fix from somebody who happens to work at a
large company does not support that, quite apart from what their trademark
policy says about it.
ADOPTERS.md is the mechanism for the stronger claim. An organisation that
wants to be listed as a user opens a pull request adding itself, which is both
the evidence and the permission, and is publicly auditable afterwards. The
file says plainly what the website does and does not claim, so the next person
to ask "can we add some big names" has the answer in the repository.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] [WebSearch]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(website): split the site, move docs to /docs, add a landing page
The Hugo docs site has always been localai.io itself, which left nowhere to
explain what LocalAI is or show what the team builds. This adds a separate
marketing site at the root and moves the documentation under /docs/.
Docs:
The existing site keeps its content tree and its Relearn theme, and now
builds with baseURL <root>/docs/. Its _index.md, which held a hand written
landing page, becomes a real documentation home.
Every previously published URL keeps working. GitHub Pages has no server
side rewrites, so .github/ci/gen-redirects.sh walks the built docs output
and leaves a meta refresh plus a canonical link at each old root path. It
covers bare .html files too, which is what keeps /gallery.html alive, and
it never overwrites a path the marketing site already owns.
Website:
A second Hugo site under website/ with its own layouts and no external
theme, so the marketing side does not have to fight Relearn's home rooted
menu and asset pipeline. CI builds both and merges them into one Pages
artifact.
The design is derived from the project logo rather than invented: the navy
of the triangle, the cyan of the llama, the purple of the speed bars. Those
offset bars became the motion signature. The background renders a real
depth-anything.cpp depth map as contour lines and switches to a
locate-anything.cpp style detection overlay over the engines section.
Also included: an /engines/ index driven entirely by data/engines.yaml, a
/blog/ section with five posts written from the release notes and the
engine benchmark suites, install.sh and a Kubernetes manifest since the
site advertises both, and a rule in .agents/ that release preparation now
includes a blog post and demo clips.
Every figure on the site is derived from the repository or the GitHub API,
not from memory. Correcting them against their sources found one error in
README.md: voxtral-tts.c is text to speech, not speech to text.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] [Agent]
* feat(website): add a star history chart, rewrite the history post in first person
The history post read like a changelog written by a committee. It is now in
Ettore's voice, first person, with the admissions left in.
The numbers paragraph in particular read like a directory listing. It now says
what the figures mean rather than which file they came from.
Adds an interactive star history chart, built from the GitHub stargazers API
rather than embedded from a third party, so the page makes no external request
and cannot break when someone else's service is down. The four releases the
post is organised around are marked on the curve, and the labels stack into
rows because three of them land within two months of each other.
The API stops paginating at 40,000 items, so the curve is measured up to
December 2025 and the segment from there to today's total is drawn dashed,
labelled as an estimate in the caption and in the tooltip. It is a straight
line between two known points, and the chart says so rather than implying it
is data.
Also drops "marketing site" from the README heading and everywhere else it
appeared, and calls it the main site instead.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
2026-07-31 09:00:56 +02:00
383 changed files with 21613 additions and 3644 deletions
A new capability deserves its own page under `docs/content/features/`, plus cross-links from related features and an entry in `docs/content/whats-new.md`. See the pattern used by `face-recognition.md` / `object-detection.md`.
A new capability deserves its own page under `docs/content/features/`, plus cross-links from related features. See the pattern used by `face-recognition.md` / `object-detection.md`.
Announcing it is the release's job, not this page's: the capability gets covered in the release blog post under `website/content/blog/`. See [preparing-a-release.md](preparing-a-release.md). `docs/content/whats-new.md` is only a pointer at the blog and GitHub Releases, so there is nothing to add there.
## Path protection rules
@@ -334,7 +336,7 @@ When adding a new endpoint:
- [ ] Swagger block on the handler: `@Summary`, `@Tags`, `@Param`, `@Success`, `@Router`
- [ ] If new capability area (new swagger tag): entry in `instructionDefs` in `core/http/endpoints/localai/api_instructions.go` + test count bumped in `api_instructions_test.go`
- [ ] If new `FLAG_*` usecase flag: matching `CAP_*` symbol exported from `core/http/react-ui/src/utils/capabilities.js`
- [ ]`docs/content/features/<feature>.md` created; cross-links from related feature pages; entry in `docs/content/whats-new.md`
- [ ]`docs/content/features/<feature>.md` created; cross-links from related feature pages; capability covered in the release blog post (see [preparing-a-release.md](preparing-a-release.md))
**Quality**
- [ ] Error responses use `schema.ErrorResponse` format (or `echo.NewHTTPError` with a mapped gRPC status — see the `mapBackendError` helper in `core/http/endpoints/localai/images.go`)
@@ -125,7 +125,7 @@ The per-backend prefix match only sees files under a backend's own directory, so
| `backend/backend.proto` | nothing if the edit is additive-only, otherwise everything (see below) |
| `backend/Dockerfile.<x>` | the Linux entries whose `dockerfile:` names it |
| `backend/python/common/` | Python, Linux + Darwin |
| `scripts/build/package-gpu-libs.sh` | Python, Linux only |
| `scripts/build/package-gpu-libs.sh` | every Linux entry (Python, Go and C++ all run it) |
| `scripts/build/<lang>-darwin.sh` | the Darwin entries that build target routes to |
| `.github/workflows/backend_build[_darwin].yml` | everything on that OS |
| anything else under `scripts/build/` (except `*_test.sh`) | everything — conservative default for unclassified packaging inputs |
@@ -359,6 +359,26 @@ GitHub Actions caches are limited to 10 GB per repo. Steady-state worst case: ~8
One residual self-hosted reference remains in `test-extra.yml` (`tests-vibevoice-cpp-grpc-transcription` uses `bigger-runner` for the 30s JFK-decode timeout headroom). That's a separate concern.
### Small always-on jobs routed to `arc-runner-set`
The hosted pool is shared across the whole *account*, not per repo, so a burst in one repo starves the others. On 2026-07-31 it went to **zero scheduled jobs for 35 consecutive minutes** with 39 jobs queued, while `arc-runner-set` completed 12 jobs without interruption over the same window. Actions was healthy globally at the time (other public repos were scheduling normally), so this is an account-level throttle, not an outage.
`gh-pages.yml` (`build` + `deploy`) is therefore routed to `arc-runner-set` when `github.repository == 'mudler/LocalAI'`. It needs no fork-safety clause because it only triggers on push-to-master and `workflow_dispatch`, so it never executes pull-request code. The repository guard keeps forks (which have no such runner label) from queueing forever. It fetches its own toolchains via `setup-go` / `actions-hugo` and uses no `sudo`/`apt`.
#### What the `arc-runner-set` image actually contains
Measured 2026-07-31 on run `30637392862` by a preflight step, not assumed:
That is why `lint.yml` is **not** on the self-hosted pool. Both of its jobs were routed there and both failed in one second: `golangci-lint` needs `make` (for `make protogen-go`, itself needing `curl`+`unzip` to fetch protoc, and for `make lint`), and `build-scripts` additionally needs a C toolchain because the packaging-script tests compile a throwaway binary and inspect it with `ldd`. Both jobs are back on `ubuntu-latest`.
The preflight steps were deliberately left in place. They cost about a second on the hosted pool and mean that whenever the runner image gains `make` + `gcc`, re-routing is one `runs-on:` line per job and any remaining gap reports itself by name rather than as an opaque mid-build failure.
Note for any future re-route: `lint.yml` also triggers on `pull_request`, and a fork PR runs untrusted contributor code. That must never reach a persistent self-hosted runner, so any re-route has to stay push-only, e.g. `${{ (github.event_name == 'push' && github.repository == 'mudler/LocalAI') && 'arc-runner-set' || 'ubuntu-latest' }}`.
## Touching the cache pipeline
When changing `image_build.yml`, `backend_build.yml`, any of the `backend/Dockerfile.*` files, `Dockerfile.base-grpc-builder`, `.docker/install-base-deps.sh`, `.docker/<backend>-compile.sh`, or `scripts/changed-backends.js`:
A release is not finished when the tag is pushed. The GitHub release, the blog post and the demo clips ship together, because the changelog says what moved and the post and the clips are what make anyone care.
## What a release must include
1.**Labels on the merged PRs.** GitHub generates the raw notes from PR labels, so label first, generate second. Wrong labels mean a miscategorised changelog that has to be edited by hand.
2.**`RELEASE_NOTES_vX.Y.Z.md`** at the repository root, in the house style: what changed, why it matters, PR numbers so people can read the diffs.
3.**A blog post under `website/content/blog/`.** One post per release, front matter with `title`, `date`, `author`, `category: "Release"`, `tags`, `summary` and `extracss: ["blog.css"]`. Cover the two or three changes that alter what a user does day to day, not the whole changelog, and link the PR numbers. See `website/content/blog/what-landed-in-localai-4-8.md` for the shape.
4.**Demo clips for the notable features.** Anything visible (a new backend, a UI change, a new endpoint, a measured speedup) gets a short screen recording. Put the file in `website/static/media/`, reference it from the blog post, and reuse it on the marketing pages where it fits.
A release without a post and without clips is incomplete, in the same way a user-facing code change without a docs update is incomplete.
## Clip conventions
- MP4, H.264, no audio track unless the feature is about audio. Keep them short (10 to 30 seconds) and loopable.
- Record the real thing. A clip from the engine's own benchmark suite or a real session, never a mockup.
- Where the change is a speedup, record both sides on the same machine on the same input, so the comparison is honest.
- Name the file after the feature, not the release (`vllm-race.mp4`, not `v4-8-demo.mp4`), so it stays reusable once the release is old.
- The marketing site plays clips with `muted loop playsinline preload="none"` and a `data-lazy` attribute, which the site's IntersectionObserver uses to play and pause them on scroll. Follow that pattern for anything you add.
## Order of work
Label the PRs, generate and edit the release notes, cut the draft release, record the clips while the branch is still fresh in your head, then write the post against the notes and the clips. Publishing the release and merging the post should happen on the same day.
The `creating-localai-releases` skill drives steps 1 to 3 and captures the React UI screenshots that go into the notes.
| [.agents/preparing-a-release.md](.agents/preparing-a-release.md) | Cutting a release: PR labels, `RELEASE_NOTES_vX.Y.Z.md`, the blog post under `website/content/blog/`, and the demo clips under `website/static/media/` |
## Quick Reference
@@ -42,6 +43,7 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
- **Docs (docs-with-code rule)**: When you change user-facing behavior (API endpoints, CLI flags, config keys, or features), update the corresponding page under `docs/content/` in the SAME change, not as a follow-up. A user-facing change without a matching docs update is incomplete. See also the documentation conventions in [.agents/coding-style.md](.agents/coding-style.md).
- **New API endpoints**: LocalAI advertises its capability surface in several independent places — swagger `@Tags`, `/api/instructions` registry, auth `RouteFeatureRegistry`, React UI `capabilities.js`, docs. Read [.agents/api-endpoints-and-auth.md](.agents/api-endpoints-and-auth.md) and follow its checklist — missing any surface means clients, admins, and the UI won't know the endpoint exists.
- **Admin endpoints → MCP tool**: every admin endpoint that an admin would manage conversationally (install/list/edit/toggle/upgrade) MUST also be exposed as an MCP tool in `pkg/mcp/localaitools/`. The LocalAI Assistant chat modality and the standalone `local-ai mcp-server` consume that package; drift between REST and MCP is a real risk. Read [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) — the `TestToolHTTPRouteMappingComplete` test fails until you wire the new tool and update the route map.
- **Releases ship with a post and clips**: a release is not done at the tag. It needs labelled PRs, `RELEASE_NOTES_vX.Y.Z.md`, a blog post under `website/content/blog/`, and a short demo clip in `website/static/media/` for each notable feature. See [.agents/preparing-a-release.md](.agents/preparing-a-release.md).
- **Build**: Inspect `Makefile` and `.github/workflows/` — ask the user before running long builds
- **Backend OS coverage**: a new backend must target every OS it can build for, not just Linux. `.github/backend-matrix.yml` has two matrices — `include:` (Linux) and `includeDarwin:` (macOS / Apple Silicon). Most C/C++/GGML and many Python backends build on Darwin too — wire the `includeDarwin` entry + `backend/index.yaml``metal:` entries, or say in the PR why an OS is unsupported. See the darwin checklist in [.agents/adding-backends.md](.agents/adding-backends.md).
- **Gallery variant ranking**: a gallery entry can declare `variants` (alternative builds of the same weights), and LocalAI ranks the ones a host can run by engine preference first, size second. A new backend that should be preferred on some hardware must be listed in `engineNamePreferenceRules` in `pkg/system/capabilities.go`; the sibling `backendBuildTagPreferenceRules` speaks build tags rather than engine names, and using the wrong table matches nothing without erroring. See [.agents/adding-backends.md](.agents/adding-backends.md).
@@ -161,7 +161,7 @@ local-ai run https://gist.githubusercontent.com/.../phi-2.yaml
local-ai run oci://localai/phi-2:latest
```
To test a running LocalAI server from the terminal, open an interactive chat session from another shell. Inside the prompt, `/models` lists installed models and `/model <name>` switches between them.
To work with a running LocalAI server from the terminal, start the built-in agent from another shell. It answers questions, reads your files and runs commands on your machine, asking you to approve anything that changes state. Inside a session, `/models` lists installed models and `/model <name>` switches between them. See the [Terminal agent](https://localai.io/docs/features/terminal-agent/) docs.
```bash
# Terminal 1
@@ -195,7 +195,7 @@ For more details, see the [Getting Started guide](https://localai.io/basics/gett
- **August 2025**: MLX, MLX-VLM, Diffusers, llama.cpp now supported on Apple Silicon
- **July 2025**: All backends migrated outside the main binary — [lightweight, modular architecture](https://github.com/mudler/LocalAI/releases/tag/v3.2.0)
For older news and full release notes, see [GitHub Releases](https://github.com/mudler/LocalAI/releases) and the [News page](https://localai.io/basics/news/).
For older news and full release notes, see [GitHub Releases](https://github.com/mudler/LocalAI/releases) and the [blog](https://localai.io/blog/).
## Features
@@ -238,7 +238,7 @@ Most backends wrap a best-in-class upstream engine. A handful of them are native
| [magpie-tts.cpp](https://github.com/mudler/magpie-tts.cpp) | C++/GGML port of NVIDIA's Magpie TTS Multilingual 357M: 22.05 kHz mono text-to-speech in 5 voices and 9+ languages, with the NanoCodec neural codec and tokenizer/G2P embedded in a single GGUF |
| [ced.cpp](https://github.com/localai-org/ced.cpp) | C++/GGML port of the CED audio-tagging models: sound-event classification (527-class AudioSet) over REST and the realtime API for live recognition |
| [voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp) | Speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion), replacing the Python speaker-recognition backend |
| [voxtral-tts.c](https://github.com/mudler/voxtral-tts.c) | Voxtral Realtime 4B speech-to-text in pure C |
| [voxtral-tts.c](https://github.com/mudler/voxtral-tts.c) | Mistral Voxtral-4B-TTS text-to-speech in pure C: 20 preset voices across 9 languages, 24 kHz WAV output, no dependencies beyond libc |
| [vibevoice.cpp](https://github.com/mudler/vibevoice.cpp) | Native port of Microsoft VibeVoice for TTS (voice cloning) and long-form ASR with speaker diarization |
- [Integrations & community projects](https://localai.io/docs/integrations/)
- [Installation video walkthrough](https://www.youtube.com/watch?v=cMVNnlqwfw4)
- [Media & blog posts](https://localai.io/basics/news/#media-blogs-social)
- [Blog: release write-ups, benchmarks and engineering notes](https://localai.io/blog/)
- [Examples](https://github.com/mudler/LocalAI-examples) — including the [realtime voice assistant demo](https://github.com/localai-org/localai-realtime-demo) (Go client for the Realtime API with tool calling)
python3 -m venv "$(MLX_VENV)"||{echo"vllm-cpp: python3 with venv is required to build the MLX provider; pass VLLM_CPP_MLX=off to build Metal without it" >&2;exit 1;};\
returnfmt.Errorf("model %q is not available. Run `local-ai models list`, install a model with `local-ai models install <model>`, or switch with `/model <name>`",model)
case403:
returnfmt.Errorf("model %q is disabled. Enable it from LocalAI settings or choose another model with `/model <name>`",model)
returnfmt.Errorf("model %q is not available. Run `local-ai models list`, install a model with `local-ai models install <model>`, or switch with `/model <name>`",model)
return"",errors.New(`no chat models are installed.
Install a model first, for example:
local-ai models list
local-ai models install <model>
local-ai run
Then start a chat session:
local-ai chat --model <model>`)
caserequested==""&&len(models)==1:
returnmodels[0],nil
caserequested==""&&len(models)>1:
varbstrings.Builder
b.WriteString("multiple models are available; choose one with --model:\n")
b.WriteString(formatChatModelList(models,""))
return"",errors.New(b.String())
case!slices.Contains(models,requested):
return"",fmt.Errorf("model %q is not available. Use `local-ai models list` and `local-ai models install <model>`, or pass an installed model with --model",requested)
// ChatCMD runs the built-in terminal agent. Everything after the first
// positional argument is forwarded to the agent verbatim, so its own
// subcommands (plugin, skill, mcp) and their flags work unchanged. LocalAI's
// own flags must therefore come first.
typeChatCMDstruct{
Modelstring`short:"m" help:"Model name to use. Defaults to the only model returned by the server when exactly one is available"`
Endpointstring`env:"LOCALAI_CHAT_ENDPOINT" default:"http://127.0.0.1:8080" help:"LocalAI server endpoint. The /v1 path is added automatically when omitted"`
APIKeystring`env:"LOCALAI_API_KEY,API_KEY" help:"API key to use when the LocalAI server requires authentication"`
Modelstring`short:"m" help:"Model to use. Defaults to the only model the server offers, or asks when there are several"`
Endpointstring`env:"LOCALAI_CHAT_ENDPOINT" default:"http://127.0.0.1:8080" help:"LocalAI server endpoint. The /v1 path is added automatically when omitted"`
APIKeystring`env:"LOCALAI_API_KEY,API_KEY" help:"API key to use when the LocalAI server requires authentication"`
ConfigDirstring`env:"LOCALAI_CHAT_CONFIG_DIR" help:"Directory holding the agent's config, plugins, and skills. Defaults to ~/.config/localai/chat" type:"path"`
TraceDirstring`env:"LOCALAI_CHAT_TRACE_DIR" help:"Write a session LLM trace (NDJSON) to this directory" type:"path"`
CLIbool`help:"Run in plain CLI mode instead of the full-screen interface"`
TUIbool`help:"Force the full-screen interface"`
Heightstring`help:"Run as an inline drop-down of this height, e.g. '40%'"`
Tmuxbool`help:"Run in a tmux split"`
NoTmuxbool`name:"no-tmux" help:"Never use a tmux split, even inside tmux"`
Initstring`help:"Print the shell integration script for Ctrl+Space (zsh, bash, or fish)"`
Yolobool`env:"LOCALAI_CHAT_YOLO" help:"Auto-approve every tool call without prompting"`
Args[]string`arg:"" optional:"" passthrough:"" help:"Arguments forwarded to the agent, e.g. 'plugin install <url>', 'skill list', 'mcp add'"`
RunRunCMD`cmd:"" help:"Run LocalAI, this the default command if no other command is specified. Run 'local-ai run --help' for more information" default:"withargs"`
ChatChatCMD`cmd:"" help:"Open an interactive chat session against a running LocalAI server"`
ChatChatCMD`cmd:"" help:"Run the built-interminal agent against a LocalAI server"`
FederatedFederatedCLI`cmd:"" help:"Run LocalAI in federated mode"`
ModelsModelsCMD`cmd:"" help:"Manage LocalAI models and definitions"`
BackendsBackendsCMD`cmd:"" help:"Manage LocalAI backends and definitions"`
@@ -30,6 +30,12 @@ var instructionDefs = []instructionDef{
Tags:[]string{"inference","embeddings"},
Intro:"Set \"stream\": true for SSE streaming. Supports tool/function calling when the model config has function templates configured.",
},
{
Name:"moderation",
Description:"OpenAI-compatible text moderation using a local completion model",
Tags:[]string{"moderation"},
Intro:"POST /v1/moderations accepts a text string or array plus a LocalAI completion model. LocalAI constrains the model to the OpenAI moderation category schema and returns one result per input. Multimodal moderation inputs are not yet supported.",
// GetAPITracesSummaryEndpoint returns counted totals over a recent window
// @Summary Summarize recent API traces
// @Description Returns request, failure and latency totals over a recent window, plus a bucketed series for sparklines. Exists so callers wanting three numbers do not have to fetch the whole trace list and count it themselves.
// @Tags monitoring
// @Produce json
// @Param hours query int false "Window in hours (default 24, max 168)"
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
