The JetPack r36.4.0 row dies on the first line of NeMo-Speech.cpp's
CMakeLists.txt:
CMake Error at CMakeLists.txt:3 (cmake_minimum_required):
-- Configuring incomplete, errors occurred!
Upstream opens with cmake_minimum_required(VERSION 3.26). That base image is
Ubuntu 22.04 jammy, whose apt cmake is 3.22.1, so configure aborts before it
reads a single one of the backend's -D flags. Every other Linux row in this
block is noble, which ships 3.28 and clears the bar, so the failure is one
base image wide rather than a code problem. Everything before it on that row
had already worked, including the OpenFST and Sparrowhawk ITN build.
No other Go backend needs this. parakeet-cpp and moss-transcribe-cpp share
the same JetPack base and both declare cmake_minimum_required(VERSION 3.18),
and nothing in the repo installs a cmake newer than the distro's, so there is
no existing pattern to reuse. Nothing depends on jammy's cmake staying 3.22
either: build_itn_deps.sh never invokes cmake at all, since OpenFST and
Sparrowhawk are autotools builds.
Kitware's release tarball rather than their APT repo or pip. The tarball is a
pinned URL with a published checksum, so an upstream release cannot change
what lands here. The APT repo does carry jammy arm64, but it serves a moving
latest that today is CMake 4.4, and 4.x drops compatibility with
cmake_minimum_required below 3.5, which vendored third_party subprojects
still declare; pinning it there would mean tracking Kitware's Debian revision
string instead of an upstream version. pip would drag a Python toolchain into
a backend that has none. 3.31.12 is the last 3.x release, so it clears 3.26
while keeping the CMake 3 policy surface, and it stays close to the 3.28 the
green noble rows already use. The binaries need only glibc 2.17 and carry no
libstdc++ DT_NEEDED, well under jammy's 2.35. doc/, man/, ccmake and cmake-gui
are not extracted; the final image is FROM scratch, but there is no reason to
page 100 MB of Qt GUI and docs through the CI cache.
Gated on the installed cmake actually being older than 3.26, so the rows that
already build green keep configuring with exactly the cmake they use today,
and folded into the existing ${BACKEND} block rather than added as a new
instruction, so no other Go backend image gains a layer and nothing above the
Vulkan SDK, CUDA, Go and protoc layers moves.
The symlink lands in /usr/local/bin and shadows apt's cmake. Unlike the protoc
shadowing that broke Sparrowhawk earlier in this series that is inert: protoc
has to agree with the libprotobuf headers it generates against, whereas cmake
links nothing into the product and has no ABI relationship with anything in
the image, and it resolves the symlink back to /opt to find its own Modules/
tree, so a 3.31 binary can never read 3.22's modules.
The version test avoids $(...) deliberately. BuildKit delivers a RUN heredoc
through an outer shell with an unquoted delimiter, so a command substitution
runs there, too early, in a container where the files it reads do not exist
yet, and its empty output is pasted into the script; the first draft took the
install branch on every row because of it.
Verified by building the block against nvcr.io/nvidia/l4t-jetpack:r36.4.0
arm64 under qemu, the row's actual base image: cmake 3.22.1 detected, tarball
checksum verified, 3.31.12 installed, and a cmake_minimum_required(VERSION
3.26) project configures with -G Ninja and builds, with CMAKE_ROOT resolving
to /opt/cmake/share/cmake-3.31. Same on ubuntu:22.04 amd64 and arm64.
ubuntu:24.04 skips the install, gains no /opt/cmake and still configures on
/usr/share/cmake-3.28. The NeMo-Speech.cpp compile itself on JetPack CUDA 12
is not reproducible here and remains for CI.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The first WITH_NORM=ON build failed compiling fst_normalizer.cpp against the
installed OpenFST 1.8.3 headers:
fst.h:690:59: error: no match for 'operator=' (operand types are
'std::unique_ptr<fst::SymbolTable, ...>' and 'fst::SymbolTable*')
FstImpl's copy-assignment operator assigns the raw pointer returned by
SymbolTable::Copy() straight to a std::unique_ptr member. No C++ standard
allows that, so the line is ill-formed everywhere; it survived because nothing
instantiates FstImpl::operator= and gcc up to 13 only checks a template
member's body when it is instantiated. gcc 14 resolves non-dependent operator
expressions at template definition time, so it rejects the line in any
translation unit that includes <fst/fst.h>. The CI diagnostic confirms the
phase: it reads "In member function", not "In instantiation of", and carries
no instantiation backtrace.
That is why this surfaces only here. build_itn_deps.sh compiles OpenFST with
gcc-12 and upstream's own images build the runtime with gcc-13, so neither
compiler reaches the check; backend/Dockerfile.golang installs gcc-14 and
promotes it with update-alternatives, and fst_normalizer.cpp is the one
translation unit in this backend that includes OpenFST.
Fix it in the installed ITN prefix, which is the only copy the cmake build
compiles against, using the same .reset() spelling FstImpl::SetInputSymbols
already uses for the identical operation. libfst.so is linked before this runs
and cannot contain the function, since no compiler could ever have emitted it,
so there is no ABI or ODR consequence. The rule is guarded on both sides so a
pin bump to a fixed OpenFST fails loudly rather than silently no-opping.
Verified with a real gcc 14.2: the CI error reproduces byte for byte from a
file whose entire content is '#include <fst/fst.h>', and gcc 14 reports
exactly two errors over the whole OpenFST include closure this backend uses,
both of them these two lines. After the patch that closure compiles clean
under gcc-14 with the target's own flags. The step is reachable only under
WITH_NORM=ON, so 'make -n stage-libs WITH_NORM=OFF' mentions neither it nor
the ITN build, and darwin, which defaults WITH_NORM to OFF, never evaluates it.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
NEMO_SPEECH_TTS_WITH_JA=ON compiles Open JTalk's bundled MeCab, and
mecab/src/dictionary.cpp derives a comparator from std::binary_function,
which C++17 removed. libstdc++ still ships it as deprecated-but-present
under -std=gnu++17, so Linux never notices. libc++ compiles it out and
the macOS arm64 build dies with "no template named 'binary_function' in
namespace 'std'".
This is ours, not an upstream regression: upstream defaults both
NEMO_SPEECH_TTS_WITH_JA and NEMO_SPEECH_TTS_WITH_ZH to OFF and the OSS
drop carries no CI at all, so that target is never built there. Upstream
does already carry the equivalent workaround for MSVC's STL
(_HAS_AUTO_PTR_ETC plus /FIfunctional) but has no libc++ branch.
libc++ gates the two templates on
_LIBCPP_ENABLE_CXX17_REMOVED_UNARY_BINARY_FUNCTION, and has since LLVM
16, older than any clang Xcode still ships. The name is the whole
problem: _LIBCPP_ENABLE_CXX17_REMOVED_BINDERS covers bind1st, bind2nd,
ptr_fun and mem_fun and not unary_function or binary_function, and the
umbrella _LIBCPP_ENABLE_CXX17_REMOVED_FEATURES no longer exists in
libcxx at all. A wrong name preprocesses fine and fixes nothing.
Applied through CMAKE_CXX_FLAGS rather than to the one target, because
the tokenizer CMakeLists is upstream's and sources/ is a pinned
checkout. Project-wide is also the safer scope: the macro decides
whether libc++'s internal __binary_function alias resolves to
std::binary_function or to __binary_function_keep_layout_base, a base
class of std::less and friends, so defining it for a subset of
translation units would give those class templates two spellings in one
binary. Both bases are empty and, at C++17, carry identical members, so
the define changes no layout and no ABI.
Darwin only. On Linux the branch is unreachable and the macro is not a
name libstdc++ knows, so it would be inert even if taken; a Linux
configure with the flag forced on puts it on all 23 C++ TUs of
nemo_speech_openjtalk_frontend including dictionary.cpp at -std=gnu++17,
and on none of the 16 C TUs.
Mandarin needs nothing: cppjieba v5.6.7 and limonp have no removed C++17
constructs left (limonp replaced std::not1 and std::bind2nd with
lambdas) and cppjieba's own CI builds macos-14 and macos-latest at C++11
through C++20.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The macOS backend build died in patch-ggml:
scripts/apply-ggml-patches.sh: line 56: mapfile: command not found
make[1]: *** [patch-ggml] Error 127
mapfile is a bash 4 builtin (and its -d flag needs 4.4). macOS ships bash
3.2.57 as /bin/bash and GitHub's runner images add no newer one, so the
bare `bash` the recipe resolves from PATH cannot run upstream's script.
Rather than hunt for a capable bash that the runner does not have, drop
the step where it does nothing. ggml-patches/ is a CUDA series: every
kernel it adds is under src/ggml-cuda/, and its whole footprint outside
that directory is an op enum plus prototype in include/ggml.h, the
constructor and a name-table entry in src/ggml.c, and two ggml-cpu lines
that make the CUDA-only op report unsupported and abort. Nothing it
touches is compiled into a Metal kernel or changes a CPU one.
The project's own references to patch-only ggml symbols sit behind
NEMO_SPEECH_FUSED_RELPOS_ATTN and NEMO_SPEECH_FASTCONFORMER_CUDA_FUSIONS,
which cmake already forces OFF without GGML_CUDA, or behind
NEMO_SPEECH_GGML_PATCHED itself, which guards a GGML_TENSOR_FLAG_Q8_PLANAR
write that a non-CUDA buffer throws before reaching. So passing
NEMO_SPEECH_GGML_PATCHED=OFF costs the Metal build nothing, and it is
required once the series is skipped: that flag is what stops the ASR
sources referencing a tensor flag stock ggml does not define.
This is upstream's own Metal configuration. Its metal-* and vulkan-*
CMake presets inherit the cpu-* ones, which set NEMO_SPEECH_GGML_PATCHED
to OFF; docker/Dockerfile and scripts/windows/build.ps1 do the same for
their non-CUDA targets. LocalAI's Makefile never passed the flag at all
and so inherited the CUDA default everywhere.
Linux is untouched and keeps applying the series, including its
idempotency and its hard failure on a patch that does not apply. The gate
is the same uname test the WITH_NORM block above already uses, and both
branches keep the order-only clone prerequisite, which on a WITH_NORM=OFF
tree is the only thing that pulls sources/ in.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
gosec flags 13 alerts on this backend: one G304 and twelve G103. Each was
checked individually rather than blanket-suppressed, and each annotation
states what makes that particular site safe.
The G304 at audio.go is a false positive. The opened path is
filepath.Join of a directory the function just created with os.MkdirTemp
and a constant basename; the request-controlled path is the input to
AudioToWav and never reaches the open.
The twelve G103 sites are the package's three established shapes, and
every one was verified against them: cstr and pinPtr take the address of
something pinned on the line above and return it one-way (nothing in the
package converts either result back, which is what keeps checkptr out of
it under -race), and each *Create hands C a stack-local POD config whose
uintptr members are cstr allocations or pinPtr addresses held by a pinner
the loader unpins only after the call. The two slice-building sites are
bounded by construction: DiarSegments is handed exactly len(buf) with the
buffer sized under maxDiarSegments and a reported count larger than it
rejected rather than sliced to, and the TTS callback copies out a slice
whose length is the length the runtime declared for that buffer.
Separately, sampleRateOf gets a real fix rather than an annotation.
go-audio reads the WAV header's sample rate from an unsigned 32-bit field
into an int, so a header claiming more than 2^31-1 passed the "> 0" test
and then narrowed to a NEGATIVE rate, which the runtime would take as a
resampling ratio. AudioToWav cannot produce one today, but that is a
property of another package and this function exists precisely because
the rate is read back rather than assumed, so the bound is enforced here
and pinned by a spec.
The four remaining integer narrowings are annotated with the bound that
makes each safe: the WAV payload length is already checked against
maxWAVDataBytes, the speaker count is bounded by maxDiarSegments, and the
two segment ids are the proto's own int32 wire type.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
INVALID_ARGUMENT was translated to codes.InvalidArgument at exactly one of
sixteen C call sites. Everywhere else a non-zero status collapsed to
codes.Internal, so the same backend answered an unsupported language pair with
HTTP 400 and an unknown TTS voice, which is the same class of caller mistake
against the same process, with HTTP 500. Status 4 is CANCELLED on the ASR and
TTS surfaces and was reported as a backend failure rather than as the consumer
having stopped listening.
asr.h, tts.h and nmt.h each declare their own status enum and diar.h reuses the
ASR one; the values they share agree, and the single divergence is that NMT
declares no CANCELLED because nemo_speech_nmt_translate has no callback for a
consumer to stop with. That is an absence, not a disagreement, so one table
serves all three. status.go carries it, with the header line numbers and a note
that a pin bump has to recheck it: purego binds by name and the status crosses
as a bare int32, so nothing in the build or the linker can see a drift.
New specs cover the whole enum, unknown values, and one real INVALID_ARGUMENT
per family driven through the shared objects rather than through the Go mapping
asserting against itself.
Also add UsecaseChat to this backend's capability entry, which the docs already
told operators to set for translation models. chat is a gallery filter key and
completion is not, so GET /api/backends/usecases would have greyed the Chat
filter out and hidden a Riva-Translate gallery entry from the one filter that
fits it. The flag gates no endpoint; it makes the model eligible as the default
chat model and puts it in the web UI chat picker, both of which Predict and
PredictStream already serve.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Three factual errors found in review, all of them the kind a user would act on.
The translation limits were described backwards. Input longer than the 1024-token
context is rejected, not truncated: translator.cpp throws "nmt: prompt too long
(N tokens) for context 1024", which reaches the caller as a failed request. What
is silently cut is the output, by the max_new_tokens loop at 256. The bullet now
separates the two and says which one fails quietly.
The macOS gap covers TTS text normalization as well. Both directions sit behind
the single NEMO_SPEECH_WITH_NORM flag, which the Makefile forces off on Darwin,
so tn_dir is as inert there as itn_dir. Neither fails the load: both warn and
carry on. pnc_model really is unaffected, since punctuation is compiled in
unconditionally. The tn_dir row in the option reference gained the caveat the
itn_dir row already had.
The TTS conversion procedure produced a model that could not load. It converted
MagpieTTS and stopped, leaving no NanoCodec, which the same page lists as
required; following it gave "no NanoCodec GGUF found next to ...". Both halves
are now there, each with the download that feeds it, so the block runs top to
bottom on a clean machine.
Also: any negative gpu value pins TTS to the CPU, not only -1, and FLAG_CHAT
additionally surfaces the model in the web UI chat picker.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Adds docs/content/features/nemo-speech-cpp.md, alongside the audio.cpp page
that is its closest sibling, and cross-links it from the speech-to-text,
diarization, text-to-speech, backend-type and compatibility-table pages so the
backend is reachable from every surface that lists its modalities.
The page covers the architecture-to-family table, every option key with a model
YAML per family, the translation prefix directive, the acceleration matrix, and
the four limitations this backend ships with: Linux-only inverse text
normalization, suppressed interim streaming results, the library's default
translation context and generation limits, and the absence of gallery entries.
knownPrefOnlyBackends gains the backend so it appears in the /import-model
dropdown. It stays preference-only and AutoDetect=false: general.architecture
lives inside the GGUF where no remote-repo probe can read it, and a translation
model carries an ordinary LLM architecture with no NeMo-specific marker.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The nvidia-l4t-cuda-13 capability pointed at nvidia-l4t-arm64-nemo-speech-cpp,
which is built on nvcr.io/nvidia/l4t-jetpack:r36.4.0 and therefore links ggml
against CUDA 12. A Jetson whose CUDA 13 runtime is present reports that
capability and would have pulled an image with no libcudart.so.12 to dlopen,
failing hard at load. That is worse than omitting the key: with no key
Capability() falls back to "default" and the host gets a working CPU build.
Fixed the way parakeet-cpp and moss-transcribe-cpp already do it, by shipping
the second L4T image rather than dropping the key. Nothing prevents building it
here: those peers use plain ubuntu:24.04 on ubuntu-24.04-arm with the same
Dockerfile.golang as this backend's other rows, and every package in the
nemo-speech-cpp apt gate exists on noble arm64.
Adds the -nvidia-l4t-cuda-13-arm64-nemo-speech-cpp matrix row and its two index
entries, repoints the key on both metas, and rewrites the capability-map comment,
which had the reasoning backwards.
Also adds the documentary inferBackendPath branch, matching all six sibling
*-cpp Go backends. Behaviour is unchanged; the generic golang fallthrough
already resolved this backend correctly.
The previous commit message said "all seven handlers" of the shared gRPC
wrapper. There are eight RPC entry points: seven are guarded by
checkModelIdentity and AudioTranscriptionLive is the unguarded eighth, which
that message already called out separately. Wording only.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Registers nemo-speech-cpp across every surface .agents/adding-backends.md
requires, and adds the CI job its unit suite never had.
backend/index.yaml gets the meta backend (capabilities map, no uri), a
development meta and 12 image entries. No amd and no intel capability keys:
upstream NeMo-Speech.cpp builds ggml with CUDA, Vulkan or Metal only, and
SystemState.Capability falls back to "default", so those hosts get the CPU
build rather than a tag that does not exist. The nvidia-cuda-* and
nvidia-l4t-cuda-* keys are present because getSystemCapabilities() refines an
NVIDIA host to them whenever the CUDA directory exists; without them every
modern CUDA host and Jetson would miss the map and quietly run on CPU.
.github/backend-matrix.yml gets 7 include rows and 1 includeDarwin row. No
hipblas and no sycl rows, for the same upstream reason. cpu and vulkan are
per-arch pairs sharing a tag-suffix so backend-merge-jobs builds a multi-arch
manifest: an ARM host with no NVIDIA GPU reports "default" and the Jetson image
does not cover it.
The CI job is the substantive part. make test-extra is dead on master, because
prepare-test-extra depends on a protogen-python target that does not exist and
no workflow invokes it anyway, so the entry added earlier in this series ran
nowhere. abi_test.go asserts the size and field offsets of every Go mirror
struct against the C ABI it is dlopened into, and those assertions are the only
defence against silent memory corruption after a purego symbol rename or an
upstream header change. tests-nemo-speech-cpp in test-extra.yml now executes
them on pull_request and on master, gated on the backend's own path filter.
The recipe sets NEMO_SPEECH_REQUIRE_LIBS=1, so a missing library fails rather
than skips. WITH_NORM=OFF skips the OpenFST leg and costs no coverage: nothing
in the four C ABI headers is conditional on it, so the layouts are identical.
Also registers the upstream pin with the bump bot, which the backend Makefile
already claimed but was never wired up, and adds the BackendCapabilities entry
so a hand-written model config gets a real usecase surface. PossibleUsecases is
the union of the four families and DefaultUsecases is transcript alone, the
audio-cpp pattern. No VoiceCloning key: MagpieTTS synthesizes from baked
speaker ids, not a reference clip.
No gallery entries: publishing converted GGUFs is a follow-up.
ModelIdentity needs no work in this backend. main.go serves through
grpc.StartServer, so every RPC lands on pkg/grpc's shared server wrapper first,
and checkModelIdentity is the first statement of all seven handlers this
backend implements. A second check inside NemoSpeech would be unreachable and
would risk diverging from the cross-language sentinel the router matches on.
AudioTranscriptionLive stays unguarded because TranscriptLiveRequest carries no
ModelIdentity field at all, which is a proto-level gap affecting every backend
and needs its own change.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The directive regex allowed an unbounded run of two-letter segments per side, but
nothing tested it: narrowing that run back to a single optional segment left every
spec green. resolve_tag accepts a ready pair tag in one field with the other empty
(src/nmt/langpairs.cc), and those tags run to three segments (en-zh-cn, pt-br-en),
so a shorter pattern does not mis-split the tag, it fails to match the directive at
all and the whole bracket is handed to the model as text to translate.
The justification on the regex was also wrong and is corrected: pt-br and zh-cn are
two segments and parse either way. It is the single-field form that needs the run.
Renames the NMT handle to n.nmt so it stops sharing a name with the translator
interface, following n.synth, which is shortened for the same reason.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
nemo_speech_nmt_translate takes explicit source and target languages and has no
free-form generation or token-callback entry point, so there is no prompt in the
LLM sense. The pair comes from the source_language / target_language model
options, with an optional leading [src->tgt] directive as the only per-request
override, and PredictStream emits the whole translation as a single chunk
because the C API has nothing finer to give it.
Both RPCs wrap their body in withEngine so the family check and the C calls that
trust the handle share one acquisition of engineMu. PredictStream closes its
channel on every path, including the family rejection: this is the legacy
streaming contract, and pkg/grpc/server.go blocks on a drain goroutine that only
finishes when the channel closes, so leaving it open hangs the RPC rather than
failing it.
nmtTranslatorConfig is extracted so its four adjacent pointer fields can be
asserted against distinct sentinels. Transposing two of them changes neither the
struct size nor any field offset, so the layout assertions cannot see it.
Also removes goString, which had no production caller: every string-returning
symbol in abi.go is bound with a Go string return that purego converts itself.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The PCM callback is compiled once per process behind a sync.Once, not once per
request and not once per load. purego.NewCallback writes into a fixed table of
2000 entries (purego/syscall_sysv.go) and never releases one, so a per-request
callback panics the backend process on the 2001st synthesis, and a per-load one
reaches the same ceiling on a server that swaps models. Synthesis is routed
through that single callback plus a user_data id: engineMu is per-model, one
process holds several models, so a single current-sink pointer would be
overwritten by two TTS models synthesizing at once.
Deviations from the brief, all verified against the real headers and proto:
- TTS is TTS(*pb.TTSRequest) error and TTSStream is
TTSStream(*pb.TTSRequest, chan []byte) error, per pkg/grpc/interface.go.
The brief's context/pb.Result and server-stream forms do not implement the
interface. The channel is closed on every path, including the family
rejection, because pkg/grpc/server.go blocks on its drain goroutine and an
unclosed channel hangs the RPC with the backend lock held.
- The callback takes unsafe.Pointer, not uintptr. Converting a uintptr
parameter back to a pointer is a checkptr violation that aborts under
-race.
- resolveSpeaker refuses to turn a negative number into a speaker index. -1
is the C API's "use the default" sentinel, so the brief's rule would have
made a request naming an invalid voice synthesize in the default voice
instead of being rejected.
temperature and cfg_scale each write their override flag as well:
magpietts/runtime.cpp reads the float only when the flag is set, so a
temperature without it is silently discarded.
Also folds in Task 8's review finding on asr.go: the six bare -1 sentinels in
loadASR move to an asrDiarConfig builder reusing diarGeometryDefault, with
specs. src/asr/c_api.cpp applies left_context_frames at >= 0, so a dropped
sentinel pins the model geometry to 0 and no layout assertion can see it.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The six frame-geometry overrides were written as -1 with nothing
asserting it. c_api.cpp applies left_context_frames at >= 0 while the
other five need > 0, so a dropped sentinel there pins the model's left
context to zero, and the struct keeps exactly the same shape, which is
all the layout assertions can see. Extracting diarModelConfig makes the
values assertable: five specs now pin all six frame fields, the device
index, the declared size and the NULL preset, each frame field on its
own line so a missing sentinel names itself.
distinctSpeakers had a spec with three segments over three distinct
labels, which len(segs) satisfies just as well as the real thing. Four
segments over three labels makes it a spec that can fail.
collectSegments sized its buffer straight from a count the C side
reported, and make() panics rather than erroring on a length it cannot
satisfy, so an uninitialised size_t coming back across the ABI killed
the backend process instead of failing one request. A ceiling of 2^22
segments, upwards of 93 hours of audio at one 80 ms frame each, turns
that into a diagnosable error.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
loadDiarizer creates the Sortformer diarizer and Diarize serves the RPC
over a diarization stream: decode, chunked push, finish, then the
count-then-fill segments protocol.
nemo_speech_diar_segment carries start_time and end_time in SECONDS
already, not frame indices, so no conversion happens on the way to
DiarizeSegment.start/end and the model's seconds-per-frame is not
involved at all. The speaker label is the runtime's 1-based tag as a
decimal string, matching what wordsToSegments emits on the ASR path, so
the same speaker reads the same way whether a caller diarized a file or
transcribed it.
The six frame-geometry overrides are written as -1 rather than left
zero. c_api.cpp applies left_context_frames when it is >= 0 while every
other override needs > 0, so a zeroed config would silently pin the left
context to zero and change the model's streaming geometry.
nemo_speech_diar_segments writes *count before it rejects a buffer that
is too small, so a rejected fill still reports the size to retry with.
collectSegments uses that rather than truncating, bounded at four
attempts because the RPC holds engineMu for its whole body and an
unbounded retry would block an unload behind it.
Two DiarizeRequest knobs map onto the segmentation config, and the
proto and header names cross over: min_duration_on is the C
min_duration_sec and min_duration_off is the C min_gap_sec. Six fields
have no equivalent in this pipeline and are logged rather than dropped
in silence: num_speakers, min_speakers and max_speakers (Sortformer's
capacity is fixed by the checkpoint), clustering_threshold (there is no
clustering stage), include_text (no ASR here) and threads.
The empty-PCM guard fires before the stream is opened, so a silent clip
never reaches a purego entry point that would dereference &pcm[0].
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
runLive wrote the inter-utterance separator into the accumulated
transcript but emitted the delta without it, so a two-utterance turn sent
"one." and "two." while the terminal result read "one. two.". The live
consumer is the one that really concatenates: the realtime semantic-VAD
path joins the accumulated deltas with the empty string and clears them
only at a turn reset, never at an endpoint, so the running caption read
"one.two.". The separator now goes into the delta, as it already did on
the file path, and the terminal text is the verbatim concatenation rather
than a trimmed rebuild.
TranscriptSegment.Words was never populated, so a request asking for
timestamp_granularities ["word"] came back with no words at all even
though the timings were decoded. wordsToSegments now attaches them,
gated on the granularity the same way parakeet-cpp gates it, so a
transcript that did not ask for word timestamps does not pay for them.
Also: the final that comes back from the tail flush no longer claims an
end-of-utterance. It is the end of the stream, not a user yielding the
turn, and eou is what the realtime turn detector acts on.
The comment explaining why interims are suppressed led with the runtime's
postprocessing. The wire contract is the stronger reason and now comes
first: consumers concatenate deltas, so forwarding a growing hypothesis
assembles to "hehellhelloHello.". The postprocessing only explains why no
diffing trick would rescue them. It is also ITN and strip_formatting
rather than punctuation, which is off by default here.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
AudioTranscriptionStream drives a whole clip through the cache-aware
streaming API in 100 ms pushes, emitting each finalized utterance as a
delta and closing with the assembled result. AudioTranscriptionLive
serves the bidirectional RPC over the same session: config first, a ready
ack, deltas with word timings as utterances land, and a terminal result
when the caller closes its send side.
Both wrap their body in withEngine, so a stream holds engineMu for its
whole life and Free waits on it rather than destroying the recognizer
underneath a half-finished stream. That makes the way out load-bearing:
the file loop honours the request context between pushes, and the live
loop ends when the host closes the request channel, so a disconnected
client cannot pin the model against unload.
Only finals become deltas. The runtime applies punctuation and inverse
text normalization on finals only, so a final rewrites the utterance
rather than extending its interim, and delta on the wire is
newly-finalized text that consumers concatenate. Forwarding interims
would duplicate and mispunctuate every utterance.
The four streaming entry points sit behind an asrSession interface. No
NeMo GGUF is small enough to keep in the tree, so without that seam the
need-more-audio drain would have no test at all: nemo_speech_asr_stream_next
reports OK with a NULL handle when it wants more audio, which is a pause
rather than an end, and reading it either way round drops results or
spins forever.
Also folds in three items from the offline transcription review:
- empty audio is now refused before anything crosses the ABI, not
inside recognizeF32. The added integration spec caught the old
ordering panicking on an unbound entry point instead of failing;
- an undecodable sample rate is an error rather than 0, which this
runtime reads as "already at the model rate" and would have made a
wrong rate silently pitch-shift the audio;
- AudioTranscription guards its result pointer instead of relying on
an unstated invariant.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Create the ASR recognizer in loadASR and serve AudioTranscription.
Segment times are int64 nanoseconds, not seconds: the proto field is an
int64 that core/backend reads straight into a time.Duration, while the
runtime reports word offsets in milliseconds. Words are grouped into one
segment per consecutive speaker run, with the 1-based speaker tag carried
through and 0 (untagged) left unlabelled.
The whole RPC body runs inside withEngine so the family check and the C
calls happen under one acquisition of engineMu. Free runs without the
backend lock, so checking the family and then relocking would let a
teardown destroy the handle in the gap. The audio decode is inside the
closure too, which costs nothing: base.SingleThread already serialises
this backend's RPCs.
recognizeF32 guards zero-length PCM. &pcm[0] panics on an empty slice, so
Go never reaches the C side's own "empty audio" rejection, and a silent
clip or a truncated upload is ordinary input.
pkg/utils has no WAV decode helper, only the ffmpeg normalisation, so
audio.go pairs AudioToWav with go-audio the way parakeet-cpp does. It
returns the sample rate rather than a duration, since the C API resamples
off that number.
Also closes the write-side half of the race Task 5 fixed on the read
side: Load now holds engineMu across the family switch and the n.fam
commit, matching Free. The loaders still must not take it.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Three review items, plus a defect the race detector turned up.
The spec covering "no family selected after a failed load" wrote junk to
a .gguf, so Load returned at ggufArchitecture before a family was ever
chosen and the assertion was vacuous. Generalised the GGUF test helper
to take a string architecture, and added a spec that loads a magpietts
GGUF with no sibling codec, so familyFor succeeds and discoverTTSAssets
then fails. It self-guards on ggufArchitecture so it cannot degrade back
into the earlier path.
requireFamily read n.fam unlocked while Free wrote it under engineMu,
which the race detector confirms is a real race. pkg/grpc/server.go
calls Free without the backend lock every other RPC holds, so teardown
can land mid-request. withEngine now takes the lock, checks the family
and runs the body under one acquisition; two would leave a window for
Free to destroy the handle between check and use. The locking protocol
is stated in both directions for the RPCs still to be written.
Running -race also enables checkptr, which aborts on cstr's pointer
being read back by goString: converting a uintptr to a pointer is fatal
whenever the address lands in a Go allocation, so a pinned Go buffer can
never be dereferenced from Go. The pointer is for C alone. Both helpers
now document the one-way contract, and goString is tested against a real
C-owned string by rebinding the version symbol to return a raw char*.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Load sniffs the GGUF architecture, maps it to a family and dispatches to
the family's loader. requireFamily gates every other RPC, returning
Unimplemented naming both the loaded and the wanted family so a
misconfigured model YAML produces a message a user can act on.
The family is committed only once its loader has succeeded. A load that
fails part way through would otherwise leave the gate open on a handle
that was never created.
cstr uses runtime.Pinner rather than an ordinary Go allocation. The
address crosses the ABI as a uintptr, which the collector does not
trace, so incidental reachability through the release closure is not a
guarantee: a caller discarding that closure could have the bytes
collected before the create call reads them. Pinning is the sanctioned
mechanism, makes the release function do real work, and turns a dropped
release into a loud leaked-Pinner panic instead of silent corruption.
Free overrides the base no-op to destroy the handle and reset the
family. Every family owns C memory only its own destroy entry point can
release, so without this an unloaded model leaks an acoustic model.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The layout assertions were inert. TEST_PATHS does not cover this backend and
the per-backend list in test-extra had no entry for it, so nothing invoked the
package's tests. Add it next to depth-anything-cpp, supertonic and vllm-cpp,
the group whose own test target carries its build prerequisites; stage-libs
already pulls the native build chain, so no prepare-test-extra entry is needed.
The skip guard was also loader-inconsistent: librariesPresent stats bare
filenames relative to the working directory while openLibraries resolves them
through the loader search path, so any invocation other than make test skipped
every library-backed spec and still reported green. NEMO_SPEECH_REQUIRE_LIBS=1
turns that into a failure naming the directory and the remedy, and the Makefile
test target sets it. Unset, the plain skip survives so a developer without a
build can still run the pure-Go layer specs.
Trim the default-value fingerprint from roughly forty assertions to eight. It
was pinning tunables such as threads and flush_partial_chunk, so a legitimate
pin bump would have failed with a message reading like a layout error. What
survives is only header-documented contract: the lone non-zero max_alternatives,
the run of -1 sentinels and the zero that witnesses where it stops. Verified the
narrowed spec still catches a mirror and offset table corrupted in lockstep,
which is the one class only this layer sees.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
purego binds by name at runtime and the config structs are passed by pointer,
so both a renamed symbol and a mismatched struct layout would otherwise survive
a green build. registerSymbols names the failing symbol, and the layout specs
compare each Go mirror against the size the library reports for itself, against
the offsets a C compiler produces for the installed headers, and against the
default values upstream writes into the structs it returns.
Two of the bindings differ from the plan because the headers do. The plan's
nemo_speech_diar_segments signature omits the segmentation-config pointer that
diar.h declares as the second parameter, which would have shifted the output
buffer, the capacity and the count pointer one position each. And
nemo_speech_diar_stream_push_f32 was missing from the symbol table although
standalone diarization cannot work without it.
Also close the two panic and equality gaps left in family.go: ValueString panics
on a mistyped general.architecture, and the self-codec guard compared a Cleaned
candidate path against an uncleaned one, so a doubled separator let the primary
GGUF be selected as its own codec.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
An empty value like "vad_model:" must stay empty, since callers read the
empty string as "unset". That branch of resolve() had no spec: dropping the
guard left every spec green while parseOptions started returning the models
directory itself. Add the spec that fails without the guard.
A known key with an unparseable value is a typo, not a config from a newer
backend, and "gpu:banna" failed expensively: the model loaded, produced
correct output, and ran on CPU with no signal anywhere. Log it. Unknown keys
stay silently ignored, which is what keeps configs forward compatible.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Addresses the fourth review round.
The nemo-speech-cpp apt block sat immediately after the shared apt layer, above
the Vulkan SDK build, the CUDA and ROCm installs, the Go toolchain and the
protoc download. Docker keys each layer on its parent, so inserting a step there
re-keys everything below it: a byte-identical shared layer is not enough, and
merging as it stood would have forced all of those to re-execute once for every
Go backend image. Move it down beside the existing opus, crispasr and
sherpa-onnx gates, which sit after those layers for the same reason.
Checked the ordering both ways before moving. Nothing between the two positions
uses these packages: the Vulkan and opus blocks install their own ninja and
pkg-config, go install protoc-gen-go needs the Go toolchain rather than protoc,
and the protoc 27.1 step is a release-binary download that needs neither
protobuf-compiler nor libprotobuf-dev. Nothing in the block needs anything those
layers provide; it uses only apt, and the mirror rewrite from the first RUN
persists in the image. It also runs no update-alternatives, so the default
compiler stays untouched for later layers. The diff against master is now a
single additive hunk with no shared layer touched.
Also preflight ITN_PROTOC. configure gates a preset PROTOC on test -n alone, so
a path that does not exist is accepted and the error surfaces much later as a
bare "No such file or directory" from inside make -C src/proto. The pin
introduced that failure on a box whose only protoc is in /usr/local/bin, which
worked before. Check it alongside the gcc-12 check and name the ITN_PROTOC=
override in the message.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Addresses the third review round.
Dockerfile.golang installs protoc 27.1 into /usr/local/bin, ahead of /usr/bin,
while libprotobuf-dev is the distro's 3.21 on noble and 3.12 on jammy.
Sparrowhawk resolves protoc from PATH at make time (configure.ac uses
AC_CHECK_PROG, so PROTOC substitutes to the bare word, and src/proto/Makefile.am
invokes it) and commits no pregenerated stubs, so the rule always runs. Code
generated by 27.1 includes google/protobuf/runtime_version.h and a
PROTOBUF_VERSION guard the older headers lack, so the WITH_NORM build could not
complete. Pin PROTOC to the apt one for that step; configure documents that a
pre-set value wins. The apt protoc and libprotobuf-dev come from one source
package at one version, which is the property that makes this correct.
The text-normalization stack is now a target keyed on a file build_itn_deps.sh
actually produces, rather than a side effect of the runtime library rule. As a
side effect make could not see whether it existed, so once the library was up to
date the script could never run again: a tree built WITH_NORM=OFF could not move
to ON, and make test hard-failed with no escape but a full 345 MB clean. It is
now built on demand and reachable on its own as 'make itn'. Staging keys on the
prefix existing rather than on WITH_NORM, so it stages what the tree actually
built, and package.sh's closure guard remains the backstop.
An already-configured build tree also now wins over the platform default, so a
tree built WITH_NORM=OFF is not silently reconfigured to ON by a bare make test,
which is what demanded gcc-12 from developers who chose not to have it. An
explicit WITH_NORM= on the command line still overrides both, and the ITN rule
preflights for gcc-12 with an error that names the alternative.
Move ninja-build out of the shared apt layer into the existing BACKEND-gated
block. Dockerfile.golang serves 225 matrix entries and only this backend
configures with -G Ninja, so the common list is byte-identical to master again
and no other image loses its cache.
Drop libabsl-dev and correct the comment that justified it. No base image here
ships protobuf 25, so nothing needs the absl split, and the cmake glob looks in
/usr/lib rather than the multiarch directory Ubuntu actually uses, so the
package could never have contributed anything.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Addresses the second review round.
The dependency-closure guard failed open. Its glob expands once per pass, so
each pass advanced the closure by exactly one level, and the fixed count of five
passes then fell out of the loop without checking whether anything remained. An
eight-deep chain packaged six libraries, exited zero and reported success. That
is the case the guard was written for: asr to sparrowhawk to protobuf to absl
already runs several levels deep, so a WITH_NORM build could ship missing its
deepest libraries and fail at first dlopen. The loop now runs until the staged
set stops growing, and exhausting the bound is a hard error rather than a silent
exit.
For the same reason, a build image with neither readelf nor objdump no longer
warns and skips. It cannot show the package is complete, so it refuses to ship
it. The guard is entered only when there is something to check, so an empty
package cannot trip the new error.
Dockerfile.golang installed ninja-build only in the Vulkan branch while this
Makefile runs cmake -G Ninja unconditionally, so the CPU, cuBLAS and L4T images
could not configure at all. ninja-build moves to the common apt list; it does
not change CMake's default generator, so it is inert for the other backends.
gcc-12 was nowhere in the tree, yet WITH_NORM defaults ON and
build_itn_deps.sh needs it, so the committed default was unbuildable in CI.
Install it, with the protobuf, absl, re2 and autotools that Sparrowhawk and
OpenFST need, gated on BACKEND so the other Go images do not carry it. The list
follows upstream's own docker/Dockerfile, trimmed of the gRPC, portaudio and
python entries a BUILD_GRPC=OFF build does not use. Text normalization stays ON:
downgrading it silently would ship a backend advertising a feature it lacks.
Also: make test depend on stage-libs, so LD_LIBRARY_PATH is not an empty
directory on a clean tree, and add an engine target so Dockerfile.golang's
cacheable prebuild layer is not skipped and a CUDA build stops recompiling all
of upstream on every Go-side change.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Addresses the review of the scaffold commit.
backend/Dockerfile.golang runs 'make -C backend/go/$(BACKEND) build' and then
copies package/ into the final image, so 'build' has to end with a populated
package/. It only staged shared objects, which would have shipped an image with
no binary and no libraries at all. The old staging recipe is now stage-libs and
the chain is stage-libs, nemo-speech-cpp-grpc, package, build, matching every
sibling Go backend.
Text normalization was packaged incorrectly. nemo_speech_text_normalization is
STATIC but links sparrowhawk, fstfar and fst PUBLIC, so they land as DT_NEEDED
on libnemo_speech_asr.so, and they live in a project-local prefix that nothing
else provides. WITH_NORM stays ON by default on Linux, since normalization is a
wanted feature. Instead stage_libs now copies .deps/itn/lib when WITH_NORM=ON,
and package.sh bundles it.
Staging that prefix is still not enough on its own: Sparrowhawk drags in
protobuf, re2 and absl, which neither build_itn_deps.sh nor
package-system-libs.sh provides. Rather than hard-code another hand-maintained
list, package.sh now walks the DT_NEEDED entries of everything staged and copies
whatever is unresolved, skipping the core set and the GPU set that the shared
scripts already own. It fails at package time, not at first dlopen, when
something cannot be resolved. On a WITH_NORM=OFF build the closure is already
complete and it copies nothing.
Restore CGO_ENABLED=0 on the Go build to match whisper, parakeet-cpp and
omnivoice-cpp. Note that purego reaches dlopen through fakecgo, so the binary is
dynamically linked either way; what the flag changes is the NEEDED set, and
lib/ld.so routing in run.sh exists precisely because the binary is not static.
Replace the hand-rolled .patched sentinel with upstream's
scripts/apply-ggml-patches.sh. It applies the series in filename order, exits
non-zero when a patch does not apply, and detects "already applied" by comparing
the full-series tree hash rather than an mtime, so it is safe to run every time
and there is no sentinel left to go stale or to wedge the build when deleted. It
is wired as an order-only prerequisite so running it does not force a relink.
Also: correct the package.sh header, which claimed three shared objects when
there are five and none of the TTS ones carry a _c suffix; give 'make test' the
LD_LIBRARY_PATH the dlopen tests will need; document that a NEMO_SPEECH_VERSION
bump needs 'make purge'; and extend 'clean' to remove package/ and the ITN
libraries.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Adds the backend skeleton and the NeMo-Speech.cpp build, pinned at
2e12e2def8a98ed06666f7ee3ca94e7193e04be4. The Go side is deliberately a stub:
it dlopens the runtime and starts the gRPC server, later work fills in the
symbol table and the model logic.
Three details of the upstream layout differ from what the plan assumed, and the
build reflects the real tree:
* The TTS C ABI ships as libnemo_speech_tts, not libnemo_speech_tts_c. Upstream
compiles c_api.cpp straight into the implementation library and only aliases
the nemo_speech_tts_c CMake target, so no _c object exists on disk. ASR and
NMT do build a real _c shim.
* Shared objects land in build/bin, since upstream points
CMAKE_LIBRARY_OUTPUT_DIRECTORY at ${CMAKE_BINARY_DIR}/bin.
* The ASR and NMT _c shims carry a DT_NEEDED on libnemo_speech_asr and
libnemo_speech_nmt, so those are staged and packaged alongside them.
Otherwise dlopen fails at startup.
The ggml patch step uses an order-only prerequisite. cmake writes into the
checkout and bumps its mtime past the sentinel, which would otherwise re-run
git apply over an already-patched tree and break every incremental build.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The backend could configure four of the engine's knobs (block size, KV block
count, max sequence length, max concurrent sequences) out of a config surface
that is considerably larger. Speculative decoding, prefix caching, the
chunked-prefill token budget, the scheduling policy and the external KV
connector were reachable from vllm.cpp's own HTTP server and from nothing
LocalAI could write in a model config.
Config now goes through `engine_args:`, the same map the vLLM and SGLang
backends take, with keys spelled as vLLM's own CLI flags so a speculative_config
or kv_transfer_config block written for vLLM works verbatim. The legacy
`options:` list keeps working and reads every key too; engine_args wins where
both set one. Unknown keys are logged and ignored rather than fatal: the field
is shared with the other engines, so a config carrying their knobs must not take
the model down.
Two details worth knowing:
`enable_prefix_caching: false` maps to the ABI tri-state force-OFF (2), not 0.
0 means "let the model capability decide" and dense architectures default the
cache on, so collapsing the two would silently enable it against an explicit
false. enable_jump_forward (ABI v10) shares the encoding, deferring to
VT_ENABLE_JUMP_FORWARD instead of to the model.
The importer probes config.json on a vllm-cpp import and writes
speculative_config: {method: mtp} when the checkpoint declares an MTP head, the
safetensors analogue of the llama-cpp importer's GGUF probe. DFlash draft repos
are refused with a warning instead, since a drafter cannot serve alone and the
pairing is not derivable from either repo. The draft path is resolved against
LocalAI's model directory, because the engine only looks in a directory holding
config.json or in the HF cache and never downloads: the repo-id spelling the
vLLM docs teach used to die deep in the load with "draft checkpoint not found".
docs/content/features/text-generation.md gains a vllm.cpp section covering the
engine_args table, all three speculative methods, LMCache and the legacy list.
The backend had no documentation page before.
This replaces a branch that had gone stale behind master and carried its own
route to ABI v10, which #11386 has since landed in minimal form. Rebased onto
that as a single commit rather than replaying the intermediate steps, whose
ABI v9 mirrors no longer make sense against master's pin. The Darwin build
fixes for Apple Clang's gnu-folding-constant diagnostic on C++, Objective-C and
Objective-C++, originally authored by localai-org-maint-bot, are folded in here.
Verified: `make abi-check` agrees at v10; unit specs, core/config and
core/gallery/importers green; and the full e2e passes in 1330s against a CPU
libvllm.so reporting ABI v10 with Qwen_Qwen3.5-0.8B-Q4_K_M.gguf (load, blocking
completion, streaming, chat and tool calls).
Assisted-by: Claude:claude-fable-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Add the MTP and plain Q4_K_M GGUF builds with their shared vision projector so LocalAI users can select accelerated or fallback llama.cpp inference.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add Q4_K_M and Q8_0 builds of the popular refusal-removed Qwen3.5 9B fine-tune, including its multimodal projector.
Assisted-by: Codex:gpt-5 [web]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Switching from Backend Traces back to API Traces crashed the page
with "can't access property status, e.response is undefined" (#11376).
The API table briefly renders the previous tab's backend rows while the
refetch effect is still pending, and those rows carry no `response`
envelope. The status column dereferenced it unguarded. Render a neutral
placeholder instead of throwing, and cover the tab-switch scenario with
a regression spec.
Assisted-by: opencode:big-pickle
Signed-off-by: Nandana Dileep <110280757+nandanadileep@users.noreply.github.com>
The Go bindings mirror vllm.h by hand and refuse a library whose
vllm_abi_version differs from what they were written against. Two
automated pin bumps (#11174, #11352) moved VLLM_CPP_VERSION onto engines
declaring ABI v10 while govllmcpp.go still mirrored v5, so every
vllm-cpp image built since then panics at startup on every platform:
panic: vllm-cpp: ABI mismatch: library reports v10, backend built against v5
Grow both PODs to the v10 layout: vllm_model_params gains
speculative_config, enable_prefix_caching, max_num_batched_tokens,
scheduling_policy, kv_transfer_config and enable_jump_forward (88 bytes),
vllm_sampling_params gains the v8 logits-processor pair (136 bytes). The
offsets in the specs come from offsetof() against the pinned header. All
of the new fields are inert when zeroed, so the engine behaves exactly as
it did under v5; the backend sets none of them.
Nothing cross-checked the two files, which is why a blind pin bump could
ship a backend that cannot load. The library build now runs abi-check
first: it compares VLLM_ABI_VERSION in the fetched header against
abiVersion in govllmcpp.go and fails the build naming both, instead of
leaving the mismatch for a user's runtime.
Fixes#11379
Assisted-by: Claude:claude-fable-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Recover parser panics at metadata boundaries, skip unneeded remote arrays, and use the parser's overflow-hardened release. Keep detached gallery workers and CrispASR probes from terminating their processes on malformed GGUF input. Disable startup warming in the provided Compose files as an operational fallback.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
docs(blog): cover the terminal agent, and fix the counts in the intro
The 4.8 post never mentions that `local-ai chat` stopped being a REPL
and became an agent (#11291): the nib harness compiled into the binary,
with tool use behind an approval gate, sub-agents, MCP servers, plugins
and skills, auto-configured against the local instance. It also ships a
shell integration script for zsh, bash and fish that binds Ctrl+Space.
That is one of the larger user-facing changes in the release and it was
missing from both the post and the release-notes highlights. Added a
section after 3D generation, including the breaking changes for anyone
who had habits around the old REPL: `/clear` is gone in favour of
`/compact`, and a model switch now keeps the conversation.
While in the intro, corrected the counts. The post said 374 pull
requests in twenty-one days, which was accurate when it was drafted on
the 4th but not once v4.8.0 was tagged on the 5th. The published release
notes say 386 in twenty-two days, and the intro now matches them rather
than contradicting them.
For the record, neither figure is exactly right: `git log --format=%s
v4.7.1..v4.8.0 | grep -cE '\(#[0-9]+\)$'` counts 388 squash-merged pull
requests, and 389 from v4.7.0. The notes were cut before the last few
landed. Matching the published notes was the priority here, since that
is the artifact everyone else quotes, and 386 is the number already in
circulation.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The two landing-page clips predated the v4.8.0 interface work (#11288,
#11305, #11307): the gallery clip showed the retired light-theme Install
Models table, and the hero clip toured the Nodes pages in a full browser
window while its caption promised a chat completion on CPU.
Both are re-recorded from a real local-ai built from v4.8.0, dark theme,
app chrome only:
- hero-ui.mp4: a chat completion on lfm2.5-1.2b-instruct streaming on
CPU with the live tok/s meter, so the caption now matches the footage.
The poster frame is regenerated from the new clip.
- gallery.mp4: the Discover rail and detail pane, the hardware
recommendation lanes, the VRAM-by-context chart, and a real install
with the live progress banner.
The hand-typed model count moves from 1,585 to 1,255 in the three places
it appears, matching the distinct-model count the recorded UI shows on
screen. The 3d-generation clip is untouched: the post-capture UI changes
do not show in its footage.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
Add the official Q4_K_M and Q8_0 GGUF builds with their matching vision projectors so the compact agentic model can be installed through LocalAI.
Assisted-by: Codex:gpt-5 [web]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Cosign v2.4.1 does not select the Sigstore bundle format by default, while LocalAI's verifier only consumes OCI bundle referrers. Request the format explicitly for both registries and guard the producer contract with a shell regression test.
Document strict backend integrity configuration and release-tag identities for operators.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Register JSON API exchanges before their handlers run so the traces dashboard can surface active work. Replace the live entry with the completed persisted record under the same ID, and clean it up if a handler panics.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The runtime section described the small core and on-demand backends but
never stated the simple fact readers look for: one model can run on
llama.cpp while the next loads on vLLM, SGLang or MLX, behind the same
endpoint, and switching is one line in the model's config.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* docs(blog): new DeepSeek and Laguna numbers, visuals, humanizer pass
vllm.cpp master moved 26 commits past what the post was written against,
and two results changed enough to matter. Both came from the same lever:
staging weights device-resident at load instead of reading them from the
GGUF mmap over unified memory, which the GB10 reads about 20% slower per
GEMV than device memory.
- DeepSeek-V4-Flash against DwarfStar: 0.997x parity becomes 1.144x
ahead, 18.69 vs 16.33 tok/s decode, same generated tokens.
- Laguna-XS-2.1 against vLLM: 87% becomes 1.03x, 44.46 vs 43.10 tok/s.
New row in the scoreboard.
Adds three visuals. A chart of throughput against every reference engine,
which is worth having now that the spread is 0.976 to 1.144 rather than a
flat line at parity. The Activity page with four installs running, and the
model detail pane with all four pocket-35b variants. Both screenshots were
recaptured on 2026-08-04 because #11288, #11305, #11307 and #11222 had all
changed those pages since the earlier set.
llama.cpp is deliberately absent from the chart: its 1.18x is a prefill
ratio, and putting it on the same axis as throughput ratios would be
comparing two different measurements.
Also carries the media the release notes embed, since a GitHub release
body needs URLs that survive publishing and drag-and-drop has no CLI.
Supersedes #11364.
Humanizer pass on the prose. The post had collected five exactness idioms
in one section (token-for-token, byte-exact twice, byte-identical,
token-identical). One is precision, five is a tic, so the 27B row keeps
its "token-for-token identical" where identical output is the actual
claim and the rest say what they mean. That also fixed a hyphen in
predicate position ("is token-identical").
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
* docs(blog): redraw the benchmark chart as a branded card
The Flint bar chart was generic: default palette, no brand, and drawn
from zero, which made five ratios between 0.976 and 1.144 look like five
bars of roughly equal length.
Redrawn in the style of recorder-for-agents' render-card.sh cards, the
same shape as the vllm.cpp README GIF. Palette taken from the two logos
rather than invented (LocalAI navy #0E2632 and teal #469AAF, vllm.cpp
teal #3AB4CA), SVG generated by a small JS loop so the geometry is exact
at any scale, headless Chrome to PNG at 2x.
The substantive change is that bars now run from the 1.00 parity line
instead of from zero. Deviation is what the data is about, so DeepSeek's
+14.4% and MLX-LM's -2.4% are both legible, and the one row that is
behind is the one row in amber. Each bar carries its ratio and the raw
measurement under it.
Keeps the .html source next to the .png so the chart is editable later:
change a number, re-run render-card.sh.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This is the text users read in the backends list and the gallery, and it
was the last place still describing vllm.cpp as "a from-scratch C++20
port of vLLM created and maintained by the LocalAI team" with no
indication of maturity.
Three corrections, matching the v4.8 release notes and blog post:
- It leads with ALPHA. These are alpha development builds and llama-cpp
stays the recommendation for production, which is the single most
useful thing to know before clicking install.
- It is maintained by the LocalAI team but developed in its own
repository and usable without LocalAI. vLLM is named for what it
actually is, the reference implementation that output is checked
against and benchmarked against, rather than just the thing that was
ported.
- It records the featureset that has grown past vLLM: GGUF loading,
speculative decoding and KV offload, alongside the architecture and
hardware coverage that were already listed.
Also notes that the project is expected to be renamed, with the new name
still to be decided, so anyone who installs it now is not surprised
later.
vllm-cpp-development inherits all of this through the YAML anchor, so
both entries are covered by the one edit. Verified the file still parses
and that both entries carry the new text.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* docs(blog): final figures for the 4.8 post, and the MLX provider
The cycle closed at 374 PRs over twenty-one days, not the 321 over
eighteen the post was written against. Corrects the summary, the opening
line, the contributor count and the gallery total, and moves the date to
the day the release is cut.
Adds the MLX GEMM provider (#11137), which merged after the post was
written and is the one number an Apple Silicon reader wants: 1.54x to
2.19x on an M4 with time to first token roughly halving, both arms
toggled on one binary. The +/-10% caveat travels with the table rather
than being left in the PR.
Two lines edited against the no-ai-slop skill while I was in the file,
the same pass #11324 ran over the engines post:
- The opener balanced two clauses across a colon and closed on "without
lying to you", which is the built-to-be-quoted shape readers picked
out of the HN thread. It is a flat statement now.
- "This is a new modality rather than a new backend under an existing
one" is a binary contrast that says nothing the next clause does not.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
* docs(blog): call vllm.cpp alpha, and finish the no-ai-slop pass
vllm.cpp is not a released backend and the post read like it was. The
old wording buried the caveat in a block quote at the end of the section
and still said "first release of a young engine". It now says plainly,
before the caveat can be skipped, that these are alpha development
builds, that shipping them in 4.8 is about letting people try the thing
rather than recommending it, and that llama-cpp stays the default.
Also completes the no-ai-slop pass I had only half run. Counting the
lines built to be quoted, headings and section endings included, the post
is in reasonable shape: long flat informational stretches, tables
followed by a plain finding, headings that are labels rather than
epigram-verdicts. Three patterns survived, each one an item in eval.md:
- "and inverts that:" set the usual shape against ours across a colon.
The sentence works without the frame.
- "Two things were conflated there: a signal, which needs one line, and
the detail, which needs somewhere to put it" is a role-assignment pair.
Says what happens instead.
- "The maturity statement from the release notes is worth repeating in
full" is throat-clearing in front of a quote, and the quote is gone.
Left the rest alone. Minimum effective edit, not a rewrite.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
* docs(blog): present vllm.cpp as a community project, with its own numbers
The post described vllm.cpp as "a from-scratch port of vLLM, written and
maintained by the LocalAI team". Two things wrong with that. It is a
community project, and it has stopped being only a port: it loads GGUF,
runs on CPU, Metal and Vulkan, ships speculative decoding and KV offload,
and its benchmark page measures against llama.cpp, MLX-LM and DwarfStar
as well as vLLM, because those are the engines it competes with on that
hardware.
vLLM's role is now stated for what it is, the reference implementation.
Correctness is checked against it and the scoreboard is kept against it.
Also flags that the name will probably change, since it is drifting far
enough that vllm.cpp will eventually mislead.
Adds real numbers from the project's own docs/BENCHMARKS.md rather than
adjectives: 1.045x vLLM at concurrency 1 on Qwen3.6-27B NVFP4 with
token-for-token identical output, 1.010x and 1.013x at c16 and c32 on the
35B MoE and behind below that, prefill 1.18x over llama.cpp on CPU
aarch64, 97.6% of MLX-LM warm total on an M4. Upstream's own caution
travels with them: it treats c2 through c32 as ties because its noise
band is 0.5% and those margins are 0.7% to 1.7%.
Every figure was checked against ~/_git/vllm.cpp/docs/BENCHMARKS.md
rather than restated from memory. The heading is marked alpha to match
the section body.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
* docs(blog): say who maintains vllm.cpp, and add the DeepSeek Flash result
Two corrections to the previous commit.
"A community project" says nothing and was not quite true either. The
LocalAI team maintains vllm.cpp. Community-first is the intent, not a
description, so it now says that and says what backs it: its own
repository, its own docs, benchmark record and issue tracker, and it runs
without LocalAI anywhere in the picture.
Adds the DeepSeek-V4-Flash result, which makes the divergence point
better than any of the prose around it. That model does not run on vLLM
on a single GB10: every vLLM-loadable checkpoint is 156 GB or more
against a 119 GiB unified pool, and the only quant that fits is an
extreme-low-bit GGUF that vLLM cannot load. vllm.cpp reads GGUF and runs
it at 16.28 tok/s against ds4's 16.33, a parity result. Also notes MTP
speculative decoding, token-identical to vLLM's and about 4% faster at
concurrency 1.
Both figures checked against ~/_git/vllm.cpp/docs/BENCHMARKS.md.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
* docs(blog): lead the DeepSeek result with what we run, not with what vLLM cannot
The previous version opened on "that model does not run on vLLM on a
single GB10 at all". Wrong emphasis twice over: it makes a strong
negative claim about another project the headline, and it buries the
actual result, which is that vllm.cpp runs DeepSeek-V4-Flash at roughly
2-bit (IQ2_XXS mixed, about 80 GB) on a single DGX Spark and decodes at
16.28 tok/s against DwarfStar's 16.33.
The size constraint is still there, stated as the reason the quant is
what it is rather than as a point about vLLM: at 300B+ total parameters
even a 4-bit checkpoint is 156 GB or more, so a 2-bit GGUF is what fits
the Spark's 119 GiB unified pool.
The table row now names the quant and the box (IQ2_XXS, one DGX Spark)
instead of just "GGUF, GB10", since that is the part a reader with a
Spark wants.
Figures unchanged and still from ~/_git/vllm.cpp/docs/BENCHMARKS.md.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
* docs(blog): say the new name is undecided
"The name will probably change at some point" invited the obvious
question. It now says the rename is expected and the name is still to be
decided, which is the actual state.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Replace copied HauhauCS base-model text with metadata for the actual Genesis Hermes V6 artifact and link its upstream base model.
Assisted-by: Codex:gpt-5 [Hugging Face]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
No -gpu-rocm-hipblas-llama-cpp image has been published since 2026-08-01.
Every build since has been killed by GitHub at exactly its 6h job limit:
job 91830652349 cancelled 6h00m (2026-08-04)
job 91763226161 cancelled 6h00m (2026-08-03)
job 91466626154 cancelled 6h00m (2026-08-02 full matrix)
The registry shows the damage: master-gpu-rocm-hipblas-llama-cpp last
built 2026-08-01 05:53, latest-gpu-rocm-hipblas-llama-cpp 2026-07-15,
against master-cpu-llama-cpp which is current.
Same cause as #11321, different mechanism. Since #11255 every x86 GPU
image also builds ggml's CPU_ALL_VARIANTS matrix. SYCL died because icpx
stalls on one translation unit; ROCm dies on volume. hipcc compiles the
HIP kernels once per entry in AMDGPU_TARGETS, and that list is eleven
architectures (gfx908, gfx90a, gfx942, gfx950, gfx1030, gfx1100, gfx1101,
gfx1102, gfx1151, gfx1200, gfx1201). The CPU matrix lands on top of that.
The numbers are unambiguous. The same job took 2h27m in the 2026-07-26
full matrix, before #11255. #11255 merged 2026-08-01 07:26, an hour and a
half after the last image was published, and it has been 6h00m ever since.
The tail of the last run shows it 61% through ggml-hip at the 83 minute
mark, still building HIP template instances.
Route hipblas to the portable fallback, exactly as #11321 did for SYCL and
for the same practical reason: it is what these images shipped before
#11255, and run.sh already prefers *-cpu-all when present and falls back
otherwise. Expected to restore the 2h27m build with room to spare.
Not fixed here: the CPU variant matrix is genuinely wanted on ROCm for
partial offload. Getting it needs the build to fit in 6h, which means
trimming AMDGPU_TARGETS or splitting the job per architecture. Both are
larger changes than unbreaking the image, and neither should ride along
with a build that is currently not shipping at all.
Verified: make test-build-scripts passes, including the extended
llama-cpp-build-target_test.sh. bonsai is unaffected (own compile script,
ROCm builds in 1h52m) and turboquant has no hipblas variant.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): enable and vendor the MLX GEMM provider on darwin/metal
The darwin vllm-cpp image built the Metal backend with vllm.cpp's native MSL
GEMM only. vllm.cpp also ships an optional MLX provider for the dense GEMM,
kept OFF upstream because it costs a ~19 MB libmlx.dylib plus a ~105 MB
mlx.metallib, on the stated position that it must earn that cost by
measurement.
Measured on an Apple M4 (16 GiB, macOS 26.5.2) it does. One binary, arms
toggled with VT_OP_PROVIDER_DISABLE=mlx so there is no build-difference
confound, Qwen3-1.7B-bf16 p=512 g=128, 2 reps, arm order alternated per rep:
B=1 5.79 vs 3.08 agg tok/s (1.88x) TTFT 3.32 s vs 7.68 s
B=8 25.70 vs 13.69 (1.88x) TTFT 13.95 s vs 34.38 s
B=16 38.65 vs 17.69 (2.19x) TTFT 18.33 s vs 54.48 s
Peak RSS is unchanged (6.65 to 7.50 GB in both arms) and the output is
bit-identical: vllm.cpp's three-way parity test measures mlx-vs-msl NMSE of 0
on all six shapes, and mlx-vs-cpu equal to msl-vs-cpu, against a 5e-4 bar. MLX
serves the dense GEMM alone; paged attention stays vllm.cpp's own kernel
because MLX has no paged-KV primitive. Full disposition, including the
INDICATIVE status and the isolation actually achieved, is in vllm.cpp
docs/BENCHMARKS.md "MLX GEMM provider A/B on Apple M4".
Build: MLX comes from the pinned prebuilt pip wheel (MLX_VERSION, default
0.29.3) into a venv under the backend dir. Building MLX from source needs
`xcrun metal`, i.e. a full Xcode the macOS runners do not have, while the wheel
ships include/, lib/libmlx.dylib and the compiled metallib ready to link. The
install is a stamp FILE rather than a phony target, because a phony
prerequisite is always newer than libvllm and would re-link it every
invocation. VLLM_CPP_MLX=off restores the previous Metal build.
Packaging vendors libmlx.dylib, mlx.metallib and MLX's MIT license into
package/lib/. Three things this had to get right, each verified on the M4
before it was written rather than after:
1. libvllm.dylib links @rpath/libmlx.dylib and its build-time LC_RPATH points
inside the build venv, a path no user has. Every build rpath is deleted
and replaced with @loader_path/lib.
2. MLX loads its metallib from beside its OWN dylib, so both files must land
in the same directory or every Metal op fails with "Failed to load the
default metallib".
3. install_name_tool invalidates the code signature and macOS refuses to load
an arm64 image with a stale one, so the patched library is re-signed
ad-hoc.
Verified end to end on the M4 by building through this Makefile and running the
packaged artifact: `DYLD_PRINT_LIBRARIES` resolves libmlx from package/lib/,
`codesign -v` passes, no build-venv path survives in the load commands, and a
real generation runs with the provider selected (op=65 selected=mlx) and zero
metallib failures. A missing rpath now fails the build instead of the user's
first inference.
Cost: the darwin vllm-cpp image grows by about 124 MB.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
* fix(vllm-cpp): default the MLX GEMM provider OFF on darwin
This branch opened with VLLM_CPP_MLX=on, justified by an A/B that measured the
MLX provider at 1.88x to 2.19x against the native MSL GEMM. That measurement was
correct when taken and is now stale: vllm.cpp's own Metal kernels have improved
several-fold since, through mma prefill attention, a vectorised decode V
accumulation, vectorised attention staging, a fused qk-norm-RoPE preamble and a
simdgroup-per-row softmax. The native path MLX was compared against no longer
exists.
Re-measured on the same Apple M4, in the same binary, with the arms toggled by
VT_OP_PROVIDER_DISABLE=mlx, on Qwen3-1.7B-bf16 warm at p=512 g=128:
MLX provider ON prefill TTFT 1370 ms warm throughput 11.98 tok/s
MLX provider OFF prefill TTFT 1400 ms warm throughput 22.06 tok/s
Shipping the previous default would have halved Apple Silicon throughput.
MLX's steel GEMM is still about 20% faster than ours in isolation, but the
provider pays a per-op mx::eval synchronisation plus an output memcpy, because it
cannot write into our buffer. Across prefill's roughly 112 GEMMs that overhead
leaves a 2% gain; on decode, where the same synchronisation is paid once per
matmul per token, it costs 46%. The option is kept for prefill-dominated
workloads, where the margin is small but real.
The README section is rewritten rather than patched: it previously presented the
stale table as the reason for the default, so leaving it in place would have made
the new default look arbitrary.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): bump vllm.cpp and default MLX ON, gated to prefill
Bumps VLLM_CPP_VERSION from 9e1c9025 to eec09bed and turns VLLM_CPP_MLX back on.
These two must move together, which is why they are one commit.
Upstream now shape-gates the MLX provider to prefill: it declines m < 2, which is
exactly the decode GEMV. MLX's steel GEMM wins prefill, 524.5 ms of TTFT against
602 for the native path, but loses decode badly because the provider pays an
mx::eval synchronisation and an output memcpy on every call while decode makes
about 112 calls per token. Ungated it does both; gated it does only the good half.
Measured on an Apple M4 with Qwen3-1.7B-bf16 warm at p=512 g=128:
MLX gated to prefill (pin >= 89c46aeb) TTFT 524.5 ms 24.40 tok/s, 99.1% of MLX-LM
MLX ungated (older pins) TTFT 537 ms 12.7 tok/s
MLX off TTFT 602 ms 23.9 tok/s
This branch briefly defaulted the provider off, which was the correct call for an
ungated provider at the old pin. The gate is what makes on correct again, so the
pin and the flag are coupled: rolling VLLM_CPP_VERSION back before 89c46aeb while
leaving MLX on would select the middle row and roughly halve throughput. Both the
Makefile comment and the README state that dependency explicitly.
The bump also brings six Metal kernels landed upstream since the old pin — mma
prefill attention, a vectorised decode V accumulation, vectorised attention
staging, a fused qk-norm-RoPE preamble, a simdgroup-per-row softmax and a
simdgroup-per-head preamble — which take the non-MLX Metal path from 89.4% to
96.4% of MLX-LM on their own.
One caveat, recorded in the README: MLX's GEMM is not bit-identical to the native
kernel, so an MLX build produces a different greedy sequence than a non-MLX build.
That is a property of the provider rather than of the gate and predates this
packaging.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(vllm-cpp): correct the MLX-gated figure to 97.6%, from 99.1%
The previous commit quoted 99.1% of MLX-LM for the prefill-gated MLX build. That
figure divided by a two-run MLX-LM baseline, 27.135 and 27.744 generation tok/s
averaged to 27.44. Re-measured interleaved with ours over four ABBA blocks,
MLX-LM's decode is 27.848 with a 0.34% spread across six runs, so the 27.135 was
an outlier and averaging it in overstated us by roughly 1.5 points.
Corrected: the gated configuration is 24.37 tok/s, or 97.6% of MLX-LM, and the
MLX-off build is 23.9 tok/s or 95.9%. Prefill TTFT is unchanged at 524.5 ms
against MLX-LM's 532.6, so we remain about 1.5% faster there.
Nothing else changes. MLX still wins prefill and loses decode, the shape gate is
still the right disposition, and the pin and the flag are still coupled. The gate
is worth about 1.7 points over the MLX-off build rather than 2.7.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): pin MLX gate from mainline
The previous pin was a merge commit from the experimental C ABI v9 branch. Pin the same MLX prefill gate on upstream main so the backend build does not pull unrelated ABI v9 work into every platform variant.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): restore backend build portability
Keep the current master pin when enabling MLX so every backend variant builds against the known-good vllm.cpp revision. Suppress Apple clang’s GNU constant-folding diagnostic for Objective-C++ Metal compilation only, since upstream treats warnings as errors.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): demote MLX header VLA warning
MLX 0.29.3 headers trigger Apple clang's gnu-folding-constant diagnostic in the Objective-C++ provider. Keep the diagnostic visible while exempting only it from vllm.cpp's global warnings-as-errors policy.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): suppress MLX header VLA warning
Target-level Objective-C++ -Werror is appended after the directory flags, so a no-error demotion is re-promoted. Disable this single warning for the MLX header while keeping every other warning fatal.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): pin source-scoped MLX warning fix
Move the AppleClang warning exception into vllm.cpp where its target warning policy is defined, and pin LocalAI to that source-scoped fix.
Assisted-by: Codex:gpt-5
* fix(vllm-cpp): pin effective MLX warning suppression
The source-scoped no-error flag was overridden by the target warning policy. Pin the companion vllm.cpp change that disables only the MLX header diagnostic for its Objective-C++ translation unit.
Assisted-by: Codex:gpt-5
* fix(vllm-cpp): pin diagnostic pragma fix
Pin the companion vllm.cpp correction that scopes the AppleClang folding warning suppression inside the MLX translation unit, after command-line warning policy.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): pin remaining Darwin build fixes
Advance the MLX-enabled backend to the vllm.cpp revision already validated by the dependency update branch. This includes the feature guards and AppleClang pragma boundary needed by the Darwin build.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): pin MLX system dependency boundary
Pin the companion vllm.cpp change that models MLX as an imported system dependency, keeping third-party header diagnostics out of the project's warnings-as-errors policy while retaining fatal warnings for project sources.
Assisted-by: Codex:gpt-5 [Codex]
* fix(vllm-cpp): pin scoped MLX warning guard
Advance vllm.cpp to the companion fix that keeps MLX headers on a SYSTEM dependency and scopes AppleClang folding-constant suppression to the external includes.
Assisted-by: Codex:gpt-5 [systematic-debugging] [test-driven-development]
* fix(vllm-cpp): use available MLX wheel
MLX 0.29.3 is no longer available to the Darwin runner, so the backend build stopped before CMake. Pin the first available compatible wheel and keep the documented default in sync.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update antirez/ds4
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(ds4): link upstream CUDA MMQ objects
The updated ds4 CUDA object now calls into the vendored MMQ implementation. Build and link those objects into both the gRPC server and distributed worker.
Assisted-by: Codex:gpt-5 [Codex]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Wait for observable loop events instead of budgeting hundreds of milliseconds for scheduler timing. Keep a short bounded overlap observation for the two-leader exclusion check.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The blog has a deep post for 4.8 and a history post that covers the earlier
releases at summary altitude, but nothing in between. These five fill that
gap in the same shape as what-landed-in-localai-4-8: what the release was
for, runnable examples, and the limits that apply.
Every endpoint, CLI flag, env var and gallery entry is verified against the
matching release tag rather than taken from the release notes. That caught
two paths the published 3.10.0 notes got wrong: tracing is /api/traces, not
/api/v1/trace, and a stored response is fetched from /v1/responses/:id, not
/api/v1/responses/{response_id}.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* blog: rewrite the engines post without the AI tells
The HN thread on this post (item 49125065) spent most of its comments on the
writing rather than the engines. Readers quoted specific lines back as tells.
This is the same post with the same numbers, edited against the updated
no-ai-slop skill.
Every figure, table and link is unchanged, except that "27% of the memory"
is now the underlying 363 MB against 1328 MB from the table.
Two substantive framing fixes, both from the reply draft in
hn-reply-engines-post.md:
- vllm.cpp is no longer implied to be a speed win. The table is a tie, the
result is the install size, and the post now says so before a reader has to
work it out and post about it.
- Added one line on the language mix. Readers took the C++/Python/Go tree as
incoherence rather than as a Go core with per-ecosystem backends.
Cut throughout: the ledger metaphor ("what those ports buy", "not paid for in
throughput"), unearned framing ("the honest reading is", "has nothing to do
with"), the shape summary ("that is the general shape of these wins"),
confident deference ("people who are better at those models than we are"),
self-grading numbers ("a good result for a 66 MiB binary"), verbless
comparisons, three of the four exactness idioms, and the aphoristic headings
and verdicts. The double-tricolon summary is one plain clause now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* blog, website: same anti-slop sweep over the rest of the site
One-by-one pass over the other four posts and the site templates, with the
same rules used on the engines post. All figures, tables, links and PR
numbers are unchanged everywhere; the edits are to prose only.
apex-moe-quantization: ledger metaphors were the main issue, eight uses of
buy/cost/pay/spend for things that are not money. Also "the honest reading
is", "that is the comparison that matters", and two section-ending aphorisms
("Size is a speed knob as much as a memory knob", "Q6_K is the ceiling worth
paying for").
localai-since-march-2023: light touch, this one already reads like a person.
Removed "the curve is not the point", a "not the feature list, but the four
decisions" contrast, and two "X is what made / is the piece that" forms.
parakeet-cpp-asr-on-cpu: six exactness idioms across one post, "byte for
byte" twice, "character for character" twice, "byte-identical" twice and
"bit-identical" once, including in the title. Down to one, kept where the
precision is load-bearing. Also the "what end-of-utterance detection buys
you" heading and the "we say so rather than averaging it away" flex.
what-landed-in-localai-4-8: no changes. It is dense, flat and ends every
section on a PR number or a plain fact, which is the shape the other posts
should look like.
Site templates: "Most backends wrap somebody else's engine. These do not."
was the same contrast the engines post opened with. Also "Not a degraded mode
that technically runs", "A port only ships once it matches the original",
"Speed is the part we then go and win ... not a marketing run", and the last
"byte for byte" on the landing page.
Hugo builds clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* website: it is eighteen engines, not nineteen
Three places said nineteen: the /engines/ page description, the JUL 2026
timeline entry on the landing page, and the header comment in
data/engines.yaml.
Eighteen is right, confirmed two ways. The "Backends built by us" table in
the README has exactly 18 rows, and data/engines.yaml has 19 entries of which
one is apex-quant, which is a quantization recipe rather than an engine. The
two lists otherwise match name for name.
The yaml comment is the likely origin: it read "the nineteen native engines
the LocalAI team wrote, and the one quantization recipe that feeds them",
which counts apex-quant twice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
master does not compile:
vet: core/http/endpoints/mcp/localai_assistant_test.go:157:19:
method stubClient.ListScheduling already declared at
core/http/endpoints/mcp/localai_assistant_test.go:87:19
Two fixes for the same breakage landed. The four Scheduling methods were
already present at lines 87-99, in interface order after ListNodes, by
the time #11318 merged; #11318 appended its own copy after
GetRouterDecisions. The two blocks sit in different parts of the file, so
git merged both without a conflict and nothing flagged it.
Remove the appended copy and keep the one in interface order. Pure
deletion, no behaviour change.
Verified: go vet clean on ./core/http/endpoints/mcp/, and
go test ./core/http/endpoints/mcp/ passes.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The News page kept a hand-maintained "Highlights" list that had drifted:
it was missing all of 2025, duplicated the README's own news list, and
linked /features/middleware/ for a page that lives at operations/.
Both of its jobs already have owners. website/content/blog/ carries the
release write-ups and engineering notes, and GitHub Releases carries the
full changelog. Replace the list with a pointer at those two, so there is
one place to update instead of three.
The page keeps its url and front matter, so /docs/basics/news/ and the
root /basics/news/ redirect that .github/ci/gen-redirects.sh generates
both keep resolving.
Also drop the two contributor instructions in .agents that told authors
to add a whats-new.md bullet per feature: announcing a capability is the
release blog post's job, per .agents/preparing-a-release.md.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Write] [Bash]
Deploy site to GitHub Pages failed on five of the last eight master
pushes, always in the build job before Hugo runs:
Setup go version spec 1.22
...
go: downloading go1.26.0 (linux/amd64)
go: download go1.26.0: golang.org/toolchain@v0.0.1-go1.26.0.linux-amd64:
Get "https://proxy.golang.org/...": connect: network is unreachable
##[error]Command failed: go env GOPATH
The workflow pinned setup-go to 1.22 while go.mod declares go 1.26.0, so
the `go run ./.github/ci/modelslist.go` step that generates the gallery
page had to fetch the real toolchain from proxy.golang.org first. That
fetch is not reliably reachable from the runner, which is why the deploy
alternated between passing and failing rather than failing outright.
Track go.mod instead of a literal. The version the module needs is then
installed directly and there is no toolchain download to fail.
This matters beyond CI noise: the docs and the site, including the
release blog post, ship through this workflow.
Scoped deliberately to gh-pages, the workflow with the observed failure.
test-extra.yml pins 1.25.4 in a dozen places and is below go.mod for the
same reason, so those jobs also download a toolchain, but they are
currently green and rewriting twelve pins on a hunch risks more than it
fixes. Worth a follow-up.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
#11228 added ListScheduling, GetScheduling, SetScheduling and
DeleteScheduling to localaitools.LocalAIClient but did not update
stubClient, the hand-written test double in the mcp endpoints package.
The package therefore fails to typecheck, which takes out both lint and
tests on master:
cannot use stubClient{} as localaitools.LocalAIClient value in
argument to h.Initialize: stubClient does not implement
localaitools.LocalAIClient (missing method DeleteScheduling)
Red on 8f74f74b, fd4ec083 and 8a68f357; green on cd62e8ff, the commit
before.
Add the four methods with the same inert bodies the rest of the stub
uses. The real implementations are covered in the localaitools suites;
this double only exists so the holder can be constructed.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
chrome-audit.spec.js walks 25 routes in a single test, and has been the
UI E2E suite's failure on 5 of the last 6 master runs. It always dies the
same way, at the 30s per-test default:
Test timeout of 30000ms exceeded.
Error: page.waitForTimeout: Test timeout of 30000ms exceeded.
19 | await page.goto(route)
> 20 | await page.waitForTimeout(400)
The spec is new in 5cb0c1a8; the commit before it was green, and every
run since has been red on this file.
The failure is cumulative rather than one bad route. Across those runs
the clock runs out at line 19, 20 or 21 depending on where the loop
happens to be, and the timeout lands on waitForTimeout rather than on
goto, which is what running out of budget looks like as opposed to a
navigation that hangs. 30s over 25 routes is ~1.2s each, including a
deliberate 400ms settle, so there is very little headroom to begin with.
Give the test a budget proportional to its work: six seconds a route.
That absorbs a slow runner and still fails promptly if a route genuinely
hangs.
Verified: the spec passes on the current UI in 12.2s solo, and the full
suite passes 418 at 8 workers locally. What I could NOT do is reproduce
the CI timeout on this machine, which has 20 cores against the runner's
2 to 4; under synthetic CPU load it still finished in 13.5s. So the fix
is argued from the CI signature and the arithmetic, not from a local
repro, and the proof is this spec going green on the hosted runner.
Note test.setTimeout() has to be called inside the test body. At module
scope Playwright rejects it with "test.setTimeout() can only be called
from a test".
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Every CUDA sglang image failed in the 2026-08-02 full-matrix rebuild:
-gpu-nvidia-cuda-12-sglang, -gpu-nvidia-cuda-13-sglang and
-nvidia-l4t-cuda-13-arm64-sglang, all with the same build error.
Building cuda-tile==1.6.0rc3
x Failed to build `cuda-tile==1.6.0rc3`
ModuleNotFoundError: No module named 'wheel_stub'
hint: `cuda-tile` (v1.6.0rc3) was included because `sglang` (v0.5.16)
depends on `flashinfer-python` (v0.6.14) which depends on `cuda-tile`
This is the failure mode requirements-cublas1{2,3}-after.txt already
carries an nvidia-modelopt bound for, arriving through a different
package. install.sh passes a global --prerelease=allow, which is
load-bearing for flash-attn-4, so an unbounded dependency resolves to a
prerelease; cuda-tile 1.6.0rc3's build backend imports wheel_stub without
declaring it in build-system.requires; --no-build-isolation means nothing
provides it, and the build dies.
Nothing in this repo changed. cuda-tile published 1.6.0rc1 and rc3 and
the weekly cron picked them up, which is the drift that job exists to
catch.
Bound the one package rather than dropping the global flag, matching the
existing precedent. 1.5.0 is the newest stable release, so <1.6 takes the
last good one. l4t13 gets the same bound: it installs plain sglang rather
than sglang[all], but flashinfer-python is a dependency of both.
NOT VERIFIED LOCALLY: reproducing this needs a CUDA docker build, which
this machine cannot run. The diagnosis is from the CI log and the
resolver's own hint, and the change follows a fix already proven in these
same files. CI on this PR is the check that matters.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Since #11255 and #11276 every GPU image also builds ggml's CPU_ALL_VARIANTS
matrix, so a partial offload uses the host's SIMD kernels. That works
everywhere except SYCL, where the Makefile compiles the whole tree with
icpx -fsycl: icpx never finishes ggml-cpu/arch/x86/repack.cpp at
-march=sapphirerapids. In run 30765516644 both sycl_f16 and sycl_f32 stopped
at that translation unit and sat there for 5h30m with a single compile in
flight until GitHub killed the job at its 6h limit, and turboquant's f16 job
lost its runner outright. gcc compiles the same file in seconds in the vulkan
and CPU jobs of the same run, so the CPU variant matrix is only unbuildable
under icpx.
Route SYCL back to the portable fallback binary, which is what these images
shipped before #11255. run.sh already prefers *-cpu-all when present and falls
back otherwise, so nothing else has to change.
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(api): add text moderation endpoint
Add an OpenAI-compatible /v1/moderations endpoint backed by constrained local text generation. Register its auth and discovery surfaces, document the text-only MVP, and cover response shaping and access control.
Assisted-by: Codex:gpt-5
* test(mcp): update assistant client stub
Keep the LocalAI Assistant holder test stub aligned with the scheduling methods added to LocalAIClient so repository-wide type checking succeeds.\n\nAssisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(api): add POST /v1/images/upscale endpoint
Add a new image upscaling endpoint that accepts a source image and
returns an upscaled version. Supports selectable upscaler models
(e.g. realesrgan) and a configurable scale factor (2x or 4x).
- backend.proto: add UpscaleImage RPC and UpscaleImageRequest message
- pkg/grpc: implement UpscaleImage in Backend interface, client, server
and embed shim
- core/backend/upscale.go: new backend helper (mirrors ImageGeneration)
- core/http/endpoints/openai/upscale.go: new multipart/form-data handler
- core/http/routes/openai.go: register POST /v1/images/upscale
- core/http/auth/features.go: gate upscale routes under FeatureImages
- backend/python/diffusers/backend.py: implement UpscaleImage — uses
diffusers upscale pipeline when loaded, falls back to Lanczos resize
* fix(grpc): add UpscaleImage stub to Base backend
All Go backends embedding Base now satisfy the AIModel interface
without needing to implement UpscaleImage explicitly.
* fix(images): complete upscale endpoint integration
Store generated upscales under the served images directory, validate scale factors, document and advertise the endpoint, and add a functional Stable Diffusion x4 gallery model.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Give gallery operations distinct pause and cancel paths. Pause preserves partial download data so reinstalling the same model or backend resumes through HTTP Range, while cancel keeps its destructive semantics. Surface the action in the Activity UI and document the API behavior.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add multilingual 1B and 8B Q4_K_M GGUF embedding entries and link them as variants for automatic memory-aware selection.
Assisted-by: Codex:gpt-5 [web]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
fix(gallery): repair DeepSeek V4 0731 entry
Use the official single-file ggml-org MXFP4 artifact with its verified SHA256 and route it through llama.cpp instead of treating an unsloth repository page as a ds4 model file.
Assisted-by: Codex:gpt-5 [Hugging Face API]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add Q4_K_M and Q8_0 GGUF builds for the trending Instella-MoE-16B-A3B-Think model, with host-selectable variant metadata and verified Hugging Face LFS hashes.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(ui): give the Operate overview real numbers and traces a latency shape
First two items from a component-by-component comparison against the mocks.
The pattern that audit found: everything newly built matched, everything
pre-existing got the palette but not the layout, and an "absent rather than
empty" rule hid most of the overview exactly when someone was looking at an
idle installation.
**The headline grid is always rendered**, including at zero, with a fourth cell
for host memory. Hiding it removed the page's structure precisely when it was
most likely to be read, and "0 failed" is information — an absent panel is not.
The quiet case is now said in a line underneath instead of by showing nothing.
**The sections state counts** rather than listing their destinations: backends,
models, updates and running operations instead of the words "Usage and traces".
That needed installed backend and model counts in the summary context, which
are two more cheap reads on the poll that was already running.
**Traces rows carry latency as a bar as well as a figure**, scaled against the
slowest request currently in view and turning amber past two seconds. The table
had no latency column at all — the number was buried in the expanded detail, so
the shape of the tail was invisible while scanning. Scaling against the view
rather than an absolute ceiling is deliberate: what matters when reading a page
of traces is which of these are the outliers, and an absolute scale flattens
every row on a fast installation into nothing.
Full e2e suite: 409 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): name the engine on Home's resident models, and add jump-back-in
Third item from the mock comparison.
The mock showed each resident model with the engine serving it. /system carried
only the id, so the audit recorded this as blocked on a server field — but the
config loader is already in scope where that response is built, so it is one
lookup. SysInfoModel gains an optional `backend`, resolved from the model's
config and omitted rather than guessed when there is none (a loose file, or a
config since removed). Home renders the column blank in that case; the test
pins both halves of that.
Memory per model stays out. It is not one lookup — it would mean asking each
backend process — and inventing a number beside a real one is worse than
leaving the column off.
"Jump back in" is the block the mock had and Home did not. The quick-links row
above it is a set of first-run actions; these are the three places someone
returns to, each stated with what it currently holds rather than as a bare
label.
Go: routes suite passes. Full e2e suite: 412 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): rank the recommended models as lanes instead of equal cards
The hardware recommendations were a grid of equally-weighted cards. The list is
already sorted by fit, and a grid throws that order away: three cards side by
side say "pick one", when the page has actually formed an opinion about which
one.
They are lanes now, read top to bottom in fit order, with the leader carrying
the single amber "Best fit" label and the rest marked "Also fits". One opinion
per page — the alternatives are alternatives, not runners-up each worth their
own colour, which is how a strip of coloured badges ends up meaning nothing.
Below 720px the size and VRAM columns drop and the lane keeps the name and the
install action, which are the two things a narrow screen needs.
The existing panel spec moves off .rec-models-item onto .lane rather than being
deleted; dismissal, collapse, keyboard operation and install all still pass
unchanged, and there is a new assertion that exactly one row is called out.
Full e2e suite: 413 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): drop capsule chips app-wide, and un-break the empty voice library
**Pills are gone.** A capsule radius reads as a tag floating on the surface,
which fights a system whose structure is hairlines and square corners — and
with chips on Discover, Host, Activity and the biometrics pages, "some pages
have pills" was the real inconsistency rather than any one page.
Sixteen selectors move to the small radius: filter buttons, tab pills, activity
and biometrics chips, file and count badges, the jump-to-latest control, the
nav badge. Round *buttons* keep their circle — .lightbox__nav and
.home-send-btn are circles, not capsules — as do every progress track, status
dot and avatar, which are round because they are round, not because they are
tags.
**The empty voice library was unusable.** `.voice-library-empty` sets
min-height: 430px, border: 0 and background: transparent — a description of the
empty PANEL — and it had been attached to the action instead. The create button
was therefore a 430px transparent box that pushed itself out of the panel and
could not be seen. Moved onto the container it describes, which now centres its
action rather than letting it fall off the bottom. Same class-mangling shape as
the Agents header fixed earlier.
Full e2e suite: 416 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): put Host's headline figures on the shared hairline strip
Host had shadowed, clickable StatCards above a page that already has a rail, a
pane and a tab bar — a second dashboard language on one screen, and a different
one again from the figures inside its own detail pane.
The Operate overview's figure grid is generalised into a shared `.stat-strip`
and Host adopts it, so the two pages read as one system: same cell, same figure
scale, same tone vocabulary, and the same hairline grid the split-view StatGrid
already uses. The cells stay clickable and still route into the tab and filter
they describe, because a count is worth more when it is also the way to the
thing counted.
Tone is spent only where the number means something — running and updates when
non-zero — since a strip where every cell is coloured has no emphasis left.
Two bugs made on the way, both now covered:
- The first version put `<button>` elements inside a `<dl>` with `<dt>`/`<dd>`
inside the buttons. Neither is valid, the browser re-parents both, and the
cells collapsed. These cells are a set of controls, so a plain container of
buttons is also the honest markup.
- Even correct, the strip rendered 2px tall: `.page--app` is a flex column
whose split view takes flex:1, so a child with no intrinsic minimum is shrunk
away. The old cards survived only because `.stat-card` carried
min-height:96px. The strip now declines to shrink, with a test pinning it.
The stat-card specs are retargeted rather than deleted: they were written to
guard a class collision on a page that no longer uses cards, so they now guard
the strip's labels and its height.
Full e2e suite: 417 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): make Backends notices an edge rather than a filled card
The install and upgrade banners were tinted cards with a full border. A filled
panel makes every notice shout at the weight of an error, which is how notices
stop being read — and Backends shows one on most visits, so it was shouting
routinely.
They are now a hairline with a coloured left edge, the same treatment the
Operate overview gives rows that want a decision, so "this needs you" looks the
same wherever it appears. Counts in the notice take the monospace tabular
figures the rest of the console uses.
Also drops the last inline style on the page, and refreshes the inline-style
baseline, which has read 624 against a real count since #11288 landed. The gate
exits 0 either way, so nothing was failing — but a baseline 86 above the truth
would have let that many inline styles back in unnoticed. Now at 538, which
tightens the ratchet rather than loosening it.
The spec creates the upgrade it asserts on rather than skipping when the mock
has no notice: a test that skips is a test that proves nothing.
Full e2e suite: 418 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): finish the mock parity list, and stop hiding the recommendations
The last two items from the audit, plus a correction.
**Discover's use-case shelf is lanes.** These are a list of ways in, read in
order; a grid of equal cards asks the reader to compare them, which is not the
choice on offer.
**The request panel reaches every generator.** Video, 3D, Sound and Audio FX
join Images and Speech, so each one teaches its own endpoint rather than two of
six doing it. Audio FX records the fields that shape the request rather than
the bytes, since its payload is multipart.
**Recommendations no longer collapse themselves.** They were folded away by
default once anything was installed. That is the page's one opinion about this
host, and an opinion hidden by default is one the reader never gets. Someone
who disagrees can still collapse it and that choice is remembered — the
difference is that we no longer make it for them. Three specs asserted the old
default and now assert the new one.
The use-case heading also sat a line's width from the text it introduces, so
the two read as one paragraph. It has air under it now, and the shelf is
separated from the recommendations above it.
Two tests removed rather than kept: a generator loop whose only real assertion
was `expect(endpoint.length).toBeGreaterThan(0)`, and an earlier card-gap guard
that could only skip. A test that cannot fail is worse than no test, because it
reads as coverage.
Full e2e suite: 418 passed, 4 skipped. Inline styles at baseline.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): make the Host figures legible and give the strip its spacing back
Three defects introduced by the Host redesign, all found by looking at the
running app rather than by the suite.
**The figures were invisible.** "Running now" and "Updates available" rendered
pure black on the dark ground. Two causes compounding: the `--muted` tone alias
never landed, because the source rule has extra spaces before its brace and the
exact-match edit missed it silently; and a `<button>` does not inherit colour,
so with no tone rule the value fell back to the user agent's `buttontext`.
Both fixed, and a test now fails on any figure computing to pure black.
**The strip sat flush against the resources panel.** `.stat-strip` declares
`margin: 0 0 ...` and is declared later in the file than `.manage-summary`, so
the shorthand quietly won and the top margin became zero. Raised to
`.stat-strip.manage-summary` so it beats the shorthand on specificity rather
than on declaration order, which is the kind of thing that breaks again the
next time a rule moves.
**Discover's use-case heading had a doubled gap.** `.zero-pane` is a flex
column that already separates its children; adding a margin on top of the gap
stacked the two. The margin is gone and the heading keeps only its own breathing
room.
Full e2e suite: 420 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): make Studio's tabs path segments rather than a query parameter
`/app/studio?tab=images` reads like a filter applied to a page. It is
navigation: a different generator, with its own state and its own deep link. It
is now `/app/studio/images`, with the overview at `/app/studio`.
Legacy `?tab=` links are redirected once to the path form, replacing the
history entry so Back does not bounce between two spellings of the same place.
Bookmarks and older links keep working and land on the canonical URL rather
than a second version of it, which is the part worth having a test for.
The nine `?tab=` references were all in specs, none in docs, so the migration
is contained. They move to paths, and a new spec pins the redirect.
Full e2e suite: 421 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): make the hardware recommendation a section, not a dismissable card
It was a bordered card with a collapse control and a close button, sitting
inside a pane that is otherwise hairline sections. Two problems: it read as
something bolted onto the page rather than part of it, and treating it as an
interruption to be shut is the wrong frame for the one thing the page has to
say about the machine it is running on.
It is now a plain section with the same heading treatment as the shelves below
it. The collapse state, the dismissal, their storage keys and the legacy key
read for backwards compatibility all go with it, along with the installedCount
prop that existed only to pick a default collapse.
Five specs described behaviour that no longer exists and are removed rather
than adjusted — collapsing, dismissing, persistence of both, and the toggle's
keyboard handling. One new spec asserts the replacement contract: no control
with aria-expanded, no dismiss, and no card border.
Full e2e suite: 416 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): restore every stripped icon and every default-chrome control
You reported two broken icons. They were not two: an earlier automated edit had
stripped the `fa-*` class from twenty `<i>` elements across eleven pages, and an
`<i>` with no icon class renders nothing at all. Settings' save button, the
voice-profile back link, and eighteen others — agent row actions, task and job
buttons, import and create actions — were all drawing empty space.
Each is restored from its own context rather than a blanket icon: the agent row
gets pause/play, pen, comments, file-export and trash; the fine-tune toggle
swaps plus for xmark as it opens; the P2P documentation link gets the
external-link glyph.
The same edit left controls without their classes. Fine-tune's "Import config"
was rendering in the browser's own chrome, and `.p2p-cmd__copy` set a border
but no background, so it fell back to `buttonface` — a pale grey chip on a dark
command block. FineTune's "New job" also had its icon classes folded into the
button's className, the same mangling already fixed on the Agents header.
Rather than fix the reported two and wait for the next report, this adds a
standing audit: twenty-five routes are walked and the test fails on any visible
control rendering with user-agent chrome, or any `<i>` without an `fa-*` class.
It found the three remaining cases after the first sweep, and it is the reason
the next one cannot ship quietly.
Full e2e suite: 417 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): give Operate a front door and fold six rail groups into four
Opening Operate ran firstVisiblePath() and landed on Backends, because
Backends happens to be written first in operateConsole.groups. The section
that should answer "is anything wrong" opened on a package manager, and
nothing was reported until you visited it.
Adds /app/operate. Its one irreplaceable block is "Needs attention", which
is empty when nothing is wrong and says so in a line rather than rendering a
reassuring green panel. It collects stale backends, failed operations and
unhealthy nodes. Everything else on the page is a summary you could already
assemble by visiting four others.
The rail regroups from six headings to four: Inference and Activity were both
"the runtime right now", Access and System were both administration. No
destination is removed and no gate changes, so isConsoleItemVisible and
consolePaths are untouched. Overview leads the first group, which is what
makes firstVisiblePath() return it without knowing it exists.
Rail entries now carry a signal beside the label. This does not replace the
sidebar badge and is not built as if it does: the badge stays on the
always-visible sidebar entry for the reason recorded in Sidebar.jsx, that the
rail exists only on Operate routes and can be collapsed. The signals are
orientation while inside Operate, so they are aria-hidden and nothing urgent
depends on them alone.
OperateSummaryContext polls once for the whole console, following
OperationsContext, which exists because per-consumer setInterval against one
endpoint was the defect it fixed. It is mounted by ConsoleLayout for the
Operate console only, so "poll only while in Operate" needs no route check.
Built on usePolling, so it pauses on a hidden tab. Operations are read from
OperationsContext rather than polled a second time, and each source degrades
to no-signal on its own so one dead endpoint cannot blank the rest. It reads
the cached GET /api/backends/upgrades and never the POST that forces a real
registry check.
Traces and Usage get no signal yet: /api/traces returns the list, so a count
would mean fetching every trace to render one number. A counts endpoint is
the honest fix and is scoped separately.
Full e2e suite green (369 passed, 4 skipped), including a render-smoke entry
for the new route.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): open Studio on what this machine can actually make
Studio was a tab strip over six generators that opened on Images, which was
never a decision, only the first entry in BASE_TABS. Nothing said which
modalities this installation could run, so the way to learn that video had no
model was to pick the tab and find an empty select.
Adds an overview tab and makes it the fallback. Explicit tabs still win, so
existing deep links keep working; anything unrecognised or gated now lands on
the overview rather than Images.
Each tab carries a dot: filled when an installed model advertises that
modality, hollow when nothing serves it. That is the feature in one detail,
turning the strip from navigation into a report of what the machine can do
before anything is clicked. The dot is aria-hidden because the overview states
the same facts in words and the dots change as models load.
Two kinds of unavailable, which had to stop looking alike:
- switched off, via a permission: no tab and no lane, unchanged
- available with no model: a lane, and a route to installing one
Studio now owns one MODALITIES table so the tab strip and the overview cannot
disagree about what exists, and calls useModels() once, unfiltered, grouping in
the browser. useModels(capability) fetches the whole list and filters locally,
so a hook per modality would have been six identical requests to
/api/models/capabilities on every mount. There is a test for that.
Recent outputs read every localStorage store through a new
readAllMediaHistory(), which avoids mounting five hooks that carry save timers
the overview has no use for. 3D is read separately through use3DHistory rather
than folded in: its entries are GLB blobs in IndexedDB, so they cannot come
from the same synchronous read.
Typical cost is the median of this machine's own history, not a guess, and
renders as a dash when there is nothing to go on.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): stop the stat cards and the console rail breaking on small screens
Two unrelated causes behind one report that /app/manage looks wrong when the
window is narrow.
The stat cards were being laid out by the wrong rule at every width. Two
different components both claimed `.stat-grid`: the dashboard card strip that
holds .stat-card children, and the detail-pane StatGrid the split views
introduced further down App.css. Being later, the second won every shared
property, so the cards got its 120px columns and its 1px hairline gap in place
of their own 180px columns and spacing-md. Four cards were packed onto a row
that fits two, labels wrapped to three lines and clipped, and the icon crowded
the value. Renamed the strip to `.stat-cards`, after the children it actually
holds, which also removes the mismatch of a `.stat-grid` container full of
`.stat-card`s. The split-view component keeps `.stat-grid` and its BEM parts.
The expanded console rail had no bounded height. Thirteen destinations stacked
in one column is taller than a phone, so opening the menu pushed the page's own
heading past the fold: the menu replaced the page rather than annotating it.
Capped at 55vh with internal scrolling below 768px, so the content behind stays
reachable.
Both are asserted on behaviour rather than markup: no stat-card label may be
clipped, the card gap must not be the detail pane's hairline, and expanding the
rail must leave the page heading on screen.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): retemper the palette to localai.io and add the lane primitive
The token half of the style transfer, plus the shared list idiom the two
overviews had each grown their own copy of.
theme.css moves from Nord to the website's palette, variable names preserved so
every consumer moves with it: ground #13171f -> #0d1117, accent frost cyan
#88c0d0 -> action blue #4f8cff, success sage -> mint #56d6a4, warning -> the
amber #f1b95d the site spends only on the thing asking for a decision. Eyebrows
go mint. Dividers become an opaque #29384a hairline rather than alpha over a
varying surface, which is what makes stacked surfaces read crisply on the site.
Light is derived, not inverted. The site ships one theme and never had to
answer this, but the app does: blue darkens to #2f62d8, mint to #0d8b60 and
amber to #8a5d0b, all clearing 4.5:1 on a cool paper ground, where the
dark-mode values sit near 2:1. Same three roles, different values.
Three files restate the palette because CSS variables cannot reach them:
cmTheme.js (the whole CodeMirror theme), VoiceVisualizer and WaveformPlayer
(canvas). Left alone they would have quietly kept the app half-Nord.
The `.lane` primitive replaces the near-identical row CSS that OperateOverview
and StudioOverview had each written: a full-bleed row on a hairline that insets
on hover, with no card and no shadow. Callers supply only the column template.
Both pages now use it, along with `.lane-head` for section rhythm and a
`.page-pad` container for top-level pages outside a console shell — without
which Studio sat flush against the sidebar with its eyebrow clipped.
Studio's tab strip wraps rather than running off the edge at narrow widths.
Full e2e suite: 386 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): put Home's resident models on lanes and give the footer one line
Home's status line was three chips saying a thing was true. It now reports
figures: how many models are resident, how many nodes are healthy, what share
of memory is in use, set in tabular monospace so the digits line up. A chip
answers whether; a figure answers how much, which is what someone opening the
page at a glance is after.
Resident models move from status chips to lanes, with the id set in a new
`.lane__name--id` because an id is something you might type or paste and the UI
face makes it read as a label. /api/system-information carries only the id, so
there is deliberately no backend or memory column: inventing one would mean a
server change this does not make.
The footer was three centred rows and cost the bottom sixth of every page for
chrome. It is one line now, version left and links right, wrapping to centred
when the viewport is too narrow to hold both. Every link it had, it keeps.
Full e2e suite: 392 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): correct three contrast failures and stop a guaranteed-404 poll
A contrast audit of the new palette found three values below WCAG AA, one of
which the previous commit message claimed was fine:
- White on the #4f8cff button is 3.22:1, which is large-text only. The website
does exactly this, but a button label in an app is not large text, so the
label goes to dark ink at 5.88:1. Light mode keeps white, which is 5.44:1 on
its darker blue.
- Light-mode success was 4.08:1 on paper, not the 4.5 claimed. Darkened to
#0a734f, 5.56:1.
- Nord red was already 4.28:1 on raised surfaces, a pre-existing miss carried
over unexamined. Lifted to #c96f78, 5.02:1.
Lanes gain the two states they were missing: a 44px target on coarse pointers,
matching what EntityRail already does so the two list idioms feel the same
under a thumb, and a reduced-motion variant that keeps the background feedback
while dropping the hover inset, which is a position change.
The Operate summary no longer asks for /api/nodes on a single-node install. The
cluster API answers 503 when distributed mode is off, so it was a guaranteed
miss every fifteen seconds; it is now gated on useDistributedMode, the same
condition the rail already uses for the Nodes entry.
Full e2e suite: 392 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): restore the gap between overview blocks, and stop claiming zero nodes
Two defects a design review surfaced.
`.lane-head:first-child { margin-top: 0 }` was meant to stop the first block on
a page carrying a top margin. But every <section> makes its lane-head a first
child, so the reset applied to all of them and the gap between blocks vanished:
"Sections" sat flush against the attention row above it. The header supplies its
own bottom margin, so a uniform top margin is correct everywhere.
The Cluster summary read "0 nodes" on a single-node install, which looks like a
fault when the cluster API is simply switched off. It now says "Single node".
Full e2e suite: 392 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): open dark by default, and stop clipping the collapsed sidebar footer
Dark is the identity rather than a preference: localai.io ships one theme and
it is this one, so an install should look like LocalAI before anyone has chosen
anything. The OS setting no longer selects light on first load. The toggle
still does, and a stored choice wins forever after, which the tests assert
both ways.
The collapsed sidebar footer stacked its controls but kept the expanded row's
inline padding, so their edges were clipped against the 51px rail.
Full e2e suite: 394 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(api): count traces server-side and give the Operate overview real totals
The overview's headline block had no source. /api/traces returns the trace
list, so "37 errors in 24h" meant fetching every buffered exchange to count it
in the browser — waste that grows with the buffer, to produce three integers.
Adds GET /api/traces/summary: totals, failures, p95 and a bucketed series for
sparklines, over a window that defaults to 24 hours and is capped at a week.
Deliberate calls, each with a spec:
- A 4xx is the caller getting it wrong, not the installation being unhealthy,
so only 5xx and transport errors count as failures.
- p95 is a nearest-rank percentile rather than the slowest request, which is
what a max would report and what makes latency panels lie.
- Buckets are oldest-first so a sparkline reads left to right, and the slice is
never nil: nil serialises as null and breaks .map() on the other side, which
is a silent runtime error rather than an empty chart.
- Exchanges outside the window are not counted at all.
The route is registered before /api/traces/:id so "summary" is not captured as
a trace ID.
On the client, Traces and Usage gain the rail signals they were shipped
without, the Observability section summary now states counts instead of listing
its destinations, and an installation that has served nothing says so rather
than showing three zeroes dressed as telemetry.
Sparkline is a bare stroke with an emphasised endpoint and no axes: the figure
above it already states the value, so its only job is the shape.
Go: 185 middleware specs pass. Full e2e suite: 396 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): stop the memory chart calling a trade-off an error
The VRAM-by-context chart rendered any build over the limit in error red, and
escalated the verdict to the error tone as soon as two context sizes crossed
it. But an over-limit build still installs — #11288 keeps a test on exactly
that — so red overstates what is happening. A model that fits at 32k and not
64k is a trade-off, not a fault.
Over-limit bars and the limit line now use the warning tone, which is the
constraint colour used everywhere else in this branch: know what you are doing,
not you may not. The error tone is reserved for "fits nowhere", where the model
genuinely cannot run on this host.
Full e2e suite: 397 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): give the new surfaces orchestrated motion
Uses the reveal system already in the codebase rather than adding a library:
pageReveal, .reveal-stagger and staggerStyle() were built for exactly this, and
anime.js would be ~17KB duplicating four lines of CSS for list reveals.
The overview's headline figures, attention rows and section lanes stagger in,
as do Studio's modality lanes and recent outputs, so a page assembles in the
order it is read instead of appearing all at once.
Two additions beyond stagger. Rail signals transition on opacity when a poll
lands, so a number changing reads as an update rather than a jump cut, and it
stays on the compositor so it cannot reflow the rail. The attention block
animates its left edge in — the one thing on the page that should announce
itself, and on the border rather than the text so nothing moves under a reader.
Both are dropped entirely under prefers-reduced-motion, alongside the lane
hover inset already handled.
Full e2e suite: 397 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): put the generators on a hairline field stack and record the request
The workbench treatment from the mocks, applied where it costs least: both
changes land on shared surfaces, so all six generators get them at once rather
than drifting apart page by page.
The control column stops being a shadowed card of boxed groups and becomes a
hairline field stack — the panel is the page's left half, not an object
floating on it — with uppercase micro-labels matching the eyebrow treatment
used elsewhere. Because .media-controls is shared, Images, Video, 3D, Speech,
Sound and Audio FX all move together.
RequestPanel shows the request the form actually built, with a copy-as-curl.
LocalAI is API-first and Studio is the best place in the app to teach its own
endpoints: the form stops being a black box, and a result worth keeping can be
reproduced from a shell without reverse-engineering which fields the page sent.
It records what was sent rather than what the form currently holds, and renders
nothing until a request has been made — a panel describing a request nobody
made is a tutorial, not a record. Wired into Images and Speech.
Full e2e suite: 401 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): make Chat a transcript instead of a bubble thread
Rounded, filled, asymmetric bubbles fight a system built on hairlines, and they
carry the speaker in shape and side rather than in words. The assistant side
had already given up its bubble; this finishes the job.
Both roles now run full width down one column, separated by a rule, each with a
mono role label. The user turn keeps a left edge in the action tone so the two
are still told apart at a glance, without a fill or a corner radius. The
avatars go: the accent and the label carry the speaker, so the glyph was
decoration once neither side had a bubble.
Saying who is speaking in words rather than in geometry is also what survives
being read aloud, printed, or looked at by someone who cannot pick the sides
apart by colour.
Full e2e suite: 404 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): dress the API reference in LocalAI's palette
The Swagger page was the last surface still shipping in someone else's colours,
which is conspicuous now that everything it links from is dark.
Swagger UI has no theming hook, so rather than fork it we serve our own index
ahead of the library's wildcard and restate the palette over its stylesheet.
The library's own bundle and assets are still what load, so a swagger-ui
upgrade cannot silently break the page — this is a skin, not a fork.
Two things needed real care. Swagger tints the entire operation row per method
via .opblock.opblock-post and friends, so the palette had to match that
specificity rather than reach for !important; the method now lives on one edge
instead of washing across the row, because a page where every row is a status
colour has no status colour left. And the filled method chip put white on pale
green, which was the least readable thing on the page — it is an outlined mono
chip now, carrying the method in its border and text.
Palette values are copied from theme.css rather than referenced: this page is
served by Go and never sees the app's CSS. The comment says so, and says to
keep them in step.
Go: routes and middleware suites pass. Full e2e suite: 405 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): make tall split-view pages reachable, repair the Agents header, scale titles
Three things found by actually using the app rather than measuring it.
**Host was unusable.** The shell above a split view is overflow:hidden so the
document cannot grow, which left anything taller than the viewport simply
unreachable — and Host stacks a resources card, four stat cards and a tab bar
above its split, so the bottom of the pane fell off at every window height with
nothing to scroll. Every sweep I ran for this was horizontal, which is why it
kept coming back clean.
The page now scrolls inside the pinned shell. The pane keeps its own scroller:
letting it grow instead pushes the document taller and stretches the rail to
match, which is the regression e2e/discover-height.spec.js exists to catch, and
which the first version of this fix duly caused.
**The Agents header controls were unstyled** — "Create Agent" was rendering
with the browser's default chrome. The markup had been mangled at some point:
six unrelated classes merged into one string on the link, and the label and
button left with none at all and empty icons. Repaired, with the inline flex
replaced by a shared .header-actions class.
**Page titles take the editorial scale from the site**: larger, tracked at
-0.04em, on a line height near 1, so a two-word title reads as a statement
rather than a label. The typeface is unchanged — DESIGN.md keeps the existing
type system — so the whole difference is scale, tracking and leading, which is
where the site gets its voice from. This was the biggest reason the running app
still did not look like the mocks.
Full e2e suite: 404 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Follow-up to #11288, which warmed the VRAM estimate caches at startup and left
the variant picker paying its own way.
Describing an entry's variants probes the weight files of every build it
offers, so the first time a model is opened costs 1.2-1.9s against a cold
cache. That is the same cost as an estimate wearing a different hat, and it
lands in the same caches underneath, so it belongs in the same pass rather than
in a second mechanism.
The warm-up now describes variants for the entries it walks. Entries that
declare none cost nothing: the call is gated on HasVariants rather than
attempted and discarded. The host resolve env is derived once for the run,
since it describes the machine rather than the entry.
Failure handling matches the estimate half. An entry whose variants cannot be
described is logged at debug and skipped, and the estimate for that same entry
is unaffected, because neither half is allowed to fail the other.
Measured against a live instance with 1,595 models, first ever call to
/api/models/variants/:id after a cold boot:
before 1.2-1.9s
after 2ms
The warm-up's own cost barely moves: 3m0s to 3m19s for 300 entries, of which
40 declared variants. It stays bounded by the same knobs, and
LOCALAI_VRAM_WARM_LIMIT=0 still turns the whole thing off.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): rename the Install Models nav entry to Discover
"Install Models" named the action rather than the destination, and it was
the only multi-word entry in a rail of one-word ones (Home, Chat, Studio,
Talk, Build, Operate). A bare "Models" was the obvious fix but it collides
with the installed-models view under Host, which is a different page for a
different job.
"Discover" keeps the rhythm and says what the page is for. The icon moves
from a download arrow to a compass for the same reason: the page is browsed
before it is installed from.
Translated in all seven locales rather than left to fall back, so a locale
switch does not leave the entry in English.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(ui): replace the gallery table with a rail and a detail pane
The eight-column table was not the real problem; the click-to-expand row
underneath it was. Variants, files and a VRAM estimate never fitted inside a
<tr>, so they were pushed into a drawer that could hold one model at a time,
could not be linked to, and had no room to say anything useful.
The gallery is now a rail to scan and a pane that answers. The pane has two
states and no third: with nothing selected it is the discovery page, and with
a model selected it is that model's detail. Selection lives in the URL, so a
model is linkable and Back steps out of the detail instead of off the page.
The rail groups by capability while browsing and flattens to results the
moment a term is typed. That is a rule rather than a toggle: once someone has
said what they are looking for, the buckets are between them and the answer,
and making the user choose would be handing them our problem.
The detail pane plots VRAM against context length with the host's own limit
drawn across it. This is new information, not a restyle. A single number
invites "so will it run?", and the honest answer is usually "yes, up to a 32k
context", which is a shape rather than a number. The estimates were already
fetched for every context size, so it costs no new request. Backends that
take no context length say so instead of being given a meaningless chart, and
a host with no GPU gets no chart at all rather than bars with nothing to
compare against.
The split-button variant menu goes with the actions column. The pane lists
every build with its backend, quantization, size, fit and a details
disclosure, each installable, which is what the dropdown was a cramped
substitute for. Its tests move onto that list; the three contracts it alone
carried (fetch-once caching, the loading state, an unfit build staying
installable) are backfilled against the pane.
RecommendedModels moves inside the pane, where it has the width to argue for
a model instead of listing one, and keeps its own dismissal and collapse.
Rail entries carry no description. Two lines is the budget and the second is
better spent on whether the thing will run; the stripped-Markdown contract
moves to the pane's lede, tooltip included.
e2e: 123 passing across models-gallery, navigation, recommended-panel,
model-artifact-operation, operations-strip and page-render-smoke. Inline
styles in Models.jsx drop from 82 to 41.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* refactor(ui): extract the split view into shared components
Discover shipped its rail, pane and detail header as private functions inside
Models.jsx. Backends and Host have the same defect and want the same shape, so
leaving them there guarantees three rails that drift.
SplitView, EntityRail, DetailHeader and StatGrid now live under
components/split/. EntityRail is deliberately data-driven: a surface maps its
own entity onto { id, name, icon, meta, stripe, groupId } and keeps its
vocabulary to itself, which is what stops the rail learning about models,
backends and loaded state all at once.
The CSS moves with it. What was .discover__rail is .entity-rail, .discover__
pane is .split-view__pane and so on, because a class named after one page is a
lie on the next two. Only what is genuinely Discover's stays behind the old
prefix: the shelves, the hero and the VRAM-by-context chart.
Two additions the shared rail needs and Discover did not: a state stripe, for
surfaces read by condition before they are read by name, and an empty label.
Discover passes neither.
No behaviour change. e2e 100 passing across models-gallery, navigation and
models-recommended-panel.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(ui): put the backend gallery on the split view
Same defect as the model gallery, so the same shape: a seven-column table over
a click-to-expand row that was the only place the repository, licence, tags and
links could go.
The rail groups backends by the use case they serve, sharing Discover's
taxonomy on purpose: a backend is the runtime a use case needs, so "vision"
ought to mean the same thing one level down. It flattens on a query for the
same reason it does on Discover.
The zero state is the one real departure. A backend's fitness is not free
memory, it is the accelerator and platform it was built for, so the pane leads
with what this host is, then what is not installed yet, then whether anything
installed has gone stale. The table listed 37 runtimes and left "which of these
can even run here" entirely to the reader.
Distribution moves into the pane, which is the one thing a row could never
carry: which nodes hold a copy and which do not, with the install-on-more
control next to it rather than squeezed against a chip.
The distributed and target-node action logic is unchanged, including the guard
that keeps a hardware-specific build off the fan-out path. The split-button
popover loses its per-row anchoring because there are no rows; one pane, one
anchor.
Selection lives in ?backend=, preserving the ?target= scope rather than
clobbering it.
e2e: 139 passing across models-gallery, navigation, backends-management,
models-recommended-panel, nodes-per-node-backend-actions, page-render-smoke,
operations-strip and model-artifact-operation. The backends spec gains six
split-view tests; its three description-cell tests move onto the pane lede.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(ui): put the Host inventory on the split view
The last of the three surfaces, and the one that is not a catalog. Both tabs
had the same click-to-expand row, so the shell transfers; what does not
transfer is the zero state, because there is nothing to discover in your own
inventory.
With nothing selected the pane reports what is happening: how many models are
loaded, what failed, what has an update, and which models are holding VRAM
right now. Every number was already on the page. None of them had been
assembled into one statement, so "what is going on" was a question the tabs
could not answer however long you looked at them.
The rail buckets by state rather than capability - Running, Idle, Disabled for
models; Update available, Installed for backends - which is the opposite of the
galleries and deliberately so: nobody opens Host wondering which of their
models does vision. Entries carry a state stripe for the same reason.
Load and Stop are promoted out of the kebab, because that is what an operator
came for; the rest stays behind the menu rather than diluting it. Adopted,
pinned and alias badges follow the model into the pane: they are facts about
the thing, not about its state, and the rail line is spent on state.
Deliberately NOT done: folding the two tabs into one rail, as the mock had it.
It costs five URL parameters, the manage-tab localStorage key and the
stat-card shortcuts, all of which are live deep-links today. The tabs stay as
the group selector; merging them is a follow-up with its own migration.
e2e: full suite 355 passing. New host-split-view spec; alias-template,
manage-logs-link, manage-action-menu-position and model-editor-back-nav move
off `.table` and the row kebab onto the rail and the pane.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* polish(ui): accessibility and consistency pass over the three split views
Findings from a pass over what the previous four commits actually shipped,
rather than what they were supposed to.
The rail was not a listbox. ARIA lets a listbox contain options and groups,
and nothing else, but each group's collapse control is a button that has to
sit inside the scroller with the entries it folds. It is now a labelled group
of buttons, which is the honest description; selection is announced with
aria-current and the arrow keys are unaffected.
Every entry was its own tab stop, so tabbing past a forty-entry rail to reach
the pane took forty keystrokes. Roving tabindex makes the rail one stop, and
arrowing now moves focus with the selection instead of leaving it behind on an
entry Tab can no longer reach.
The rail rounds its corners with overflow:hidden, which was clipping the focus
ring off the first and last entries entirely. Inset outlines fix it.
A 30px row is fine under a mouse and too small under a thumb, so coarse
pointers get a 44px target without costing density on a desktop.
One slot said three different things: "9 models loaded" on Discover, "12
loaded" on Backends, "3 of 9" on Host. All three lists are a page of a larger
set, so all three now say so the same way.
Also removed: an emptyLabel prop on EntityRail that nothing passed, its dead
CSS rule, and MODELS_COLSPAN and ResourceRowDesc, which died with the tables.
e2e: full suite 355 passing.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(ui): correct three defects only a real gallery exposed
Running the branch against a live instance with 1,595 models and 1,017
backends, rather than against mocked fixtures, surfaced three things the e2e
suite could not.
Grouping did nothing. The rails matched on the use-case keys the filter chips
send (`chat`, `tts`, `transcript`), but those are a server-side vocabulary the
handler maps onto entries. What entries actually carry is free-form and
inconsistent: models come back tagged `llm`, `gguf`, `vision`, `coding`, and
backends `LLM`, `text-to-text`, `audio-transcription`. Nothing matched, so
every model landed in "Everything else" and the feature was decorative.
Grouping now lives in utils/entityGroups.js, shared by both galleries, matching
case-insensitively against the vocabulary the API really uses, with the entry's
backend as a fallback signal - a backend named `whisper` is a speech backend
whatever its tags say. Order is specific before general and that is
load-bearing: a vision model is tagged `llm` too, so testing text first would
swallow it.
The zero state claimed GPU memory on a machine with no GPU. The resources
endpoint reports system RAM in the same field when gpu_count is 0, so the hero
read "84.4 GB of GPU memory" next to the recommendations panel correctly
saying "No GPU detected". The number was never wrong, only its label; it now
says system memory unless a GPU is actually present.
The page title still said "Install Models" under a nav entry saying Discover.
Also: the keyboard test named the model it expected to arrive at, which made it
a hostage of the grouping table and broke the moment the buckets were fixed. It
now asserts that the selection moves and returns.
e2e: full suite 355 passing.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(ui): the filters and the rail were fighting over the same job
Four things you find odd on Discover, and they turn out to be one mistake seen
from four sides.
The rail grouped the current page. The listing is paginated at nine rows, so
those bucket headers described nine entries out of 1,595, and turning a page
reshuffled the sections under the reader. The structure was never stable
because it was computed over the wrong set.
The chips were redundant for the same reason, seen from the other side. They
send tag= and filter all 1,595 server-side. The rail grouped nine of them
client-side by the same axis. Two controls for one job, and the weaker one was
the one this branch added, so it goes. Grouping stays only on Host, where the
list is complete, local, and bucketed by state rather than capability.
The search bar felt odd because it sat in a full-width band while the thing it
narrowed was a 290px rail below and to the left. The whole band now lives in
the rail column: search, backend, use cases, refinements, then the list it
narrows. One column to say what you want, one to show what you got. Nineteen
chips do not fit at that width, so they fold into a disclosure that states the
selection. A disclosure and not a popover, deliberately: picking use cases is
multi-select and interleaves with the backend select and the toggles below,
and a popover dismisses itself the moment you touch either.
The header held two counts and two buttons at arm's length from all of it. The
counts were the third statement of the same number on one screen, after the
rail's "9 of 1,247" and the pane's own headline, so they go. The buttons move
into the pane's zero state, which is the surface that answers "what do I do
here".
Also: the two first-run empty states wore .loading-center, which is
display:flex in the default row direction because it exists to centre one
spinner. With four children that put the icon, the heading, the sentence and
the buttons on a single line with no gap. They are now a proper full-height
empty state.
e2e: full suite 353 passing. Grouping tests are replaced by ones asserting the
rail stays flat; chip tests open the disclosure first; two filter-layout tests
that asserted the old three-band arrangement now assert the column.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* polish(ui): make Discover a full-height view, group the chips, name the refinements
Four things, all of them the same complaint: the page read as a document with
controls scattered on it rather than as one view.
The header is fused. A title block with its own padding, a subtitle and two
counts made the split view look like an attachment to a document that happened
to sit below it. It is now a slim bar carrying the title, the count and the two
page-level actions, and the split fills the rest of the window. Rail and pane
scroll independently, so the filters and the pane's headline stay put while a
long list moves under them.
The chips group. Nineteen in a flat row is a lot to scan even behind a
disclosure, and they already belong to the four families the rest of the UI
speaks, so they are bucketed by those. "All" sits on its own above them without
a heading, because it is a reset rather than a use case.
The refinements stop looking dumped. When the band became a column they were
three controls left where they landed; they now read as a named section with
one control per row.
The zero state suggests again. It had decayed into a "Browsing / 9 of 1,247 /
select a model" line that restated the count for the third time on one screen.
It now offers the four use cases as tiles that set the filter, which is the
shelf idea from the mock without inventing curation or paying for a second
fetch.
Two bugs found by looking at it rather than at the tests: the disclosure was
clamped to 190px, which cut it off partway through its third section so two of
the five never appeared at all; and the creation actions rendered twice, once
in the new bar and once in the pane hero a few pixels away.
e2e: full suite 353 passing. The chip-row test now holds its contract across
the per-family rows rather than a single one, and additionally asserts every
family is present and non-empty.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(ui): pin the split view's height so a long detail scrolls the pane
Selecting a model with a long description grew the whole page and dragged the
rail down with it, which is the opposite of what "full height" was supposed to
buy.
The flex chain was right and the ceiling was missing. .app-layout and
.main-content are min-height:100dvh, which is a floor: flex distributes free
space but nothing caps growth, so a pane taller than the viewport expanded the
column, the document scrolled, and the rail stretched to match. height:100% on
the pane then resolved against an auto-height parent and did nothing.
The chat route already solves this by pinning .main-content to 100dvh. The
same treatment now applies to any route containing a .page--app, selected with
:has() so the shell does not have to learn which pages happen to be split
views. Below the stacking breakpoint the pin is lifted, because two stacked
halves in two short scrollers is worse than a page that scrolls.
Measured on a live instance: document height stays at the viewport across
selection (950px either side) and the pane overflows internally instead.
Adds discover-height.spec.js, which asserts the page height and the rail height
are unchanged by selection and that the pane is the thing that scrolls. The
existing specs could not have caught this: they mock short descriptions, and
the bug only appears when the pane has more content than the viewport holds.
e2e: full suite 355 passing.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(ui): give Backends and Host the full-height view, and fix the Update button
Backends now matches Discover: the header fuses into a slim bar carrying the
title, the count and the page-level actions, the filters move into the rail
column where they narrow the rail and nothing else, and the split fills the
window. Its seven chips fit at rail width, so unlike Discover's nineteen they
need no disclosure. Host gets the bar and the height; its resource monitor,
summary cards and tabs stay above the split, because those are read once while
the rail and the pane are worked in.
Two things the height change surfaced.
The console layout is a flex row with align-items:flex-start, so its body sizes
to content. Right for the pages it was built for, wrong for a split view, which
needs a ceiling to scroll inside: without it the Backends rail ran past the
viewport and over the footer. Pinned with :has() so only split-view routes are
affected.
The filters vanished when nothing matched. Both galleries swapped the whole
shell for an empty state, which took the search box and the chips with it, so
the page said "try adjusting your search or filters" while offering neither.
The shell now stays and the empty state moves into the pane.
Also fixes the Update control on Host, which had no className at all and
rendered as bare text, next to a status span that had picked up btn classes and
two copies of `fas` and so rendered as a button you cannot press. They have
swapped appearances back.
e2e: full suite 355 passing. The render-smoke selector learns .view-bar__title,
since the pages it checks no longer all use PageHeader.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(ui): keep the view mounted while searching, and bring rail grouping back
Searching replaced the whole view with a loader. The search box lives in the
rail column, so every debounced refetch unmounted the field being typed into
and dropped its focus with it. The list, the filters and the pane went too.
The shell now stays and the rail says it is busy: a sweep bar under its header
and the stale list dimmed, so the eye knows the answer is being replaced
without losing its place. A cold start still gets the skeleton, because there
is nothing to keep.
The condition for that is "nothing has loaded yet", not "the list is empty".
Those differ exactly when someone is editing a query that matched nothing, and
getting it wrong there would unmount the view on the keystroke after a
no-results search - the worst possible moment.
Grouping comes back on both galleries. It was removed because nine rows could
not fill five buckets, so a page turn rebuilt the rail's whole structure. That
was a symptom of the page size rather than of grouping: the rail now asks for
30 rows instead of 9 (Backends 60 instead of 21), which is enough for the
sections to read as structure and turns five times fewer pages. The order of
the sections is fixed, so what changes between pages is membership, not
arrangement.
Grouped while browsing, flat while searching, as before: once a term is typed
the buckets stand between the reader and the answer.
Also gives GalleryLoader a class and a testid instead of six inline style
declarations on a bare div, which is why nothing could select it.
e2e: full suite 359 passing, including a new spec asserting the search box
keeps its focus and its value across a refetch, and that a cold start still
shows the skeleton.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* perf(gallery): stop invalidating the VRAM estimate caches on every request
Searching or turning a page felt slow. It was not the search and not the
listing: /api/models answers in 3-9ms. It was the VRAM estimate, which the
gallery asks for once per row, and which took ~2.3s every single time however
often the same model was asked about.
pkg/vram already caches what makes that expensive - the remote content-length
probes, the GGUF metadata reads and the HF repo sizes. Those caches key on a
gallery generation counter, and AvailableGalleryModelsCached triggered a
background refresh on every call, with each refresh bumping the counter. One
page view is one listing request plus thirty estimate requests, each of which
re-read the gallery and started another refresh, so the generation moved
constantly and every cache entry was stale before it could ever be read. The
caches were dead in production.
Three changes, each doing one thing:
A refresh interval. The cached list is still served immediately; this only
decides how often re-fetching from upstream is worth starting. Five minutes,
as a package variable so tests can drive it without waiting.
A generation bump only when the gallery actually changed. An unchanged gallery
re-fetched on schedule must not throw away work that is still valid, which is
the difference between an estimate costing nothing and costing a network round
trip.
A separate "loaded" flag. The cache engaged on `cached != nil`, so a gallery
that legitimately holds nothing read as never-loaded and took the blocking path
on every call, bumping the generation each time. Found by the test for the
interval, which could not pass while this was true.
Measured against a live instance with 1,595 models:
one estimate, repeated 2.3s -> 2ms
a page of 30, in parallel 10s -> 0.04s
A first, genuinely unseen model still costs its remote probe. That is inherent;
what changed is that it is now paid once per model per gallery version rather
than once per request.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* perf(gallery): warm VRAM estimates at startup, and stop the UI waiting on them
Two halves of the same complaint: the gallery stalls on VRAM estimation.
Server side, the estimates are now warmed in the background at startup.
Estimating an entry nobody has asked about costs a remote probe of its weight
files, and the gallery needs one per row, so the first visitor was paying for
the whole page. The warm-up walks the gallery in the order the UI lists it, so
the first page is ready before anyone reaches it.
It is bounded and it never blocks: 300 entries at 4 at a time by default, on
its own goroutine, stopping with the server's context. Warming the whole
gallery would be thousands of probes on every boot, which is rude to the
upstream and slow to finish; warming nothing leaves the first page paying two
seconds a row. Anything past the limit still warms itself on first view.
LOCALAI_VRAM_WARM_LIMIT=0 turns it off for an air-gapped host,
LOCALAI_VRAM_WARM_CONCURRENCY=1 slows it for a metered link.
Client side, the page no longer waits on estimates it does not need yet. It
fired one request per row at once; a browser allows about six connections per
host, so thirty estimates took every slot and the request behind a click - the
variant list, an install - queued behind work nobody asked for. That is the
freeze: the list was already usable, and the UI was busy fetching sizes. Four
at a time leaves room for the interactive request to overtake, and a row whose
estimate is still in flight says "sizing…" rather than leaving a blank where a
number will appear.
buildEstimateInput moves to core/gallery as EstimateInput, since the handler
and the warmer both need it.
Measured against 1,595 models, from a cold boot:
page 1, 30 estimates in parallel 10s -> 0.04s
full warm-up (299 of 300 entries) 3m, in the background
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* chore: untrack data/.local_user_id and ignore the runtime data dir
`local-ai run` writes its instance state under ./data when started from the
repo root, which is exactly what a contributor testing a build does. The
identity file ended up committed on this branch by a `git add -A` while
verifying the gallery changes against a live instance.
Anchored, so it matches the runtime directory at the repo root and not a
`data` directory nested inside some package.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Add the recommended Q4_K_M build and an MTP-enabled variant with the shared vision projector. Tag the existing Qwythos 9B MTP entry so serving-feature ranking recognizes it.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* chore(deps): bump cogito to v0.11 ahead of the nib harness
nib is the agent harness that becomes 'local-ai chat'. It requires cogito
v0.11, so pull that bump forward on its own: minimal version selection would
apply it to LocalAI anyway, and both repos use cogito and cogito/clients.
Landing it separately keeps the harness change reviewable.
nib itself is not pinned yet. Nothing in LocalAI imports it, and 'go mod
tidy' runs as a goreleaser before-hook in CI, so an unimported require line
does not survive. It lands with its first importer.
No LocalAI call site needed a change. Both cogito.WithMaxAttempts callers
guard the argument above zero, so v0.11's new clamp is unreachable, and
LocalAI's Multimedia values implement only URL(), so v0.11's new
TypedMultimedia routing treats them as images exactly as v0.10 did.
Binary size (cmd/local-ai): 200,301,381 -> 200,336,045 bytes (+34,664).
A throwaway probe that links nib measured 201,042,243 bytes (+740,862 over
the pre-change baseline).
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): resolve and seed the agent state directory
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): write the agent config atomically and tighten its modes
Replacing config.yaml in place truncated it first, so an interrupted write
would have destroyed the api_key nib keeps in the same file. Stage through a
sibling temp file and rename over the target instead, and match nib's 0700
directory mode.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): probe the endpoint and classify failures
Probe lists what a LocalAI endpoint advertises and separates the two
failures that need different advice: nothing listening, and rejected
credentials.
go-openai reports a rejected key as one of two concrete types depending
on the error body, and both occur against a real LocalAI. The normal
error handler sends an OpenAI error envelope, which arrives as
*openai.APIError; the opaque-errors handler replies with a bare status
and no body, which arrives as *openai.RequestError. Classifying on only
one of them misses half the cases, so the status is read from either.
A cancelled probe is not reported as an unreachable server, because it
learned nothing about the endpoint, and neither is a reply that could
not be parsed, because something did answer. Both would otherwise send
the user off to start a server that may already be running.
The model list is returned verbatim and in server order. LocalAI lists
whatever it finds in the models directory, including stray archives and
dotfiles, and deciding which advertised ids are real belongs to whoever
presents them.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): resolve the model from flag, config, or the server
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(chat): pin that model resolution sorts a copy of the caller's slice
The sort spec asserted only on what the chooser was offered, so replacing the
defensive copy with an in-place sort of req.Available still passed all 37
specs. Assert the input slice's order after the call, so the guarantee cannot
be dropped silently by a later refactor.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): offer to start a server when none is reachable
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): bound the server wait and pin readiness and stop semantics
Set cmd.WaitDelay so a backend subprocess holding the child's stderr pipe
cannot block cmd.Wait forever, which would leave exited unclosed, burn the
whole shutdown grace on a clean exit, and leak the waiter goroutine.
Two test gaps closed alongside it: the readiness spec now counts polls, so
treating 503 as ready is observable, and Stop's single-interrupt contract is
pinned by giving StartedServer interrupt/kill hooks that a spec can count.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactor(chat): drive Stop through one process interface, hide exec plumbing
Two independent interrupt/kill func fields plus a nil check admitted wirings
no test could distinguish: the pair swapped, so a SIGKILL would strand the
backends SIGINT exists to let local-ai run clean up, or kill left nil, so a
wedged server never escalates. One two-method interface that *os.Process
already satisfies leaves nothing to swap and nothing to nil.
Also translate exec.ErrWaitDelay, whose text names an os/exec struct field,
into what the user can act on. os/exec only substitutes that sentinel when the
process exited without an error of its own, so no exit status is swallowed.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): replace the REPL with the built-in agent
local-ai chat is now the nib agent harness compiled into the binary: tool
use behind an approval gate, sub-agents, MCP, plugins, and skills, all
auto-configured against the local server.
The REPL goes with it. Its model listing and its 401 classifier were
duplicates of the ones Probe now owns, and the classifier was the version
that misreads a bare 401 with no OpenAI error envelope, so keeping either
would leave the package with two divergent answers to the same question.
github.com/mudler/nib lands in go.mod in this commit rather than earlier:
go mod tidy runs as a goreleaser before-hook on every PR, so a require
line with no importer is stripped before it reaches CI.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactor(chat): split the pre-agent phase out of Run and pin it
Everything before the handoff is testable and nothing after it is: once
app.Run owns the terminal there is no seam left. prepare draws that line,
takes interactivity as a parameter so the prompts can be driven over a
pipe, and hands Run the state dir, the model, and any server it started.
The questions move onto one prompter that owns its buffered reader. A
fresh bufio.Reader per question reads ahead and discards what it buffered,
so the model choice typed behind an answer to "start a server?" was lost
and the next question saw EOF.
choose answers with a list index and refuses an empty offer, so a value
that was never on the list cannot reach ResolveModel, which persists it
and starts every later run against it.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): bound each server check with a deadline
Nothing bounded the model listing, so pointing chat at an address that
accepts the connection and then never replies left the user with no
output and no offer to start a server.
The budget is context.WithTimeout rather than a cancel plus a timer.
Probe deliberately refuses to call an endpoint unreachable on a
context.Canceled, since a caller who gave up learned nothing about the
server, and only honours a deadline. A cancel-based budget therefore
expires as the one error that suppresses ErrUnreachable, exactly for the
hung servers the offer exists to rescue.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): tell the user when their model choice cannot be saved
The choice is meant to be asked for once. When saving it fails the user
is silently asked again on the next run, and the only trace was an
xlog.Warn: the agent runs at log level error, and a --log-level=error run
swallows it entirely.
ModelRequest gains Notify for exactly this class of problem, one that is
worth telling the user about but not worth failing over, and the chat
wiring points it at the same writer the question was asked on.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): stop a session's server when the process is signalled
A server started for the session is stopped by a deferred call, and a
signal skips deferred calls: a SIGTERM between the spawn and the exit
left 'local-ai run' reparented to init with nothing left that knew to
shut it down. Ctrl+C was already safe, but only incidentally, because the
child shares this process' foreground process group.
A signal handler rather than Pdeathsig on the child. Pdeathsig is
Linux-only and, in Go, is delivered when the OS thread that forked exits
rather than when the process does, so it can fire on a healthy parent.
Setpgid would break the Ctrl+C that works today by taking the child out
of the foreground group.
SIGHUP joins SIGINT and SIGTERM: a terminal program whose terminal is
gone has nobody left to talk to. The same context is what cancels the
agent, which nib leaves to its embedder.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): only skip the server checks for work that stays local
Two argument shapes were classified wrongly. Every 'mcp ...' invocation
counted as management, so 'local-ai chat mcp --stdio', which serves the
agent over MCP and needs a model like any other session, was handed an
empty one. And --init, whose shell snippet a user pastes into an rc file
long before any server exists, went the other way: it demanded a running
server to print a static string.
The mcp split is asked of nib's own IsMCPManageSubcommand rather than
restated here, so a verb added upstream cannot drift out of this list.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): exit with the agent's status instead of reporting it twice
nib writes what went wrong to stderr and returns nothing but an exit
code, so returning that error unchanged had main log "Error running the
application error=exit status 1" underneath the message the user had just
read. The refusal to render the full-screen interface into a pipe is the
one they meet in practice: it names --cli, and burying that hides the fix.
ExitCodeError says "already reported, exit with this status". main
honours it and prints nothing more, so a piped or redirected chat still
fails a script the way it should.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* style(chat): route interactive chatter through one writer helper
The prompts and notices all write to a terminal, where a failed write is
not worth failing the session over and the read that follows the question
reports the real problem. say says that once instead of five discarded
error returns.
The command's one-line help comes along: chat is no longer "an
interactive chat session".
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(chat): record why the agent gets this process' streams
Injecting them is what makes nib refuse to draw its full-screen interface
into a pipe and name --cli, instead of rendering onto a terminal the
caller may not own. The tradeoff is worth stating where the wiring is.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): stop the session's server on cancellation, not on the way out
The deferred Stop is reached only if the agent returns, and cancelling
the context does not make it: nib hands the TUI to bubbletea without the
context, so what actually unwinds a running session today is bubbletea's
own SIGINT and SIGTERM handler. SIGHUP has no such backstop, and
registering for it removed the default disposition that used to end the
process outright, so kill -HUP left a live TUI with a cancelled context
and the started server still running.
runSession watches the context alongside the agent and stops the server
the moment it is cancelled, so the guarantee no longer depends on what
the agent does with cancellation. Stop is idempotent, so the deferred
call stays correct and free.
The doc comment on shutdownContext described the mechanism it was
supposed to work by rather than the one that does. Corrected, bubbletea's
handler included.
ResolveModel now checks the chooser's answer against what it offered.
The shipped chooser answers by list index and cannot be wrong, but
ModelChooser is exported, the answer is persisted, and every later run
starts against it, so the invariant belongs at the consumer.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(chat): bump nib to v0.5.1
v0.5.1 carries four fixes that matter to 'local-ai chat':
- --init now names the embedder's command, so the emitted widget invokes
'local-ai chat' rather than a bare 'nib' the user does not have.
- A piped CLI session that succeeds exits 0 instead of failing with EOF.
- EOF at a tool-approval prompt denies the call rather than approving it,
and the session exits 3 (app.ExitCodeApprovalNoInput) so a script can tell
"answered" from "refused to act" without reading stdout. Read-only tools
are unaffected and still run. ExitStatus already unwraps app.ExitError,
so the code propagates with no change here.
- RunTUI passes the context to bubbletea and gives up bubbletea's own signal
handler, which makes shutdownContext the single owner of the signal and
stops a SIGHUP leaving a wedged TUI behind.
Verified against a live server on 127.0.0.1:8080: the three --init shells,
a piped prompt exiting 0, a denied 'touch' that left no file and exited 3,
a read-only 'ls' that still ran and exited 0, and a SIGHUP that unwound a
TUI running under a pty.
Two comment blocks in run.go described the old TUI behavior and are now
wrong, so they are corrected in the same change. No behavior change: both
shutdownContext and runSession are untouched, and stopping the server on
cancellation is still worth keeping independent of how promptly nib unwinds.
One known gap, not addressed here. The widget --init now emits runs
'output=$(local-ai chat --height 50%)', and runAgent injects Stdout
unconditionally, so under $(...) nib refuses the TUI for a non-terminal
stream. This is the cost the runAgent comment already anticipated, now that
the snippets no longer hardcode standalone nib. Ctrl+Space should not be
documented until that is decided.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): let nib own stdout, so the Ctrl+Space widget works
The widget 'local-ai chat --init' emits runs
'output=$(local-ai chat --height 50%)', which puts a pipe on stdout by
construction. runAgent injected os.Stdout unconditionally, and nib refuses
every mode but --cli when a stream it was handed is not a terminal, so
Ctrl+Space printed "Re-run with --cli to use the injected streams" and
inserted nothing. Verified against a pty before and after.
nib reads a nil stream as "not injected" and falls back to the process
stream, which is how an embedder asks for nib's own behavior. That is what
stdout needs: the interface renders on /dev/tty but writes the chosen
command to stdout even when stdout is a pipe, and that write is the whole
of the shell-capture idiom.
Stdin is deliberately left injected. A piped or redirected stdin really is
ignored by the interface, so the refusal is the honest answer there, and it
is the one users meet: 'echo q | local-ai chat' still says to re-run with
--cli, once, exit 1. Nilling stdin the way stdout is nilled would delete
that silently. Stderr is not gated by nib at all and is unchanged.
One case does change and cannot be kept: 'local-ai chat > out.txt' from a
terminal no longer refuses, because it is indistinguishable from the
widget. It renders on /dev/tty and writes the capture line to the file,
which is what standalone nib does.
The app.Options literal moves into agentOptions so the decision is
reachable from a spec rather than being a detail of a function that takes
the terminal. Both sides of the asymmetry are pinned: reinstating
'Stdout: opts.Out' fails "hands nib nothing for the process stdout", and
nilling stdin fails "hands the process stdin over".
Also rewrites the last comments describing the pre-v0.5.1 behavior.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(chat): say what the stream refusal actually keys on
Two comments still called it the refusal to render the interface "into a
pipe". That was true when both stdin and stdout were injected, but a pipe on
stdout no longer refuses, so the wording now points at precisely the case
that was un-refused to make Ctrl+Space work. Only a stdin that cannot be
read triggers it, and both comments now say so and name the command a user
meets it with, 'echo q | local-ai chat'.
The agentOptions doc also said a "file a caller chose" stays injected and
refused, which reads as though 'local-ai chat > out.txt' still refuses. It
does not: a shell redirect arrives as os.Stdout and is nil-ed like the
widget's pipe, because the two differ only in being a regular file rather
than a FIFO and nib's gate does not look at that. What stays injected is a
writer an in-process caller chose for itself. Says that now, in the doc and
in the spec comment that had the same ambiguity.
Comments only. No behavior change.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: local-ai chat is now the built-in terminal agent
`local-ai chat` was a plain chat prompt and is now an agent that runs
shell commands behind an approval gate, so the pages that described a
REPL were wrong rather than merely thin.
Adds a Terminal agent feature page at /features/terminal-agent covering
the approval gate, piped runs and their exit codes, Ctrl+Space, model
resolution, state directory, and the pass-through management commands
(including the `--yes` caveat that leaves a plugin installed but
disabled in a script).
The three-way "looking for something else" notice becomes four-way and
moves into an agentic-routing shortcode. Four hand-kept copies of the
same paragraph is what produced the drift the new page would otherwise
have added to; the shortcode takes `current=` so each page still marks
itself, and errors the build on a name that is not one of the four.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* website: the agent is in the binary, not a second install
The nib section sold a separate tool you also install, with a GitHub
link as the only way in, which is now the wrong order: the agent ships
compiled into local-ai, and the standalone binary is the second reason
to care rather than the first.
Leads with `local-ai chat`, keeps nib as the SSH-anywhere story, and
adds a docs CTA pointing at the new Terminal agent page. id="nib" is
left alone because localai.io/#nib is linked from outside.
The two credits on the demo clip named nib as the thing that drove the
machine; they now credit the agent in LocalAI, which is the same agent.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* website: fix the exit keys, the plugin warning, and the redirect gap
Three claims on the chat-agent pages that the code does not back.
try-it-out told readers to press Ctrl+D. nib has no Ctrl+D handler: the
full-screen interface quits on Esc or Ctrl+C, and Ctrl+D is only an exit
in --cli, where it arrives as ordinary tty EOF. That sentence had
replaced the removed /exit and /quit text, so the page was left with no
working way to leave a session. Document both modes, since they differ.
The plugin warning said nothing tells you the install stopped short. It
does: the command prints that the plugin was left disabled. What it does
not do is say so in its exit code, which is 0 either way. That is the
part a script cannot work around, and it is the reason to pass --yes.
Overstating it in the paragraph that gives the advice only makes the
advice easier to dismiss.
Redirecting stdout no longer refuses; the interface goes to /dev/tty and
only the yanked command reaches the file. It is what lets the Ctrl+Space
widget capture a command at all, since a redirect and out=$(...) are the
same thing to the stream gate. It was documented nowhere. A non-terminal
stdin is still refused, and the new text says which of the two it is.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): make the CLI flags outrank the agent config file
local-ai chat routed --endpoint, --model, --api-key, --trace-dir and --yolo
through nib's app.Options.Defaults. Defaults are seeds: they sit beneath the
config file, so the file silently undoes them. That made the flags accepted and
inert, and not in an edge case, since EnsureStateDir writes base_url on the
first run and the interactive picker writes model, so from the second run on
the file carried a value for both.
Observed against a live server: with base_url: http://127.0.0.1:9999/v1 in the
config and --endpoint http://127.0.0.1:8080 on the command line, the probe hit
8080 and every agent turn posted to 9999. With model: gemma-4-e2b-it-qat-q4_0
in the config, --model lfm2.5-8b-a1b was ignored on the wire.
nib v0.6.0 adds app.Options.Overrides, applied above the config file and above
the bare environment block. Move the whole block there: all five values are
decisions this invocation already made on the user's behalf, and a flag the
config file can undo is not a flag. Nothing is left in Defaults, because
LocalAI's one genuine seed, the initial base_url, is written into the config
file by EnsureStateDir rather than handed to nib.
Two limits come with the channel and are documented on agentOptions rather than
worked around. An override can only raise a field, since nib cannot tell "set
to the zero value" from "not set", so --yolo can turn approval off but nothing
on the command line turns it back on over an approval_mode: auto in the file.
And nib's own NIB_TRACE_DIR and NIB_YOLO are resolved after the config load and
still outrank these, deliberately, upstream.
The existing spec pinned that the right values reach app.Options, which they
always did, which is exactly why it could not see nib discarding them. The new
specs resolve the config the way app.Run resolves it, against a real config
file that disagrees with every flag, and one asserts Defaults stays empty.
docs/content/features/terminal-agent.md already documented --model as winning
over the saved model; that was false before this change and is true now, so no
docs edit was needed.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): document intentional config file read
Assisted-by: Codex:gpt-5 [gosec]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The post was written against the first draft of the release notes, when the
cycle stood at 214 PRs over thirteen days. It closed at 321 PRs over eighteen
days, and three of the larger user-facing changes landed after it was written.
- Correct the counts throughout: 321 PRs, eighteen days, 24 contributors
(11 first-time), gallery 1,221 to 1,505.
- Add sections for the three new capabilities: 3D generation as a modality
(Generate3D, FLAG_3D, /v1/3d/generations, trellis2cpp), audio.cpp serving
six audio endpoints from one process, and the operations bar becoming the
Activity page.
- Cover the two further hardening fixes (tar hardlink escape, cyclic $ref
stack overflow) alongside the TRL one.
- Note the Valkey store, systemd socket activation, persistent trace history,
in-place chat edits, the self-contained SYCL backend and the site split.
- Group the new-engine sections together rather than splitting them across
the operational ones.
Embeds the existing vllm-race and magpie clips, and adds a 3D generation clip
cut from the demo recording to the conventions in .agents/preparing-a-release.md
(no audio track, 14s, named for the feature). blog.css styled figure img but
not figure video, so a clip in a post rendered outside the card; both selectors
now share the rule.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
tests-qwen3-tts-cpp has been failing on master since 2026-07-31. The suite
loads every component fine and then stops: TTS() never returns from the
native call, so a job that takes ~5 minutes runs into the 20 minute Go test
timeout instead.
goroutine 74 [syscall, 19 minutes]:
github.com/ebitengine/purego.RegisterFunc.func4
qwen3-tts-cpp.(*Qwen3TtsCpp).TTS goqwen3ttscpp.go:154
qwen3-tts-cpp.init.func2.4 e2e_test.go:90
Not a flake: reproduced on master and again on an explicit re-run.
Bisected across this cycle's eight qwentts.cpp bumps by their own check:
10832, 10850, 10902, 10964, 11006, 11039 and 11127 all pass in ~5 minutes;
11241 (abab6b3) fails at 1h58m. That PR was merged with this check already
red, which is how the hang reached master.
35ebe537..abab6b3 is three upstream commits, and the only functional one is
26dd8adb, "predictor: unroll the frame into one cgraph and sample in standard
ops", which is consistent with a generation loop that never reaches its stop
condition.
Hold the pin at the last known-good commit. The bump entry is commented out
rather than left in place, because it tracks upstream master and would put
the hang straight back on the next nightly run. Both spots carry a pointer to
the other so the hold is discoverable, and restoring it is uncommenting four
lines once upstream is fixed.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Select the CPU_ALL_VARIANTS target for x86 GPU images so partial offload uses runtime-selected host kernels. Keep GPU arm64 builds on the portable fallback until their toolchains consistently provide gcc-14.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): drop merged MiniMax-M3 patch
The bumped llama.cpp revision includes the MiniMax-M3 parser and template detection, so the carried patch now rejects during backend preparation. Remove the obsolete patch while retaining the independent score-task patch.
Assisted-by: Codex:gpt-5 [systematic-debugging]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add the new 9B Fara computer-use model alongside its existing 27B sibling, with Q4_K_M and Q8_0 llama.cpp variants plus the required vision projector.
Assisted-by: Codex:gpt-5 [web]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Build the runtime CPU variant set alongside x86 GPU backends so partial offload uses the host's SIMD kernels instead of the scalar fallback. Keep arm64 GPU images on the portable binary until their builders consistently provide gcc-14.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(utils): reject tar hardlinks that escape the extraction root
ExtractArchive pre-scans archive members and rejects symlinks, but tar
hardlink entries carry a regular file mode and so pass that check.
Header.Linkname was never validated, so an archive could create a link
to a path outside the destination directory.
Validate Linkname with the same path check already applied to member
names. Hardlinks that resolve inside the extraction root still extract,
so ordinary archives are unaffected.
pkg/oci/image.go already resolves tar.TypeLink targets before using
them; this brings the archive extraction path in line with it.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Zelys-DFKH <zelys@dfkhelper.com>
* test(utils): cover hardlink overwrite and in-root hardlinks
The existing hardlink test names a link target two levels above the
extraction root, so its final assertion checked a path the link never
resolved to and could not fail. Point the target one level up instead,
at the path that assertion already names.
Add two cases. The first uses a .tar.gz, where ExtractArchive binds a
Tar config with OverwriteExisting set, and follows the link entry with a
regular entry of the same name. Before the fix that pair linked to a
file outside the root and then truncated it through the link, which the
plain .tar case does not reach. The second extracts a hardlink whose
target is an earlier member of the same archive, covering the claim that
ordinary archives are unaffected.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Zelys-DFKH <zelys@dfkhelper.com>
---------
Signed-off-by: Zelys-DFKH <zelys@dfkhelper.com>
Use the case-sensitive Hugging Face filenames and refresh the linked SHA256 values for both gallery variants.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Detect text-to-speech MLX repositories during model import and emit a TTS-ready mlx-audio configuration. Expose mlx-audio in the backend preference dropdown for repositories without complete metadata.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(sycl): make the intel llama.cpp backend self-contained on any host
The SYCL backend shipped an incomplete oneAPI runtime AND relied on a
host-provided GPU driver, so it only ran inside the build container. On a
bare host it died with "libze_loader.so.1 / libdnnl.so.3: cannot open
shared object file", and even with the host's Intel driver installed it
SIGSEGV'd during SYCL init when the host driver was built against a newer
glibc than the backend's bundled loader (rolling-release distros).
package_intel_libs now bundles the complete, coherent oneAPI runtime
(the missing MKL ILP64 / sycl_blas / tbb_thread + oneDNN + the dlopen'd
UR adapters, plus a sweep of the backend binaries' own direct deps) and
the Intel GPU userspace driver (libze_intel_gpu + libigdrcl + IGC + gmm)
with its OpenCL ICD manifest, mirroring how package_vulkan_libs bundles
Mesa. run.sh points the Level Zero and OpenCL loaders at the bundled
driver, and install-base-deps.sh installs it in the SYCL build image.
Bundling the driver is safe across kernels because it talks to the host
i915/xe via the stable DRM UAPI (unlike NVIDIA's kernel-locked
userspace).
Validated on Arch (glibc 2.43, i915): the backend loads and runs on an
Iris Xe with no host Intel packages installed.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
* fix(sycl): install a driver that exists, and let the user choose their own
The driver install added earlier in this branch asked apt for
intel-level-zero-gpu, which is not a package in Ubuntu 24.04. apt fails
outright on an unknown name, so neither driver was installed, nothing was there
to copy, and the images carried no driver at all.
It now comes from Intel's own repository, which has 25.18 for this Ubuntu
release, against 23.43 from late 2023 in the Ubuntu archive. The archive driver
does not know any card released since, so a machine with a recent Intel GPU
would end up carrying a driver that cannot drive it. Anything that goes wrong
during that install fails the build on purpose: an unreachable repository is a
passing problem that a retry fixes, while quietly carrying a different driver,
or none, is a difference nobody would notice until a user reports an idle GPU.
run.sh used to overwrite whatever driver the user had chosen. Level Zero uses
only the driver it is given, so on a machine with a card too new for the
carried driver, the GPU would go unused with no way back. Both that setting and
the OpenCL one are now left alone when already set, and the docs say how to
point a backend at the machine's own driver.
The OpenCL setting also used to be applied whenever the backend held a driver
list, even when the driver it named had not been copied, which leaves OpenCL
with nothing instead of falling back to the machine's own driver. It now
requires the copied driver to be present, and the packaging leaves out the list
entry of any driver it did not copy. The oneAPI images list a processor-only
OpenCL library, which was being carried with nothing behind it.
Two more corrections in the packaging. The scan for libraries a program is
linked against only looked at files named llama-cpp-*, so turboquant and bonsai,
which are also built for Intel GPUs, were left with the incomplete set of
libraries this branch set out to fix; it now looks at every program in the
directory. And a build that should carry a driver but ends up without one now
says so, which is what a stale prebuilt base image looks like: such a backend
still runs on a machine that has its own driver, so nothing fails and the only
other symptom is a user reporting an idle GPU.
Backends now also ask the driver to report how much graphics memory is free,
without which llama.cpp reads zero on an integrated GPU, since such a chip
shares the system memory instead of having its own. turboquant and bonsai get
the same run.sh handling as llama.cpp.
The driver is only carried by the builds that start through run.sh, because
run.sh is what points Level Zero and OpenCL at it. The Python backends for
Intel GPUs start differently and would never load it, so they keep using the
machine's own driver rather than carrying several hundred megabytes they cannot
use.
Checked in a container on Ubuntu 24.04: the install brings driver 25.18 with
the files where the packaging expects them, an unreachable repository fails the
build, and the copied set resolves on its own once the machine's Intel packages
are moved away.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
* fix(ci): rebuild every Linux backend when the GPU packaging script changes
scripts/build/package-gpu-libs.sh decides which GPU libraries end up inside an
image. The filter that builds the backend matrix listed it as an input of the
Python images only, so changing it rebuilt no Go and no C++ backend, even
though those run it from their own package.sh. A packaging fix aimed at the
Intel llama.cpp backend could merge and reach no image, which is the same
failure this rule was written to prevent.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
* fix(sycl): carry only the driver Level Zero uses, not the OpenCL one
llama.cpp reaches an Intel GPU through Level Zero, which hands the driver
programs that are already compiled and so needs only the back end of the
graphics compiler. The OpenCL driver can be handed source code instead, so it
needs the compiler's front end as well, and that arrives with its own copy of
clang. Carrying it cost about 139 MB in every backend built for Intel GPUs, and
took the carried set from 123 MB to 261 MB.
Nothing here takes that path. No LocalAI code selects an OpenCL device, each
backend image holds one backend, and the documentation never described OpenCL
as a way to run models: the only mentions are a stale clblas row in the
BUILD_TYPE table, for a llama.cpp backend that no longer exists and that no
build matrix entry uses, and the sycl-ls troubleshooting hint. Before this
branch the packaging carried the OpenCL loader and adapter but no driver, so
the path could not work in a released image either. There is nobody to keep
working.
The driver list that OpenCL reads is no longer carried, and run.sh no longer
sets OCL_ICD_VENDORS, so OpenCL inside a container keeps using whatever the
image provides rather than being pointed at a directory with no driver in it.
Checked in a container against the real 25.18 driver: the carried set is 123 MB
with nothing unresolved, and Level Zero still reports the GPU with the
machine's own Intel packages moved out of the way. Neither the Level Zero
driver nor the compiler back end names the front end or clang among the
libraries it opens by name, so the leaner set is complete for this path.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
---------
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
* feat(chat): edit saved conversation messages
Add inline edit, save, and cancel controls for stored user and assistant messages without triggering inference. Preserve structured message attachments and cancel edits when streaming starts.
Assisted-by: Codex:gpt-5
* test(chat): preserve seeded conversation on reload
The saved-message edit test reloads the page to verify persistence, but its init script was replacing localStorage with the original fixture on every navigation. Seed only an empty store so reloads exercise the data written by the application.
Assisted-by: Codex:gpt-5 [Codex]
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Direct repository-root users to the supported make docs target and document the equivalent direct Hugo invocation from docs/.
Fixes#10062
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The page reads as a list of features with nothing joining them, so two
things did not land.
The engines section never said these are the backends LocalAI loads. The
runtime section describes a core that pulls each engine in on demand, and
the engines section describes engines written from scratch, and nothing on
the page connected the two sentences. Readers were taking parakeet.cpp and
the rest for unrelated side projects by the same people. The lede now says
whose backends they are before it says anything else.
APEX was used as a known term on first appearance, in a section that opened
onto a benchmark table. Nothing said what it is or why it follows the
engines. It now opens by placing itself in the stack: the engine decides how
fast a model runs, the weights decide whether it runs at all, and APEX is
the second of those. Then the numbers.
Also drops "Most backends wrap somebody else's engine. These do not", which
is the machine-written antithesis shape, and fixes a list that broke its own
parallel halfway through.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m]
The star, fork, contributor and release counts were typed into the templates
by hand, so they only moved when somebody remembered. They had already
drifted: stars read 48,042 against 48,067, forks 4,314 against 4,320, and
contributors 224 against 225.
They move to website/data/stats.yaml, which .github/ci/refresh-site-counters.sh
rewrites from the GitHub API, run weekly by a new workflow. The contributors
and releases endpoints never report a total, so the script asks for one item
per page and reads the count out of the Link header. It refuses to write a
zero or a non-number, which is what a rate-limited or failed call looks like,
and the workflow commits only when a number actually moved. The Discord count
has no API behind it, so the script reads the existing value back and carries
it through.
The engine count was wrong in a second way. The hero said 18, the section
heading said "Eighteen engines", the timeline said "Nineteen engines of our
own", and the /engines/ page derived 19 from the data file. All of them now
derive from that same file, so they cannot disagree again.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m]
Add Q4_K_M and Q8_0 llama.cpp entries for the newly released Qwopus3.6-27B Fusion reasoning and coding merge, with MTP enabled.
Assisted-by: Codex:gpt-5 [Hugging Face API]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The timeline set six 15rem columns in a flex row with overflow-x:auto, which
needs 90rem and so scrolled sideways on any normal laptop. It is a wrapping
grid now, and the rule that carries the dots moves from the container onto
each item so a wrapped row still gets a line above it. Column gap is zero and
the items carry their own right padding, so the rule stays continuous.
Integrations move from a card grid to a reel. Any single integration is a weak
signal and the whole moving line is the strong one, so the count is doing the
argument. It pauses on hover and on keyboard focus, since the names are links.
The list grows from 8 to 26: Open WebUI, Dify, LibreChat, RAGFlow, Continue,
big-AGI, Nextcloud, Frigate, promptfoo, Mods, TypingMind, baibot, k8sgpt-
operator and others. Each was admitted only after opening that project's own
repository or docs and reading the line that names LocalAI. The ones that
failed that test are listed in the data file so nobody re-adds them.
The blog cards were hand-written, which is how one of them came to advertise
"Porting vLLM to C++", a post that does not exist, and how all three linked to
the blog index instead of an article. They range over the posts now.
The section intro used the "a changelog tells you what moved, these posts show
you what it does" shape, which is the standard machine-written antithesis. It
states what the posts contain instead, including the perplexity regression
that APEX costs, because publishing the price is the actual claim.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m]
The band led with three sentences of hedging and printed a commit count
next to each employer, so a one-commit entry beside a large name read as
weakness rather than as the modest, true claim it was. It now opens on the
contributor count, sets the employers as a sentence instead of a pill wall,
and keeps the caveat to one line. The counts stay in ecosystem.yaml, since
they are the provenance for the list and anyone re-checking it needs them.
Two "coverage" cards were not about this project. The modelslab.com piece
reviews Frikallo/parakeet.cpp, an unrelated project of the same name, and
the snailtext.app benchmark measures Parakeet through ONNX Runtime without
mentioning LocalAI at all. Both are removed, along with the contributor
card that duplicated the band's opening line.
Press was four posts from one vendor, which read as the whole of the
coverage rather than one enthusiastic outlet. SUSE collapses to a single
series entry, and Pulumi, Semaphore and Spectro Cloud join it. Each was
opened and checked against the project before being added. K8sGPT and
LlamaIndex join the integrations; both document LocalAI as a backend.
The quotes move above the lists so the section opens on its strongest
line, which is somebody else's. The hero gains a GitHub call to action,
the APEX collection link was returning 404 and is corrected, and the
footer no longer describes the site as a design mock.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m]
Follow-up to 42541dd4f, which routed lint to arc-runner-set. Both of its
jobs failed there in one second (run 30637392862): the runner image has
git, curl, unzip, tar, ldd and python3, but not make, and build-scripts
additionally needs gcc because the packaging-script tests compile a
throwaway binary and inspect it with ldd.
golangci-lint needs make twice over: `make protogen-go` (which also wants
curl + unzip to fetch protoc) and `make lint` itself. So both jobs go back
to ubuntu-latest.
gh-pages.yml stays on arc-runner-set and is unaffected: it uses no make and
no C toolchain, and setup-go / actions-hugo fetch their own toolchains.
The preflight steps stay. They cost about a second on the hosted pool, and
they are what turned this into a one-second named failure instead of an
opaque one midway through a build. When the runner image gains make + gcc,
re-routing is one runs-on line per job. Any such re-route must stay
push-only: lint also runs on pull_request, and fork PRs execute untrusted
code that must not reach a persistent self-hosted runner.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
The GitHub-hosted runner pool is shared per ACCOUNT, not per repo, so a
burst in one repo starves every other. On 2026-07-31 it reached zero
scheduled jobs for 35 consecutive minutes with 39 jobs queued, while
arc-runner-set completed 12 jobs without interruption across the same
window. Actions was healthy globally (other public repos were scheduling
normally), so this is an account-level throttle we cannot fix from inside
the workflows, only route around.
Site publishing and lint are small, run on nearly every commit, and gain
nothing from waiting behind a saturated hosted queue, so both move to
arc-runner-set, the label already proven in generate_intel_image.yaml.
lint.yml is routed for PUSH ONLY, and this is the important part: that
workflow also triggers on pull_request, and a fork PR executes untrusted
contributor code. Running that on a persistent self-hosted runner would be
a real compromise vector, so anything that is not a push to mudler/LocalAI
stays on the ephemeral hosted pool. gh-pages.yml needs no such clause: it
triggers only on push-to-master and workflow_dispatch, so it never runs
pull-request code. Both carry a repository guard so forks, which have no
such runner label, fall back to hosted instead of queueing forever.
Neither workflow uses sudo or apt, and both fetch their own toolchains via
setup-go / actions-hugo. A self-hosted image can still be leaner than the
hosted one, so each lint job opens with a preflight that names the missing
tool (curl/unzip/make for protoc and lint; gcc/ldd/python3 for the
packaging-script tests) rather than failing opaquely mid-build. Reverting
is one runs-on expression per job.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed backend verification performs separate registry requests for manifests and referrers. Reuse LocalAI's version-aware User-Agent there so the full install flow is attributable to LocalAI.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Adds the "who turns up around this project" section, split into three lists
because the evidence behind each one is a different strength and collapsing
them into a single logo wall would overclaim.
Contributors 21 companies whose engineers have commits here. Evidence is
the commit history plus the employer on that person's public
GitHub profile, so it is a claim about the person. Commit
counts are shown next to each name, including the ones that
are a single patch, because hiding that would be the whole
problem.
Integrations six projects that reference LocalAI in their own repository
or documentation, which anyone can verify without asking us.
Press four SUSE Communities articles about running LocalAI.
Names are set in type rather than fetched as logos. A logo reads as
endorsement, and a one-line typo fix from somebody who happens to work at a
large company does not support that, quite apart from what their trademark
policy says about it.
ADOPTERS.md is the mechanism for the stronger claim. An organisation that
wants to be listed as a user opens a pull request adding itself, which is both
the evidence and the permission, and is publicly auditable afterwards. The
file says plainly what the website does and does not claim, so the next person
to ask "can we add some big names" has the answer in the repository.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] [WebSearch]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(website): split the site, move docs to /docs, add a landing page
The Hugo docs site has always been localai.io itself, which left nowhere to
explain what LocalAI is or show what the team builds. This adds a separate
marketing site at the root and moves the documentation under /docs/.
Docs:
The existing site keeps its content tree and its Relearn theme, and now
builds with baseURL <root>/docs/. Its _index.md, which held a hand written
landing page, becomes a real documentation home.
Every previously published URL keeps working. GitHub Pages has no server
side rewrites, so .github/ci/gen-redirects.sh walks the built docs output
and leaves a meta refresh plus a canonical link at each old root path. It
covers bare .html files too, which is what keeps /gallery.html alive, and
it never overwrites a path the marketing site already owns.
Website:
A second Hugo site under website/ with its own layouts and no external
theme, so the marketing side does not have to fight Relearn's home rooted
menu and asset pipeline. CI builds both and merges them into one Pages
artifact.
The design is derived from the project logo rather than invented: the navy
of the triangle, the cyan of the llama, the purple of the speed bars. Those
offset bars became the motion signature. The background renders a real
depth-anything.cpp depth map as contour lines and switches to a
locate-anything.cpp style detection overlay over the engines section.
Also included: an /engines/ index driven entirely by data/engines.yaml, a
/blog/ section with five posts written from the release notes and the
engine benchmark suites, install.sh and a Kubernetes manifest since the
site advertises both, and a rule in .agents/ that release preparation now
includes a blog post and demo clips.
Every figure on the site is derived from the repository or the GitHub API,
not from memory. Correcting them against their sources found one error in
README.md: voxtral-tts.c is text to speech, not speech to text.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] [Agent]
* feat(website): add a star history chart, rewrite the history post in first person
The history post read like a changelog written by a committee. It is now in
Ettore's voice, first person, with the admissions left in.
The numbers paragraph in particular read like a directory listing. It now says
what the figures mean rather than which file they came from.
Adds an interactive star history chart, built from the GitHub stargazers API
rather than embedded from a third party, so the page makes no external request
and cannot break when someone else's service is down. The four releases the
post is organised around are marked on the curve, and the labels stack into
rows because three of them land within two months of each other.
The API stops paginating at 40,000 items, so the curve is measured up to
December 2025 and the segment from there to today's total is drawn dashed,
labelled as an estimate in the caption and in the tooltip. It is a straight
line between two known points, and the chart says so rather than implying it
is data.
Also drops "marketing site" from the README heading and everywhere else it
appeared, and calls it the main site instead.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
PR #11041 rewrote the testllama31inputResult1 fixture and un-escaped the
backslashes inside its Go raw-string literal, turning `[^"\\]` into
`[^"\]` and `["\\/bfnrt]` into `["\/bfnrt]`. The fixture is compared
line-by-line against the grammar built from PRIMITIVE_RULES in
bnf_rules.go, which is unchanged and still emits the doubled form, so
"generates a valid grammar from JSON schema" fails on every platform.
Restore the four fixture lines to their pre-#11041 form. The cyclic $ref
and depth specs added by that PR are untouched.
The regression reached master because only the DCO check ever reported
on #11041; its test runs were cancelled during the CI purge.
Assisted-by: Claude Code:claude-opus-5[1m] [Bash] [Edit]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Add the Qwen3.5-MoE POCKET-35B and Gemma 4 POCKET-26B releases with Q4, Q2, and compact IQ1 variants where available. Verify every artifact hash against its Hugging Face linked etag.
Assisted-by: Codex:gpt-5 [web]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* refactor(ui): add layout/text primitives and per-page CSS blocks
The React UI already shipped a design system (tokens, form grids, data
tables, stat cards, callouts) that the pages largely bypassed: ~2,000
`style={{ ... }}` literals across src/pages and src/components. Each one is
a spacing or colour decision made locally, so no two pages share a rhythm,
which is the main reason the app reads as unfinished rather than as one
product.
Two additions, both to App.css:
- A small semantic primitive layer: .stack / .hstack for vertical and
horizontal rhythm, .text-note / .text-sub / .text-meta / .text-mono for
the text roles the pages kept re-deriving, .tone-* + .icon-chip for
semantically tinted icons, plus scale-locked spacing and size steps.
Deliberately short and semantic, not a utility framework: the size and
spacing classes exist mainly so that an OFF-scale value stays an inline
style and therefore stays visible.
- Named blocks for the shapes fifteen pages actually have (.p2p-diagram,
.usage-tile, .tr-code, .mw-badge, .set-rail, ...), so those shapes are
defined once instead of per call site.
Two findings worth recording. The type scale is xs 0.6875 / sm 0.8125 /
base 0.875, and 0.75rem was in use roughly 100 times without being on it
(along with 0.7, 0.85, 1.1 and 0.625rem); all now snap to the nearest step.
And there were thirteen distinct table column widths across the app where
three or four would do; they are pulled into .col-w-* so the ladder is
visible in one place, ready to normalise separately.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
* refactor(ui): move repeated inline styles onto shared classes
Six passes over src/pages and src/components, each matching a whole
`style={{ ... }}` attribute exactly so the swap is provably equivalent:
- the identical "waiting for the first response" wrapper, repeated
verbatim in 21 files
- text roles (size + colour combinations) onto .text-note / .text-meta /
.text-sub / .text-mono
- semantic colours, type-scale steps, .panel-title, .list-row
- spacing and weight steps, .stack / .hstack rows
- table column widths, small pills, chart legend swatches
- the remaining shapes appearing three or more times
One class of bug is worth calling out, because it is what a careless
style-to-class conversion produces and it is invisible to every check we
run. Adding `className="x"` to an element that already had a className
leaves TWO className attributes; JSX keeps the last and silently drops the
first, so `<i className={icon} className="text-xs" />` loses its icon while
passing eslint, `vite build` and the full Playwright suite. 112 of these
were introduced and repaired here. The gate added in a later commit fails
on them.
No visual change is intended beyond snapping off-scale font sizes onto the
type scale. Verified after every pass: eslint 0 errors, vite build passes,
Playwright page-render-smoke + navigation 22/22.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
* refactor(ui): convert fifteen pages onto named classes, add inline-style gate
Full per-page conversions, each one reading the page, naming the shapes it
actually has, and leaving inline only what is computed at runtime:
P2P 205->0 Traces 50->0 ConfigFieldRenderer 23->0 NodeDetail 29->1
ModelEditor 25->1 NodeInstallPicker 29->1 FineTune 175->2
Nodes 33->2 ImportModel 35->3 Talk 43->4 AgentJobDetails 27->4
Usage 97->6 Settings 24->6 Middleware 54->8 Backends 87->45
Every remainder is genuinely dynamic: a data-driven badge colour, a
`width: ${pct}%`, a tooltip's coordinates.
Naming the shapes made reuse fall out on its own. Nodes reuses the P2P
setup shapes (both present the same "no workers yet, here is how to add
one" flow) and Model Editor reuses the Settings section rail, which was
byte-identical. Two shared *style objects* also turned out to be classes
wearing a costume and were deleted: `monoCell` in Usage and `hintStyle` in
ImportModel.
Some fixes fell out of the conversion. The evaluation toggle on the
fine-tuning page was a hand-rolled div that rendered as a clipped circle;
it is the existing Toggle component now. That page's empty state pointed at
a "New Job" button that was scrolled off the top of the page, and now
carries its own call to action. And `.input--file` is added at the system
level rather than as a local hack, because ImageGen and VideoGen truncate
their file inputs the same way today.
scripts/inline-style-gate.mjs is the ratchet that keeps this from
regressing. It does not forbid inline styles; it fails when the total goes
UP (same discipline as the coverage baseline) and when an element carries
two className attributes. eslint would catch the latter via
react/jsx-props-no-duplicate-props, but that needs eslint-plugin-react,
which this project does not depend on, so the check lives next to the tool
that causes the problem.
npm run lint:inline-styles # check against baseline
npm run lint:inline-styles:report # per-file counts, worst first
npm run lint:inline-styles:write # refresh after converting
Net: 2,061 -> 611 inline styles. eslint 0 errors, vite build passes,
Playwright page-render-smoke + navigation 22/22, gate green on both checks.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
refactor(backends): extract shared package-system-libs.sh from package.sh
The arch-detect-and-copy-system-libs block (Darwin rpath / x86_64 / aarch64
loader + libc/libstdc++/libgcc_s/libm/libgomp/libdl/librt/libpthread) was
inlined verbatim in 31 backend package.sh scripts. Extract it into a single
sourced scripts/build/package-system-libs.sh, the CPU-side counterpart to
scripts/build/package-gpu-libs.sh and its sourcing contract.
Consolidating the copies fixes three drift classes that had crept in:
- libgcc_s.so.1 and libstdc++.so.6 were listed twice in 9 backends
(acestep-cpp, crispasr, moss-tts-cpp, omnivoice-cpp, piper,
qwen3-tts-cpp, silero-vad, stablediffusion-ggml, whisper); the shared
script copies each once.
- libgomp.so.1 was omitted from opus. OpenMP consumers dlopen it rather
than link it, so the missing copy only failed at runtime; the shared
script always includes it.
- the Darwin @loader_path/lib rpath was applied only in piper and
silero-vad; both now pass their packaged binary to the shared script,
preserving that behavior. Every other backend passes an empty binary
path so no rpath is added, preserving its current behavior.
Each backend's pre/post packaging steps (binary copy, run.sh, ldd closure
walks, ggml variant bundling, espeak/OpenBLAS extras, the ds4 validate step)
are preserved verbatim; only the inline if/elif/else arch block is replaced
by a single source line.
Signed-off-by: supermario_leo <leo.stack@outlook.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
* fix(backends): fall back to copying links when the filesystem rejects symlinks (#10890)
Backend installation extracts the OCI image tar via containerd's
archive.Apply, which calls os.Symlink directly. On filesystems that do
not support symlinks (notably CIFS/SMB mounts, commonly used to back the
/backends volume) the syscall fails with "operation not supported" and
the whole install aborts, leaving an empty backend directory. The CUDA
llama.cpp image trips this on the libcublas.so -> libcublas.so.12.x
symlink.
When archive.Apply fails with a link-unsupported error, reset the
staging directory and re-extract with a pure-Go walker that still
attempts real symlinks/hardlinks first and degrades to copying the link
target's contents in place when the filesystem rejects them.
mutate.Extract already flattened the layers, so the tar carries no
whiteouts to interpret. Link copies are deferred to a second pass so
forward references resolve.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]
* fix(oci): check deferred Close in copyFilePreservingMode (errcheck)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]
* fix(oci): reject path-traversal tar entries in the link-copy fallback
safeJoin sanitized "../.." entries by clamping them under root instead of
rejecting them, so a malicious entry was silently redirected rather than
refused. Join without the leading-slash trick and reject any entry whose
cleaned path resolves outside root; absolute link targets are still mapped
under root (image-root relative) rather than escaping.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]
* fix(oci): silence gosec on the validated link-copy file ops
Use hdr.FileInfo().Mode() instead of converting the int64 tar mode to
os.FileMode (removes two G115 overflow findings), and annotate the tar
extraction file operations with justified #nosec comments: every path is
validated by safeJoin against the extraction root before use (G304/G305).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]
* fix(oci): build extraction image from downloaded layers
Avoid appending downloaded layers to the original remote-backed image, which duplicates the layer stack and reopens the source during extraction. Building from an empty image preserves the flattened whiteout semantics while keeping extraction local.
Assisted-by: Codex:gpt-5
* fix(oci): materialize chained links in dependency order
Retry deferred link copies until their targets exist so soname chains work on filesystems without symlink support. Document that copied links can increase backend storage usage on CIFS and SMB mounts.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(api): add /v1/detokenize endpoint
Closes#1649.
Mirror of the existing /v1/tokenize path, requested by @benniekiss in
the issue thread for "complete API workflow" use cases that need to
turn token IDs back into text without local processing.
- Add Detokenize gRPC RPC with DetokenizeRequest{tokens} /
DetokenizeResponse{content} messages.
- Implement in the llama.cpp backend using common_token_to_piece, the
same primitive TokenizeString already uses internally.
- Other backends inherit the default Unimplemented from base.Base, in
line with how Detect, Rerank, etc. are gated per-backend.
- Wire up the Go gRPC interface, server, client, and in-process embed
wrapper alongside their TokenizeString counterparts.
- Add the schema types, ModelDetokenize wrapper, HTTP handler, route
registration, RouteFeatureRegistry entry (gated by FeatureTokenize so
no new feature flag is needed), and the discovery map entry under
ai_functions.
- Regenerated swagger reflects the new endpoint and types.
- Update authentication.md to list /v1/detokenize alongside /v1/tokenize.
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
* test(e2e): add mock backend tests for /v1/detokenize
Add Detokenize to the mock gRPC backend and wire up two e2e tests in
the MockBackend suite: one that posts known token IDs and asserts a
non-empty content response, and a round-trip that tokenizes first then
detokenizes the returned IDs.
Addresses reviewer feedback on #9620.
Assisted-by: Claude:claude-sonnet-4-6
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
* fix(kokoros): implement detokenize in the Rust backend service
The Detokenize RPC added in this PR grows the tonic-generated Backend
trait. Unlike the other languages there is nothing to inherit a default
from — Rust trait impls must list every method — so
backend/rust/kokoros failed to compile:
error[E0046]: not all trait items implemented, missing: `detokenize`
--> src/service.rs:72:1
72 | impl Backend for KokorosService {
Go backends pick up the Unimplemented default from base.Base, and the
generated C++/Python servicer bases default to UNIMPLEMENTED, which is
why the Rust backend was the only one that broke. kokoros is the sole
Rust crate in the tree, so this is the full extent of the fallout.
Return Status::unimplemented("Not supported"), matching how this same
file already gates tokenize_string and ~20 other unsupported RPCs.
Fixes the tests-kokoros and backend-jobs-singlearch-4 (-cpu-kokoros)
failures on the previous head.
Assisted-by: Claude:claude-opus-5 cargo
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
---------
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
On 2026-07-30, 12 of the 23 queued runs of this workflow were commits like "add
1 new model to gallery" or a docs fix, each rebuilding all 18 container images.
That was roughly 216 queued jobs producing byte-identical output, in a queue
holding 1071 jobs with an oldest entry two days old.
Verified against the shipped Dockerfile before assuming it: the final stage
copies only entrypoint.sh, healthcheck.sh and the local-ai binary, there is no
go:embed of gallery/ or docs/, and the gallery is fetched at runtime from
github:mudler/LocalAI/gallery/index.yaml@master. A gallery-only commit produces
an identical image, and the gallery change reaches users through GitHub whether
or not an image is rebuilt, so nothing is delayed by skipping.
Add a `changes` job that decides once whether the push can affect an image; the
other 11 jobs take `needs: changes` and an `if:` on its output.
A job gate rather than paths-ignore on the trigger, for two reasons that both
fail silently if got wrong:
- paths-ignore on `push` also applies to tag pushes, and a tag created on an
existing commit carries an empty commits list. That would skip the release
image build with no failure anywhere. The gate short-circuits to build for
refs/tags/*, and for a base commit that is missing, zero or unresolvable --
the same run-everything posture the backend matrix filter takes for a
truncated diff.
- the merge jobs use `if: ${{ !cancelled() && ... }}`, and !cancelled() is
true when a dependency is skipped, so they need the gate named explicitly
or they would try to merge manifest lists for images never built.
Checked the decision logic against real commits from the queue: the two
gallery/docs commits resolve to build=false, the two code commits to build=true,
and all three fallback paths (tag, zero base, unresolvable base) to build=true.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Document all stateful container paths, explain upgrade behavior and UnRAID mappings, and correct the obsolete troubleshooting mount target.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
docs: clarify reverse proxy bulk job guidance
Mention ingress controllers as another place to configure equivalent upstream response timeouts, and include an example private LocalAI URL for trusted bulk jobs.
Assisted-by: Hephaestus:openai/gpt-5.5 [opencode]
Signed-off-by: Owen Adirah <owenadira@gmail.com>
Stage the negative GPU-layer sentinels expected by the upstream argument parser, then restore LocalAI resolved values unless a passthrough flag explicitly overrides them. This avoids the parser assertion that terminated the backend for any generic option.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Version-pin bumps dominate PR volume: 48 update/* PRs in the week to
2026-07-30, from 16 pins, each a two-line diff. bump_deps.yaml runs a 28-entry
matrix daily and opens one PR per moved pin; CRISPASR and the gallery checksum
produced one every day, ik-llama-cpp six in seven days. Almost all of them edit
nothing but a single backend/*/<name>/Makefile.
Neither image-pr.yml nor build-test.yaml can observe such a change. `make build`
is `go build ./cmd/local-ai`, GoReleaser builds that plus ./cmd/launcher, and
the core image's final stage ships only entrypoint.sh, healthcheck.sh and the
binary. The per-backend trees are copied into the builder but nothing in them
reaches the output. That is 7 + 3 jobs per bump PR that cannot fail for a reason
the diff caused, roughly 410 jobs a week.
Add backend/{cpp,go,python}/** to the paths-ignore of those two workflows. The
inputs that do reach the binary are deliberately outside those prefixes and so
still trigger a full run: backend/backend.proto (protogen-go), go.mod/go.sum
(the go mod tidy before-hook), and backend/Dockerfile.* .
Not applied to the workflows that genuinely read that tree:
test.yml TEST_PATHS names ./backend/go/{cloud-proxy,local-store,
valkey-store}/...
lint.yml .golangci.yml carries backend/-scoped rules
tests-e2e.yml the e2e suite drives real backends over gRPC
backend_pr.yml its whole job is rebuilding the changed backend
Simulated against the change shapes that occur in this repo. Pin bumps and
python requirement bumps skip; backend.proto, go.mod, a core Go edit, a
backend/Dockerfile edit and any mixed diff all still run, since paths-ignore
skips only when every changed file matches.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
.agents/ci-caching.md stated that `cache-to: type=registry,mode=max` "exports
the cache mount data into the registry cache, so subsequent builds restore it".
BuildKit does not do that. A `--mount=type=cache` lives in the builder's local
state and is not part of a registry cache export, and every CI job gets a fresh
runner with a fresh builder, so /root/.ccache starts empty on every build.
The compile script already prints `ccache -s` after a `ccache -z`, so the
evidence was sitting in the logs:
job 89766266951 llama-cpp cublas-13 6369s 0 / 889 hits (and 0 / 1778)
job 89766267281 llama-cpp hipblas 8160s 0 / 537 hits
job 90210828110 llama-cpp cublas-12 5673s 0 / 813 hits
The first two are the control. Their commit, 90355cd44, changed exactly one
file: backend/go/magpie-tts-cpp/Makefile, nowhere near llama.cpp. The engine
source was byte-identical to the previous build, which is the case this section
claims ccache serves, and the hit rate was still 0.00%. A restored-but-stale
cache would show partial hits; 0-of-N is an empty cache.
So llama-cpp, ik-llama-cpp, turboquant, bonsai, ds4 and privacy-filter pay the
ccache wrapper overhead and get nothing back, and multi-hour C++ builds
recompile identical translation units every time.
Record this rather than silently extending it. Wiring the same mount into
Dockerfile.golang, which covers 215 of the 434 matrix entries, measured 18%
faster locally on a rebuild after a source edit with a 71.5% hit rate, but only
because that test reused a single builder across both builds. In CI it would be
a no-op. The note spells out what would actually work (ccache remote_storage or
sccache with a real backend, or round-tripping the cache dir through
actions/cache) and what each costs.
Also correct the composite-actions list, which still named test.yml as a
free-disk-space consumer after #11219 removed it.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Dockerfile.golang builds 215 of the 434 matrix entries: every ggml/C++ engine
wrapped in Go. Each of those Makefiles clones an upstream repo at a pinned SHA
and compiles it once per SIMD variant (depth-anything-cpp builds four: avx,
avx2, avx512, fallback), and those variant targets depend only on the clone.
They cannot observe a change anywhere else in the LocalAI tree.
The compile sat below `COPY . /LocalAI`, so any edit anywhere invalidated it and
recompiled C++ that had not changed. Move it above that COPY, behind a copy of
only the backend's own directory.
This lands the compile in the part of the image the registry cache already
restores. Measured on two real CI builds of this Dockerfile (jobs 90551029008
and 90551028904, both fresh runners): 13 of 17 layers CACHED from
quay.io/go-skynet/ci-cache. The uncached tail is exactly `COPY . /LocalAI`, the
git-config RUN and the build RUN. Putting the engine above the COPY moves it
from the uncached tail into the cached region.
Local measurement, depth-anything-cpp CPU, rebuild after editing a Go file
outside the backend:
master 78s, 216 C++ objects compiled
this change 25s, 0 C++ objects compiled (67% faster)
Note what this deliberately is not. An earlier attempt wired a
--mount=type=cache ccache into the same RUN. BuildKit does not export cache
mounts to a registry cache, so that measured well locally and is a no-op in CI
(see the ccache section of .agents/ci-caching.md). This change relies only on
ordinary layer caching, which the 13-of-17 figure above shows already works
here.
The layer copies the backend's whole directory rather than just the Makefile:
the CMake targets also need CMakeLists.txt and the file list differs per
backend. The cost is that editing a backend's own Go sources invalidates its
engine layer. The expensive cases are unaffected, since a shared-build-input or
backend.proto change, the weekly full-matrix cron and a tag push all rebuild
every backend while touching none of their directories.
Scoped to one backend for now: only depth-anything-cpp gains the `engine`
target. The other 27 fall through the `make -n engine` guard and build exactly
as before, verified against local-store and silero-vad. Rolling the target out
to the remaining 12 backends that define VARIANT_TARGETS is mechanical once this
is confirmed against the registry cache on master.
One caveat on merge: inserting layers shifts the cache keys, so the first build
of each entry after this lands is a full miss. It pays for itself on the second.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Measured over the week to 2026-07-30, 97% of CI wall-clock is queueing and 3%
is execution: a median 5-hour queue against a 4-20 minute median job. With the
queue saturated, throughput is concurrency divided by service time, so cutting
execution time raises the drain rate directly. Two steps stood out as paying
nothing for what they cost.
test.yml: drop the free-disk-space step (~3.1min per run, ~22 h/week). That
action exists to make room for docker buildx layers and this job runs no buildx
step. It was also sized for a `make test` that downloaded multi-GB GGUF/whisper
fixtures and built llama-cpp/whisper/stablediffusion-ggml; the test-suite reorg
moved all of that into tests/e2e-backends and tests/e2e-aio, as the Makefile
test target already records. Its tool-cache:true wipe was additionally deleting
/opt/hostedtoolcache, forcing setup-go and setup-node to re-download toolchains
that ship preinstalled on the runner.
build-test.yaml: build only the host target on pull_request. The three-platform
cross-compile (linux/amd64, linux/arm64, darwin/arm64) is the bulk of that job's
~6.6min median, ~47 h/week, and nothing consumes a PR's binaries. goreleaser's
--single-target still runs every before-hook (protogen-go, react-ui, go mod
tidy), so the "is the release build broken" signal is unchanged. master pushes
and tags keep building all three.
Also record why the Linux Go workflows pass cache: false to actions/setup-go,
since it reads as an oversight and is not. Set up Go has a median of 11 seconds
on those runners, so there is nothing to win, and the repo already sits at
GitHub's 10 GB Actions cache ceiling with 31 entries, where each setup-go entry
is 222-375 MB on Linux and up to 1.4 GB on macOS. Re-enabling it would evict
something that is earning its space.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(vllm): apply Options[] engine flags before engine init (#11130)
CLI-style flags in a model's `options:` array (`--quantization:gptq_marlin`,
`--enable-prefix-caching`, `--kv-cache-dtype:fp8_e5m2`) were discarded: the
backend only ever read `tool_parser`/`reasoning_parser` out of Options[], and
did so *after* `AsyncLLMEngine.from_engine_args()`, where nothing it set could
still reach the engine.
Map `--` prefixed options onto the AsyncEngineArgs dataclass before the engine
is constructed. Names are normalized the way vLLM's CLI spells them
(`--enable-prefix-caching` -> `enable_prefix_caching`), values are coerced to
the target field's type (bare flag -> True for booleans), and unknown or
uncoercible flags warn and are skipped instead of failing the load, since
Options[] is a bag shared with backend-level settings. Field types come from
the annotation's base so `Literal["auto", "float16"]` (vLLM's dtype) is not
mistaken for a float.
Precedence is typed proto fields -> `options:` -> `engine_args:`. The
production engine_args defaults seeded in hooks_vllm.go therefore skip any key
the user already set as an option, otherwise the later engine_args pass would
silently override it. Parser lookups now accept both spellings, so
`--reasoning-parser:qwen3` selects LocalAI's parser as well.
The helper's tests are stdlib-only and run in the lint workflow's
dependency-light job via `make test-python-helpers`.
Assisted-by: Claude:claude-opus-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
* backend(audio-cpp): add the native build scaffold
Links 0xShug0/audio.cpp engine_runtime through its public framework headers
and serves Health/Status. Model loading and the audio RPCs follow.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): keep the build-tree rpath at $ORIGIN
Upstream sets CMAKE_BUILD_WITH_INSTALL_RPATH in its own directory scope, so
CMake was appending its build-tree library dir to our target and baking an
absolute build-host path into the shipped binary. Set BUILD_WITH_INSTALL_RPATH
on the target so a package that forgets to bundle libggml*.so fails on the
build machine too, instead of only on a user's box.
Also document why EXCLUDE_FROM_ALL must stay on the add_subdirectory call,
correct the claim that Ubuntu ships no gRPC CMake config, stop the pin comment
from repeating the assignment token that bump_deps.sh rewrites, and make
test-engine fail rather than pass when no test is registered.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): parse namespaced model options
Splits option entries on the first colon so path values survive, and routes
load./session. prefixes to the upstream load and session option maps.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): reject out-of-range numeric model options
std::atoi is undefined once the digits exceed long and in practice wraps, so
device:2147483648 was accepted and handed the ggml backend selector a device
index of -2147483648 from a function whose error text promises a non-negative
integer. Parse with strtol and reject on ERANGE, on a value above INT_MAX, and
on any unconsumed trailing input. The error strings are unchanged.
Name the whole entry in the unknown-key error too: an entry like ':value' has
an empty key and left the user nothing to grep for in their YAML.
Tests look keys up through a helper instead of map::at, so a prefix off-by-one
fails one named check rather than aborting the binary and skipping the rest of
the suite, and cover the overflow, negative and non-numeric paths.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): route LocalAI RPCs onto audio.cpp tasks
Task-major resolution over the family's advertised capability set, with the
voice-reference and instructions signals selecting cloning and voice design,
and a streaming-to-offline fallback for server-streaming transcription only.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): use upstream's 'spk' task name and pin the preference order
The SpeakerRecognition short name was 'spkrec', which audio.cpp neither prints
nor parses; a name copied out of audio.cpp was rejected and a pinned 'spkrec'
would not survive the engine boundary. Emit 'spk', keep 'spkrec' as an
input-only alias, and correct the known-tasks lists.
Three assertions were vacuous because their fixtures advertised a single task,
so reversing a preference order or dropping the RPC name and the attempted
pairs from the capability error all passed. Give them fixtures that can tell
the orderings apart.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): convert sample, time and PCM units
Integer nanosecond conversion so 44.1 kHz stays exact, float seconds for the
VAD and diarization messages, and saturating s16le encode so an overshooting
sample cannot wrap to the opposite sign.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): harden seconds_to_samples against NaN and overflow
seconds_to_samples is the one entry point fed by untrusted-shaped input: a
float-seconds timestamp off the wire, or a boundary from a model that diverged.
Its guard covered only the low side, so NaN and out-of-range values fell through
to an undefined double-to-int64 cast and came back as INT64_MIN. A hugely
negative sample index used later as an offset or a length is a wild pointer
rather than merely a wrong timestamp. Reject NaN with the !(x > 0) form and
saturate before the cast.
Also round instead of truncating there. These functions exist to cross the float
seconds boundary the VAD and diarize messages use, and truncation lost a sample
about half the time on the samples-to-seconds-and-back round trip, starting at
n=1.
Pin the decode scale at INT16_MIN, pin nanosecond truncation on a nonzero
fraction, and record why the clamp argument order in f32_to_s16le is
load-bearing for NaN.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): map NaN PCM samples to silence explicitly
f32_to_s16le relied on std::min argument order to keep a NaN sample away from
std::lround, whose result is unspecified for NaN. That was too subtle to rest on
a comment, and the comment was itself wrong: it warned against a spelling that
the outer std::max already catches, while three real spellings leak, including
std::clamp, which is the idiomatic C++17 way to write the same clamp and so the
likeliest future edit.
Divert NaN before the clamp and encode it as 0. A NaN sample rendered as a
full-scale click is worse audio than a dropped one, and this unit converts audio
that may have originated off the wire.
Pin it with an exact-value check rather than a range check, since all three
outcomes the plausible spellings produce are finite and inside full scale, plus
an invalid-operation check that fails unless the NaN is diverted before any
ordered comparison. That second check is what catches modernizing the clamp and
dropping the guard together.
Also bound the seconds round-trip comment, which claimed unconditionally what
holds only below roughly 2^23 samples, and document NaN, saturation and that
bound in the header.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): assemble transcripts from runtime spans
The top-level transcript text is TaskResult.text_output verbatim. audio.cpp
carries text nowhere else: speech_segments, speaker_turns and word_timestamps
hold spans and labels only, so deriving the text from them empties the
transcript for any producer that omits word timing, VibeVoice diarized ASR
included. Fixtures cover every observed producer shape.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): keep a nested speaker turn's own label
A segment sourced from speaker_turns re-derived its speaker by greatest
overlap. A turn's overlap with its own span is the largest possible, so a turn
nested inside another speaker's turn could only tie with the container, and the
tie went to whichever came first. sortformer_diar binarizes each speaker
independently and sorts by start sample, so the container always comes first
and the interjecting speaker was silently erased from DiarizeSegment.speaker.
choose_segment_spans now carries the label out with the span.
Also pins the nearest-segment fallback against measuring from either endpoint
or from segment position, which a trailing-only stray word could not do, and
exercises the empty-word guard in join_words. Two fixtures that pin a rule but
do not mirror any pinned family are relabelled defensive.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serialize runs with a wedge-aware guard
audio.cpp sessions are not reentrant and a wedged CUDA call cannot be
cancelled, so a plain mutex would pile every worker thread behind a stuck GPU.
Callers waiting past the configured bound, or arriving while the holder has
already overrun it, fail fast instead.
A caller that queues behind a healthy run deliberately does not stamp the
clock: only the thread that takes the lock does. Stamping on arrival would
restart the wedge clock on every request and hide a stuck run from everyone
behind it, which is the pile-up this guard exists to prevent.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serialize inference through an InferenceLane
One audio.cpp model is loaded per backend process and its sessions are not
reentrant, so concurrent gRPC handlers have to take turns. Serialization alone
is not enough: a wedged GPU call cannot be cancelled from userspace, so an
unbounded queue behind one stuck run would swallow every gRPC worker thread
until the process is useless.
InferenceLane gives handlers a lane with room for one runner. LaneEntry occupies
it for a scope and gives it back on every exit, including an exception, and is
the only way to take the lane at all: occupy/vacate are private with LaneEntry
as the sole friend, so a caller cannot acquire without holding something that
releases. LaneEntry is immovable on purpose, because a moved-from entry would
have to stop releasing while the lane still recorded it as occupied.
A caller either waits indefinitely or brings a millisecond budget. A bounded
caller that cannot get in fails instead of waiting on, and a bounded caller
whose budget is already shorter than the age of the run in the lane fails
immediately, which is what stops a queue forming behind a wedged run. The two
failures carry different text: one names the wait it exhausted, the other states
the measured age of the run without claiming to know why it is long, since a
short budget meeting a legitimately long run lands there too.
The run's age is stamped only after acquisition. A waiter that published itself
as holder would restart the measurement and hide a genuinely stuck holder from
every caller behind it.
Budget negotiation and the overrun decision are pure functions taking their
inputs explicitly, so both are covered without threads or sleeping. The
per-model ceiling arrives as an int of milliseconds; a request may tighten it
and may never loosen it.
Replaces the previous run_guard unit, which was a derivative of an
Apache-2.0 file upstream and could not stay in an MIT tree. Written from a
behaviour contract with no reference to the removed code.
Tests: 65 checks, standard library only, single translation unit, clean under
-Wall -Wextra. Mutation tested at 23/23 killed; two of those mutants exposed
missing coverage and the tests were extended until they died. ThreadSanitizer
clean.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): make the B10 test able to fail, and document LaneEntry
Review of the previous commit found the B10 test could not fail for the reason
it was named. It aged the in-flight run to about 120 ms and then tried two
budgets, 30 ms and 60 ms, both under that age, so both callers took the
fail-fast path. "The two failure modes do not share one message" was comparing
two fail-fast messages that differ only in the budget they print, and the
timeout path was never reached. The second budget is now 400 ms, well over the
run's age, so that caller queues and times out, and a new check asserts which
path each caller took instead of inferring it from inequality. A mutant that
makes the fail-fast path emit the timeout message previously died only on B4 and
B8 checks; it now also dies on B10.
Comment-only changes elsewhere. LaneEntry now says it is not reentrant and does
not detect reentrancy: a second entry on a thread that already holds the lane
surfaces as LaneUnavailable with a positive budget, but parks silently in
unbounded mode, which matters because a handler may hold one across a whole
stream. The immovability note now names the shapes that work, an optional
emplaced in place or a unique_ptr, rather than saying to hold the entry
indirectly without saying how; all three documented forms were compiled before
being written down, which is how the note came to say that an optional of an
immovable type cannot itself be returned.
The header's explanation of why fail-fast exists is reworded. Two clauses traced
back to a specification written after reading the Apache-2.0 upstream header,
and while that was judged de minimis, this unit was rewritten precisely to carry
no upstream expression at all.
The margin table in the report was also wrong about which wall-clock margins are
load-sensitive: there are four, not one, and the tightest is the B3 arrival
check, which is now flagged at the call site. No margin value changed and none
moved across 65 runs.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): gate model loading on the audio.cpp family
Refuses any GGUF without an audiocpp.model_spec.family key and any non-GGUF
path without an explicit family option, so the model loader's greedy backend
probe cannot bind an unrelated llama.cpp GGUF to this backend (#9287).
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): load models and cache sessions per task
Loads one ILoadedVoiceModel and creates an IVoiceTaskSession lazily per
(task, mode), so the same model serves both the unary and streaming RPCs.
LoadModel derives the family from GGUF metadata or an explicit option and
fails with INVALID_ARGUMENT otherwise, so a failed load is a gRPC error the
backend probe can see.
audiocpp_backend::Task mirrors engine::runtime::VoiceTaskKind positionally,
and drift there is silent: every unit still compiles and every test still
passes while the backend runs a different task. Two mechanisms pin it. The
static_asserts in loaded_model.cpp catch an insertion or a reorder, and
-Werror=switch on that one file turns an appended upstream enumerator into a
build failure rather than a warning in a 600 file log.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): stop aborting the process on SIGTERM
The signal handler called grpc::Server::Shutdown directly. Shutdown takes an
absl::Mutex, which is not async-signal-safe: the handler can interrupt a thread
already holding that mutex, and abseil's deadlock detector responds by aborting.
Every SIGTERM therefore ended in exit 134 and a 'dying due to potential
deadlock' stack rather than a drained shutdown.
The handler now sets a lock-free atomic and returns. Server::Wait moves to a
helper thread so the main thread can poll that flag and call Shutdown itself,
outside any signal context. A condition variable would not have helped, because
notifying one from a handler is not async-signal-safe either.
SIGTERM and SIGINT both exit 0 with no stack trace, where both previously
exited 134.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): correct the status, lifetime and state contracts of LoadedModel
An environment fault during session creation was reported as UNIMPLEMENTED. A
missing libggml-cpu-*.so surfaced to the client as 'family silero_vad advertises
vad/offline but refused to create the session: Failed to initialize CPU
backend', which tells LocalAI the model cannot do this and must never be
retried, and sends an operator hunting a capability bug instead of a packaging
one. A throw from create_task_session is now a plain runtime_error, so it maps
to INTERNAL. Only a null return, where the family genuinely declined, stays a
CapabilityError.
The model.'s task: option was parsed and then dropped: it lived in a local that
died at the end of LoadModel and had no route to RequestShape::pinned_task.
LoadedModel now keeps it and exposes pinned_task().
The global model becomes a shared_ptr reached through snapshot(). An audio RPC
runs for seconds and cannot hold g_model_mu for its duration, so under a
unique_ptr a Free arriving mid-request would destroy the model underneath it.
Handlers now take a counted reference and whichever finishes last does the
teardown, outside the lock.
session_for documents the streaming state contract rather than resetting the
session itself. Resetting on a cache hit was tried first and is not possible:
silero_vad throws 'session prepare() must be called before Silero VAD reset()',
so it would turn an ordinary second fetch into a hard error. start_stream's base
implementation is already a reset, so a caller that runs prepare then
start_stream per stream gets a clean session; a probe against the bundled
silero_vad confirms an identical replay when it does and a carried-over stream
when it does not.
Also: an unknown backend: name is rejected before the model loads rather than
after; MainGPU is parsed instead of passed through std::atoi, which turned
'gpu1' into device 0 silently; and device carries a device_set flag, because 0
is both the default and a real device index, so MainGPU was overriding an
explicit device:0 that the neighbouring threads: handling promises will win.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serve the VAD and Diarize RPCs
Both emit float seconds, converted from the runtime's sample-index spans, and
both take a counted reference to the loaded model through snapshot() and hold it
for the whole call: a Free arriving mid-request drops only the global's
reference, so whichever request finishes last destroys the model instead of one
of them running on freed weights. An AddressSanitizer build reproduces exactly
that heap-use-after-free inside ggml_vec_dot_f32 when the handler keeps a raw
pointer instead, which is why the shape is what it is.
The inference lane is taken before session_for, not after. session_for reads and
writes an unsynchronised session cache and the offline run calls prepare(),
which mutates the session, so both belong inside the lane.
Diarize routes before it reads the input file, so a family that cannot diarize
at all says so rather than complaining about the audio first. Its per-segment
text stays empty because audio.cpp's SpeakerTurn carries a span and a speaker
label only, and nested or overlapping turns are passed through untouched: a
sortformer turn inside another speaker's turn is correct output for overlapped
speech, and LocalAI is overlap-tolerant downstream. Duration counts frames
rather than floats, so a stereo input does not report twice its length.
Verified end to end against upstream's bundled silero_vad, which needs no
download, using the bundled 16 kHz speech asset: a synthetic tone returns
nothing, correctly, because silero detects speech and a sine is not speech.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): enforce ModelIdentity on VAD and Diarize
audio-cpp was the only C++ backend without the model-identity guard, and no
later task in the plan added it. pkg/grpc/server.go enforces checkModelIdentity
on exactly these two RPCs, for the reason #10952 records: in distributed mode a
worker can recycle a stopped backend's gRPC port for another model's backend,
and the controller's liveness-only probe cannot tell a stale cached route from a
live one. Without this guard a stale route gets a different model's VAD or
diarization answer back with a 200.
The loaded identity lives on LoadedModel rather than in a separate global, which
is where this differs from llama-cpp. A handler holding the model through
snapshot() then necessarily judges against the identity that model was loaded
with, and a concurrent reload cannot swap one without the other. The refusal is
NOT_FOUND carrying the verbatim grpcerrors.ModelMismatchSentinel substring.
session_for and run_offline now take a const LaneEntry & proof-of-holding
parameter. The rule that both must run under the inference lane was prose, which
is exactly how the plan came to specify the inverted order; it is now a compile
error. Restoring the inverted order fails to build rather than racing on an
unsynchronised session map with a mutating prepare().
Diarize's speaker-hint comment claimed the dropped hints were "not a silent
failure". From the caller's side that is what they are, and backend.proto
documents num_speakers as forcing, so the comment now says plainly that the
forwarding is dead for sortformer and that the family which lands must either
honour num_speakers or refuse it. read_audio_file inspects the error_code from
exists(), so an unsearchable parent directory no longer reports as a missing
file. The VAD handler records the stimulus that actually works, since silero
correctly ignores synthetic tones and the next task would otherwise rediscover
that.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): make the lane and identity guards structural
Two hardenings ahead of the eleven handlers still to be written, both of which
get harder to retrofit later.
The lane proof-of-holding parameter was a const reference, which binds to a
temporary, so session_for(rpc, shape, model->acquire(0)) compiled. Each such
temporary dies at the end of its own full-expression, releasing the lane between
two calls that must share one: precisely the split the parameter exists to
prevent, and the form a future author is most likely to reach for because it
reads as tidy. A non-const reference requires an lvalue, so the temporary form
now fails to compile while the named-local handlers build unchanged. The header
comment no longer implies the check is total either: it proves a lane was taken,
not that it is this model's lane.
The identity check was two lines each handler had to remember, with nothing
failing if a new one forgot them and no C++ equivalent of
model_identity_modalities_test.go to notice. snapshot() becomes
snapshot_unchecked(), whose only legitimate caller is Status, since HealthMessage
carries no ModelIdentity. Handlers go through snapshot_for(), which takes the
counted reference, refuses when nothing is loaded, and runs the identity check
before anything can route. Every handler already has to call something to obtain
the model, so the guarded call is now the shortest path and skipping it means
deliberately typing snapshot_unchecked. A convention that has to be remembered
can rot; this cannot.
Verified: the temporary-argument and inverted-order forms each fail to compile
with the expected diagnostic, the real handlers build, and bypassing the guard in
Diarize alone turns the identity test red on that RPC while VAD stays green.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serve the AudioTranscription RPC
Adds result_map, the engine-to-proto boundary, and wires the offline
transcription RPC.
The handler branches on the ROUTED task: for Asr the request's prompt is
whisper-style decoding context and becomes a request option, for Alignment
the same field IS the transcript to align and becomes the text input.
Routing has already decided which.
The result text is TaskResult.text_output verbatim and is never derived
from the segments. audio.cpp carries transcript text in text_output and
nowhere else, so deriving it returns an empty transcript for every
producer that reports segments without word timing. transcript_assembly
already enforces that; this commit's job is not to undo it at the proto
boundary, and result_map_ctest pins it there.
read_audio_file now takes the sample rate the caller needs. Both file-fed
speech handlers ask for 16 kHz mono, for two reasons: silero_vad and
sortformer_diar refuse anything else outright, which turned an ordinary
44.1 kHz upload into INTERNAL, and nemotron_asr emits word timestamps in
its own 16 kHz feature domain whatever the input was, so only a 16 kHz
buffer makes the emitted nanoseconds right. Zero keeps the file's native
rate and channels, which is what source separation will need.
LoadedModel::check_can_serve answers a capability refusal before the lane
is taken and before the input file is read. Routing is a pure read of the
immutable capabilities, so a model that cannot serve an RPC no longer
waits out somebody else's run to say so. VAD and Diarize use it too.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): stop linking sentencepiece's vendored protobuf
engine_runtime links sentencepiece, whose default SPM_PROTOBUF_PROVIDER
builds the protobuf-lite 3.14.0 sources it vendors. The generated
backend.pb.cc is built against the toolchain's protobuf 3.21.12. Both
ended up in the binary: 476 google::protobuf:: symbols came from the
archive, 278 of them also defined by libprotobuf.so, and the archive won,
because once ld pulls a member in for sentencepiece's own code every
reference binds to the definitions that member carries.
The visible symptom is one function.
ParseContext::ParseMessage(MessageLite*, const char*) is what a generated
_InternalParse calls for a submessage field and for nothing else, so flat
messages parsed and nested ones did not: a TranscriptResult carrying
segments serialized to correct bytes that the same process could not read
back, and TranscriptLiveRequest, a oneof of submessages, could not have
been parsed at all. Underneath that, 3.21 generated code was running 3.14
arena, ArenaStringPtr and ExtensionSet code.
-Wl,--exclude-libs does not fix it. It makes those symbols LOCAL in
.dynsym and the parse still fails, because the binding was decided at
static link time and no visibility flag revisits it.
Setting SPM_PROTOBUF_PROVIDER to "package" before add_subdirectory points
sentencepiece at the protobuf the generated code was already built
against. Zero google::protobuf:: definitions remain in the executable
afterwards, every nested message round trips, and citrinet_asr, which
parses a SentencePiece ModelProto at load time and would break first if
this were wrong, still tokenizes and transcribes correctly.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): fix the segment text a transcription response is built from
Segment text is not decoration. core/http/endpoints/openai/transcription.go
routes response_format text, srt, vtt and lrc through
schema.TranscriptionResponse, which builds the entire body out of
Segments[].Text and never reads the top-level text. So for those four
formats the segment text IS the response.
nemotron_asr emits one word_timestamp per SentencePiece token, and the
word boundary is carried as a LEADING SPACE on the piece ("So", "me",
" call"). join_words inserted a space unconditionally, so
response_format=text returned "So me call me na ture ," while the
correct sentence sat unread in the top-level field. The separator is now
chosen from the words themselves: whole words are space-joined, subword
pieces are concatenated, and one leading space anywhere selects the
latter. Concatenating the real nemotron pieces reproduces text_output
exactly, verified end to end.
This does not touch the top-level text, which is still text_output
verbatim. The rule that forbids deriving the transcript from the segments
is about the direction segments -> text; segment text has no source other
than its words.
Two smaller corrections in the same area:
timestamp_granularities ["word"] set only "word_timestamps", a key no
family in the pinned upstream reads. It now sets "return_timestamps",
which qwen3_asr does read and which both runs its forced aligner and
shortens its chunk window, so asking for word granularity no longer
silently returns nothing.
The request-option comment claimed more than it delivered. prompt,
translate and temperature are read by no ASR family, and are forwarded
only so a family adopting them works unchanged; the comment now says so
per key, and gives TranscriptRequest.diarize the same explicit treatment
threads already had.
Also: the shipping target now carries -Wall -Wextra -Wpedantic, which it
never did, so "the build is clean" starts meaning something; and
fill_transcript_result no longer swallows a null response pointer, since
answering OK with an empty transcript is the one failure mode this unit
exists to prevent.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serve the AudioTransform RPC
Covers voice conversion, singing voice conversion, speech to speech and source
separation, the four tasks LocalAI's AudioTransform can represent.
AudioTransformResult carries one dst while htdemucs and mel_band_roformer
produce several named stems from a single run, so inference runs ONCE, every
stem is written as a sibling file <dst-stem>.<name>.<ext>, and params[stem]
selects which one dst receives, defaulting to vocals and falling back to the
first output. An unknown stem name is INVALID_ARGUMENT listing the real stem
names rather than a silent substitution, and the selection happens before the
first write so a refused request leaves no files behind. params[stem] is
consumed here and is not forwarded into the engine's request options.
The stem decision lives in stem_selection, which is stdlib only and therefore
tested by backend/cpp/run-unit-tests.sh. It also validates the names, because
they come from the model (htdemucs reads them from the GGUF's config.sources)
and each becomes a component of a path this backend writes: a name carrying a
path separator would escape the caller's output directory, and two stems
sharing a name would silently overwrite one another.
Both files are read at their native rate and channel count. Separation forces
it, since demucs and roformer refuse any rate but 44.1 kHz and lose the stereo
image that separates a centred vocal from a wide mix. The conversion families
all resample internally (seed_vc, vevo2, miocodec, chatterbox were each
checked), so passing the file through unchanged is also strictly better than
band limiting it to 16 kHz first.
Verified end to end against htdemucs f16 on a 44.1 kHz stereo mix: four stems
plus dst, dst byte identical to the selected stem, params[stem] selecting a
different one, an unknown stem refused with no files written, and mono input
preserved as mono output. Also against miocodec for the single output path,
where params[stem] is refused rather than ignored.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): refuse an impossible stem early, and stop blaming the caller for a failed write
Four fixes from the first review of the AudioTransform RPC.
check_can_serve now returns the resolved route, so params[stem] on a route that
is not source separation is refused from the route instead of after a full
inference: 11 ms rather than the 4.5 s a miocodec conversion costs, and far
worse on seed_vc or vevo2. The post-run refusal stays as the backstop for a
separation-routed family that returns no stems anyway. The typo'd-stem-name
case still needs the run, since no framework header publishes the stem names
before one.
Stem names carrying control bytes are refused. GGUF strings are length prefixed
and demucs reads its sources from JSON, so an embedded NUL survives to here:
two names differing only after the NUL are distinct std::strings, so the
duplicate check passes them, and then path::c_str() truncates both and they
open the same file. That is exactly the silent overwrite the duplicate check
exists to prevent, with the .wav lost as well.
A failed write is now INTERNAL rather than INVALID_ARGUMENT. The destination is
LocalAI's own generated-content directory, not anything the caller named, so a
full disk or a permission fault there is a server fault and is worth retrying,
which is the opposite of what INVALID_ARGUMENT tells a client. An empty output
path stays INVALID_ARGUMENT.
Two comment corrections and one clarification: the separators' required rate is
their checkpoint's declared samplerate rather than a hardcoded 44100, seed_vc
resamples with soxr and falls back to sinc-hann, and the "no files left behind"
guarantee covers a refused request, not a write that fails partway through the
loop.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(audio-transform): stop folding every upload to 16 kHz mono, and name the separation stems
Two defects that made source separation unusable through LocalAI's own API,
even though the backend served it correctly over gRPC.
/audio/transform normalized every upload to 16 kHz mono s16 through
utils.AudioToWav, with no way past it. htdemucs and mel_band_roformer refuse
any rate but their checkpoint's own and separate a centred vocal from a wide
mix using the stereo image, so every separation request through the HTTP API
died with "HTDemucs prepare() sample rate mismatch: expected 44100, got 16000"
while the same call over gRPC worked. The fold is not wrong, it is
backend-specific: LocalVQE's echo cancellation genuinely wants 16 kHz mono and
needs the reference in the same shape. So it becomes a declaration,
BackendCapability.AudioTransformInputMono16k, set for localvqe and for nothing
else. A backend that declares nothing gets its upload unchanged, which means no
backend has to opt in to work. utils.AudioToWavPreservingShape is the
non-folding conversion: a 16-bit PCM WAV passes through byte for byte at any
rate and channel count, anything else is transcoded to WAV with its rate and
channel layout kept.
The other defect is that the run-once stem design bought nothing. A separation
backend writes every stem beside dst from one inference, but AudioTransformResult
carried only dst, so the other three were files no caller could find and a
caller wanting all four had to run four separations. AudioTransformResult grows
a repeated AudioTransformStem, the backend fills it, core/backend validates that
each path really is inside the generated-content directory it handed over, and
the endpoint publishes them as an X-Audio-Stems JSON header beside the existing
X-Audio-Input-Url. JSON because a stem name is the model's own string and could
contain any separator a hand-rolled format would use.
Verified end to end through the HTTP endpoint with htdemucs f16 on a 44.1 kHz
stereo file: 200 with a 44.1 kHz stereo body, all four stems named and fetchable
through /generated-audio/, body byte identical to the selected stem, and
params[stem]=drums returning a different one. The same upload sent to a model
whose backend is localvqe still reaches the backend as 16 kHz mono, confirmed
both by the engine's own rate refusal and by the persisted input file.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(audio-transform): reject extensible WAV from the passthrough, escape stem URLs, convert stems with dst
Four fixes from the second review, plus one bug they made visible.
isPCM16Wav tested only the bit depth, and go-audio's IsValidFile never looks at
the format tag, so a 16-bit WAVE_FORMAT_EXTENSIBLE (0xFFFE) upload was passed
through untouched where the old fold would have transcoded it. audio.cpp's WAV
reader accepts 16-bit only when the tag is 1, so such a file died with
"unsupported WAV encoding". Extensible is what many DAWs and Windows tools
write and music files are this endpoint's new headline input, so it is a
first-contact failure rather than a corner. The check now requires tag 1, with a
spec that fails against the old implementation.
Stem URLs are percent-escaped. A stem name is the model's own string and legally
contains a space, a '#', a '?' or a '%'; an unescaped '#' truncates the URL
before the request is even sent. The name field keeps the raw name.
sample_rate and response_format are applied to the stems as well as to dst.
Applying beat documenting: dst IS one of those stems, so leaving them alone
broke the "dst duplicates the selected stem" invariant the whole design rests
on, and both conversions are no-ops when unset. A stem whose conversion fails is
dropped from the header rather than advertised in the wrong shape.
Verifying that turned up why it had never been noticed: the two fields were
never bound at all. The request arrives as multipart/form-data and echo's binder
falls back to the FIELD NAME without a form tag, matching only
case-insensitively, so "SampleRate" never matched "sample_rate" and "Format"
never matched "response_format". Both were documented in the endpoint table and
silently ignored. Two form tags fix it, and with them the conversion is
observable end to end.
Docs: audio-transform.md now documents what LocalAI does to an upload before the
backend sees it, which backend gets the 16 kHz mono fold and why, params[stem],
and the X-Audio-Stems header with a worked example.
Also records the known limitation that the fold lookup is on the bare backend
name, so pinned variants (vulkan-localvqe) do not match, and points at
IsLlamaCppBackend as the suffix-tolerant precedent.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serve the TTS and SoundGeneration RPCs
TTSRequest.voice is treated as a speaker reference clip when it names an
existing regular file, which makes routing prefer VoiceCloning, and as a named
preset otherwise, in which case it travels as VoiceReference::cached_voice_id.
Both the clip and SoundGenerationRequest.src are read at the file's own rate and
channel count: upstream's own CLI and server do exactly that, every consuming
family resamples internally and mostly with a better resampler than ours, and
ace_step and stable_audio resample their input per channel, so a downmix here
would delete the stereo image they are built to consume.
The request builders live in their own unit rather than in grpc-server.cpp's
anonymous namespace so they can be tested; grpc-server.cpp has a main() and
cannot be linked into a test binary. The option keys are the whole point of
these functions, so each one was grepped against the pinned upstream and the
accounting is written down beside it. instructions maps to "instruct", which is
what upstream's own server maps the OpenAI field to and what qwen3_tts and
omnivoice read, and to "caption" for irodori_tts; the style tag is spelled
"instruct" too, because "instructions" is looked up nowhere. duration maps to
"duration_seconds", read by all three generation families, with the proto's own
name kept only as a forward-tolerant alias. Keys that no family reads say so.
Both handlers answer a capability refusal before taking the lane and before any
file read, so a model that cannot synthesise does not queue behind somebody
else's run to be told no.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): stop emitting an empty style language, and name the missing clip
StyleCondition::language was set whenever has_language() was true, with no
!empty() guard, while the language option twelve lines below had one.
core/backend/tts.go sets Language unconditionally, so has_language() is true on
every request LocalAI sends and carries "" when the caller named none. An
engaged-but-empty style language is worse than an absent one: supertonic reads
text_input->language behind its own !empty() guard and then overrides it from
style->language with no guard at all, so "" replaced its "en" default and its
tokenizer threw "invalid Supertonic language: ". Every /v1/audio/speech request
that set instructions and no language would have been an INTERNAL against a
supertonic model. A plain request never saw it, because the style condition only
exists when instructions are non-empty, which is why the chatterbox end to end
run did not catch it.
TTS also stops discarding the Route that check_can_serve already returns. A
family routed to voice cloning without a reference clip used to be refused from
inside its own prepare(), which meant an INTERNAL naming neither the RPC nor the
field to set; chatterbox advertises clon and no tts, so that was every
preset-only request to it. It is now an INVALID_ARGUMENT naming
TTSRequest.voice, answered in about 4 ms, and it cannot misfire because
has_voice_reference is what selected cloning in the first place. Reading
CapabilitySet::supports_speaker_reference to generalise this stays a follow-up.
The src read carries a written caveat rather than a family blocklist, because
ace_step's editing routes legitimately need src: setting src on a stable_audio
model corrupts the heap and aborts the process in the pinned upstream, and the
only thing keeping that off the network is that
schema.ElevenLabsSoundGenerationRequest has no field for it. Nobody reading that
Go schema would know why, so the reason is recorded where the field is read.
build_tts_shape is extracted so TTSStream cannot describe the same request
differently, and it arrived untested: two mutations of it survived until a
test_tts_shape case was added.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serve the TTSStream and AudioTranscriptionStream RPCs
TTSStream leads with a streaming WAV header carrying 0xFFFFFFFF sizes, matching
the convention backend/go/vibevoice-cpp established, so an HTTP client can start
playback before the full PCM exists. Its chunks are read from
StreamEvent::named_audio_outputs and not audio_output: supertonic, omnivoice and
voxcpm2 all put their streamed audio there and leave audio_output empty until
the very end, so reading the obvious field yields a stream with no audio in it.
The finish_stream result is the family's own merged whole rather than a tail, so
it is emitted only when nothing was streamed.
Streaming transcription sends incremental deltas and degrades to a single delta
plus the final result on families that offer no streaming ASR, which is the same
message sequence with fewer deltas. The four streaming ASR families disagree on
what partial_text means: nemotron_asr, vibevoice_asr and higgs_audio_stt report
incremental fragments while voxtral_realtime reports the whole hypothesis and
reports it twice, so the reconciliation lives in one tested unit rather than in
the handler. nemotron_asr reports only through the stream event sink, and only
from inside finalize, so the audio driver installs one and clears it again
before returning: the session is cached and a sink left holding the caller's
frame is a use after free waiting for the next stream.
begin_stream is now the only implementation of the streaming state obligation,
prepare then start_stream. Streaming sessions are cached, and what clears the
previous stream is start_stream's reset; a family override that dropped it would
break every call site with no compile error, so there is one call site.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): keep streaming deltas on UTF-8 boundaries, refuse dtypes that abort
TranscriptStreamResponse.delta is a proto3 string, whose wire format requires
valid UTF-8. voxtral_realtime reports its hypothesis as a concatenation of raw
token BYTES (tokenizer_text.cpp:171-183), so the cumulative difference between
two consecutive reports is eventually a lone continuation byte, and the C++
runtime serializes that with only a logged warning while the Go runtime refuses
to unmarshal it: the client loses the remaining deltas AND the final_result.
Measured on a trace of a non-ASCII sentence, 11 of 31 messages failed to
unmarshal and every accented character was lost. TranscriptDeltaTracker now
holds back an incomplete trailing sequence and merges it into the next fragment;
reconcile flushes it, which it always can because the final text is complete.
The same trace now unmarshals in full with zero failures.
A streaming buffer whose float count is not a whole number of frames is refused
rather than truncated. The integer division dropped the tail floats from the fed
audio and therefore from the transcript, with no diagnostic; vibevoice_asr
refuses the same thing from the other side of the call.
A supertonic GGUF whose weights are not f32 is refused at load. It reaches
ggml_concat with mismatched operand types and ggml_abort takes the whole backend
process down on the first request, so nothing downstream can report it: the
model loads, then every request kills the process. Attributed rather than
assumed, the unary TTS path aborts identically, and upstream records that
package as untested. The refusal names the orig package and says what to run
before deleting the guard.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): stop a repeated lead byte from orphaning the next delta
The first UTF-8 fix closed the cumulative half only. Rule 2 discards a fragment
the known text already starts with, and when that fragment is the LEAD BYTE of a
new character it looks exactly like a repeat of an older character beginning
with the same byte. It was discarded rather than held, its continuation bytes
then arrived alone and began the next delta, and utf8_complete_prefix_length
only ever inspected the trailing sequence, so a delta invalid at the FRONT went
out whole. Through a real Go proto.Unmarshal the review's four-character repro
gave 3 deltas, 2 unmarshal failures and a lost transcript.
Reachable from the incremental families, not only from voxtral: nemotron_asr's
decoder cuts at a byte offset and vibevoice_asr's common_prefix_size compares
bytes, so both split characters. Measured over 30,000 randomized incremental
traces, 53.28% of Japanese traces and 9.52% of French ones carried at least one
delta the Go runtime refuses.
Two changes. Rule 2 no longer judges a fragment that ends mid-character, so the
lead byte is held instead of swallowed and the character survives intact; the
cost is a few duplicated bytes in a shrinking cumulative report, which no pinned
family produces. release() additionally drops leading orphan continuation bytes,
so no delta can begin mid-character whatever the rules above it decide. Losing a
byte keeps the stream alive; emitting one ends the RPC and takes the
final_result with it.
Post-fix all 60,000 traces produce zero unmarshal failures, and the cumulative
streams plus both pure-ASCII incremental streams are byte-identical to the
previous commit, so nothing changed for the families already working.
The weight-dtype allow list moves to family_gate, where it is stdlib-only and
pinned by a test rather than only by a comment. Two comment citations corrected.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): read only an exact repeat as a repeat, not any prefix
Rule 2 discarded any partial the known text merely started with. For a cumulative
family that is a duplicate; for an incremental family it is an ordinary short
fragment that happens to coincide with the start of the transcript, and it was
dropped, silently corrupting the text. Pure ASCII, no multi-byte character
anywhere: the fragments "pure ", "ascii ", "trans", "c", "ri", "p", "t" left the
client holding "pure ascii transcrit". Over 5,000 randomized traces per
transcript, 9.50% of pure-ASCII and 29.12% of French traces ended with the client
holding something other than final_result.text, with a 200 and no diagnostic.
Both incremental families emit fragments that small routinely, since nemotron_asr
cuts at a byte offset and vibevoice_asr at a common prefix.
Narrowing rule 2 to an exact repeat drives that to zero on all six transcripts
and changes no cumulative stream at all: 30,000 randomized cumulative traces are
byte-identical to the previous commit.
What rule 2 guarded was established from upstream rather than from its own
comment. The only duplicate any pinned family produces is voxtral_realtime's,
where process_available_stream_chunks feeds each event to the sink from inside
its loop and returns the last of the batch, so that event arrives twice with
byte-equal text. A duplicate is an exact repeat, so equality still covers it. The
case given up is a cumulative report that SHRINKS, which no pinned family can
produce: voxtral decodes a token vector that is only push_back'ed and cleared by
reset(), so within a stream it can only grow.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serve the AudioTranscriptionLive RPC
The one bidirectional stream this backend serves. The client sends a
TranscriptLiveConfig, then TranscriptLiveAudio frames; the server acknowledges
with ready, emits deltas as the audio arrives, and sends final_result once the
read side closes. There is no offline fallback: live transcription has to
consume audio incrementally, so a family with no streaming ASR is refused
rather than served a batch run, which is what this RPC's Streaming-only
mode_candidates list already says.
The driver is a new sibling of run_streaming_audio, run_streaming_live, because
the audio does not exist yet: instead of slicing a buffer it pulls frames from
the caller until the read side closes. It installs the same ScopedStreamSink in
the same order, which is not optional, since nemotron_asr returns a bare event
from process_audio_chunk and reports every partial through the sink from inside
finalize(). It buffers the wire's frames up to the family's own preferred window
rather than feeding whatever size the client's audio callback produced, and it
does not call finish_stream at all when no audio arrived, because nemotron_asr
throws "finalize requires streamed audio" and an empty transcript is the
truthful answer to transcribing nothing.
Three things the handler had to get right and one it cannot:
- The audio contract. A live request carries no samples, but nemotron_asr's
streaming prepare() throws without an audio contract, and
build_preparation_request derives it from TaskRequest::audio_input, so that
field is an EMPTY buffer holding only the rate and the channel count.
- 16 kHz or a refusal. The families express their spans in their own 16 kHz
feature domain whatever the input was, and live frames cannot be resampled
on the way in the way a file can, so an 8 kHz session would return
timestamps 2x off with a 200. core/backend hardcodes 16000 anyway.
- A mid-stream Config is refused. backend.proto calls it a decoder reset, but
deltas already on the wire cannot be retracted, so a reset would leave the
final text contradicting the transcript the client assembled. Ignoring the
message would hand a client that believes it reset the decoder a transcript
that silently continues the audio it thought it discarded.
- The stale-route identity check cannot run here: TranscriptLiveRequest
carries no ModelIdentity in either arm of its oneof, so snapshot_for does
not instantiate for it. snapshot_unchecked's comment now names that as a
second legitimate class of caller and says the fix is a proto change.
eou and eob stay false. They exist for cache-aware models that emit
end-of-utterance and end-of-backchannel tokens; audio.cpp's StreamEvent has no
equivalent signal, and a client uses eou to decide the speaker yielded the turn,
so a guess inferred from silence cuts people off mid-sentence.
The lane is held for the whole stream, which is as long as the user keeps
talking: the streaming session is stateful and cached, so a concurrent run would
interleave two callers' audio and corrupt both transcripts.
Verified against nemotron_asr over a real connection with a 14 s WAV in
512-sample frames: ready first, 59 incremental deltas with no repeated prefix,
concat(deltas) equal to final_result.text, word timestamps in nanoseconds, eou
and eob false. citrinet_asr answers UNIMPLEMENTED naming the family and listing
asr/offline. A config followed by a close returns an empty final_result rather
than hanging, and a first message that is not a config is INVALID_ARGUMENT. Two
concurrent streams both return the complete transcript.
Two cleanups on lines Task 12 touched, folded in. The DtypeAllowList terminator
is now asserted at compile time: the reported out-of-bounds read did not exist,
the single entry does terminate, but the loops have no other bound and any edit
that widened an entry would walk off the end. And the dtype guard now
short-circuits on "is there a table entry" through a new predicate rather than
on the emptiness of the description string, which would have skipped the check
on an entry with an empty allow list, i.e. on precisely the entry that refuses
every dtype.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): bound the lane a live stream can hold
AudioTranscriptionLive holds the model's inference lane for the whole stream,
which is correct (the streaming session is stateful and a concurrent run would
interleave two callers' audio) and newly dangerous. Every other RPC holds the
lane across compute, or across a write to a slow reader, and both of those
terminate on their own. A live stream instead blocks in a client-driven read,
and a peer that goes silent WITHOUT closing the stream never terminates
anything: the lane stays taken and every other request against that model queues
behind a client that stopped speaking.
live_watchdog is a one-shot idle timer that ends the stream when no frame has
arrived inside a window. It is standard library only, so it is unit tested
without an engine. gRPC's synchronous Read has no timeout and cannot be given
one, so the only way to unblock it is ServerContext::TryCancel, which decides
the wire status itself: the client sees CANCELLED rather than the
DEADLINE_EXCEEDED the handler returns, the reason is logged, and the lane coming
back is the point. When it fires the read loop throws rather than reporting
end-of-input, so the driver does not go on to finalize a decode nobody is
waiting for.
It is armed only after the lane is taken and disarmed as soon as the read side
closes, and both ends matter. Arming earlier would cover acquire(), which
legitimately blocks while another live stream runs, so a queued caller would be
cancelled for waiting its turn. Disarming later would cover our own decode,
where a window overrun is not a peer going quiet and cancelling would throw away
the transcript the client is waiting for.
The window is the new live_idle_timeout_ms option, 30 s by default, 0 meaning no
limit. core/http/endpoints/openai/realtime.go drives a 300 ms ticker and feeds
every tick that produced new audio while a turn is open, so 30 s of silence is a
hundred ticks that delivered nothing. It is also longer than any pause a speaker
takes mid-utterance, which is the case that must never be cut off, and
backend.proto lets one stream span many utterances, so a client that pauses
longer between them raises the option rather than discovering it.
Two smaller corrections in the same handler:
- check_can_serve now runs BEFORE the sample rate check.
pkg/grpc/grpcerrors/errors.go degrades to the file path on UNIMPLEMENTED and
on nothing else, so a live-incapable model asked at a wrong rate was
answering INVALID_ARGUMENT and costing the caller its fallback.
- a negative sample rate is refused instead of silently becoming 16000. Zero
still means 16000, which is what the proto documents; -1 is malformed rather
than absent and gets the same refusal every other bad rate gets.
And one thing recorded rather than changed, at the handler: "live" here means
incremental INPUT, not low latency, and with the pinned families it does not yet
mean incremental OUTPUT either. nemotron_asr's process_audio_chunk only appends
to its buffer, so its whole decode and every delta happen inside finalize(),
after the client closes its send side. The policy-window buffering is inert for
that family and matters only for vibevoice_asr and higgs_audio_stt.
Verified on the wire with live_idle_timeout_ms:3000. A silent client acked at
371 ms and was cancelled at 3.371 s; a second live stream opened one second
later received its ack 2.37 s in, i.e. at the instant the first was cancelled,
and then transcribed successfully on the same cached session. Without the
watchdog it would still be waiting. Re-ran the live transcription (ready first,
59 incremental deltas, concat equal to the final text, word timestamps in
nanoseconds, eou and eob false), the citrinet refusal at both a right and a
wrong rate (UNIMPLEMENTED either way now), and Task 12's AudioTranscriptionStream
on nemotron_asr, which is unchanged.
Mutation testing the watchdog found a weakness in its own test: the destructor
test slept past the window inside the watched scope, so a destructor that
DETACHED the thread instead of joining it passed unnoticed. The test now uses a
window longer than the scope, which kills that mutant, and says why.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): refuse the unsupported RPCs with a reason
AudioEncode, AudioDecode, AudioTransformStream, AudioToAudioStream and
VoiceEmbed have no counterpart in audio.cpp's VoiceTaskKind. Each now returns
UNIMPLEMENTED naming the loaded family, what that family does support, and the
upstream limitation, instead of the generated base class's bare status. The
reasons live in a table in capability_routing.cpp so they are data rather than
literals copied into five handlers, and so a test can assert every one of them.
The five claims this was planned against were re-read at the pinned upstream
e800d435d130dc776baf6f3e6129bb62b1495c89, and one did not hold. "audio.cpp
streams tts and asr only" is false: silero_vad advertises vad with
RunMode::Streaming. The refusal stands on the narrower claim that survives, that
no family advertises streaming for any task AudioTransform routes to, and a test
asserts the refuted wording does not come back.
VoiceEmbed is the one refusal whose request carries a ModelIdentity, so it runs
the #10952 check before answering: a stale route must get NOT_FOUND and the
router's sentinel, not "audio.cpp cannot embed speakers" about a model that is
not loaded here. It cannot use snapshot_for, whose no-model branch would tell
the caller to load a model when no model can help, so it takes the reference
through snapshot_unchecked and checks identity itself. That function's comment
now names three classes of caller instead of two.
The two bidirectional surfaces refuse without reading their stream, verified
with a client that writes a config and eight frames first and gets the status
rather than hanging.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): correct the vevo2 clause, and assert the absences
Review found a false clause in the AudioToAudioStream refusal. It said s2s is
"offline voice conversion ... which converts one clip into another speaker's
voice", which is true of miocodec and false of vevo2: vevo2's s2s route is
`editing` and only `editing` (default_route_for_task and route_matches_task in
src/models/vevo2/session.cpp), documented as "Edit source speech into new target
text while using the target voice" and requiring --target-text, so it rewrites
what was said. vevo2's voice conversion is its separate vc task. It now reads
"offline clip-to-clip processing against a target voice, declared only by
miocodec (voice conversion) and vevo2 (speech editing)", and a test asserts the
miscast cannot come back. The conclusion is unchanged: neither family converses.
That defect was undetectable on the wire, since vevo2 does not load here, which
is the argument for upstream_absence_ctest.cpp. It links engine_runtime purely
to interrogate make_default_registry() and asserts the five premises the refusal
reasons rest on: no codec task kind, no family advertising spk, no streaming for
sep/vc/svc/s2s, miocodec advertising exactly vc and s2s, and s2s advertised by
exactly miocodec and vevo2. The last two are exact sets, so an addition fails
here rather than leaving a message stale. A positive control proves the registry
is populated and the query works before any absence is believed, and every
assertion has a reproduced negative control. This turns an AUDIO_CPP_VERSION
bump from "remember to re-read five prose paragraphs" into a test failure.
unsupported_surface now switches over UnsupportedRpc with no default label, so
-Wswitch reports a sixth enumerator added without a row at build time; the
runtime bounds guard it replaces is deleted.
The AudioTransformStream reason had a true premise and an overreaching
conclusion: an offline sep family could be buffered into a stream, as other
LocalAI backends do. It now says this backend declines to offer a buffered
offline call in disguise, rather than implying impossibility.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): make the missing-switch-case diagnostic fatal
unsupported_surface() switches UnsupportedRpc onto the table row that explains
it, with no default label, so -Wswitch reports an enumerator nobody handled. As
a warning that is not enough: adding a sixth enumerator and building the shipping
target gives exit 0, a binary and one warning, and the trailing
`return surfaces[0];` then answers the new RPC with AudioEncode's codec reason.
That is a confident, specific and false statement about audio.cpp on the wire, on
the one code path whose entire job is to be truthful about what this backend
cannot do, and it is worse than the runtime fallback it replaced, which at least
named itself as a bug in this file.
capability_routing.cpp therefore joins loaded_model.cpp on the existing
-Werror=switch pin, whose comment already made this argument for the engine enum.
The comment now covers both files. The pin stays per-file rather than
project-wide because upstream's own ace_step/vae_decoder.cpp has unhandled
-Wswitch cases of its own.
Verified: a sixth enumerator now fails `make grpc-server` with exit 2 and no
binary; appending a 14th VoiceTaskKind upstream still fails loaded_model.cpp, so
the two pins fire independently; both reverted clean.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): package the backend image
Bundles the dependency closure for the from-scratch image, the dlopened ggml
CPU-variant shared objects that ldd cannot see, and upstream's bundled
silero_vad and marblenet_vad assets so VAD works with no download.
The bundled loader sits in the package ROOT rather than at lib/ld.so. run.sh
execs it, which makes /proc/self/exe name the loader, and this backend has two
consumers of that path: ggml discovers the libggml-cpu-*.so by listing
dirname(/proc/self/exe), and resolve_model_path expands bundled:<name> under the
same directory. Rooting the loader makes the binary, the ggml objects and
assets/ share the one directory all three resolution mechanisms agree on.
llama-cpp's lib/ld.so layout would need assets/ moved into lib/ as well.
The image builds against apt gRPC and protobuf, like Dockerfile.ds4 and unlike
Dockerfile.privacy-filter. The from-source gRPC that install-base-deps.sh and
the base-grpc-* images supply vendors protobuf 26, which pulls abseil into
message_lite.h; with SPM_PROTOBUF_PROVIDER=package that collides with
sentencepiece's vendored mini-abseil and every absl::internal reference becomes
ambiguous. Noble's protobuf 3.21.12 predates the abseil dependency and is the
pair every earlier verification of this backend ran against.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): exempt the driver libraries from the packaging gate
package.sh already left libcuda.so* and libnvidia-* to the host when copying,
because the driver has to match the kernel module on whatever host runs the
image, but the validation gate had no matching exemption. With BUILD_TYPE=cublas
ggml is static and links CUDA::cuda_driver, so grpc-server carries DT_NEEDED
libcuda.so.1 and the gate would have rejected the very absence the copy loop
created, failing every cublas build in CI. One regex now feeds both.
Building a control for that found a second defect: ld.so --list refuses to trace
an object with an unresolvable dependency at all, exiting 127 without emitting a
per-library line, so the "=> not found" rule was dead code and no exemption could
have applied to it. The gate now traces with LD_TRACE_LOADED_OBJECTS and
LD_LIBRARY_PATH, which reports the missing name and exits 0, and which is also
what run.sh does at run time.
Adds a layout assertion so a future move of the loader into lib/ fails the build
instead of shipping a package that resolves bundled: models into lib/assets and
finds no ggml CPU backend, and records for Task 16 that the Darwin script must
not be a straight copy of privacy-filter-darwin.sh, which never calls package.sh
and would silently drop assets/.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): register the backend with CI and the gallery
Adds the five Linux matrix entries (cpu amd64/arm64 sharing a tag-suffix so the
manifest merge fires, cuda 12, cuda 13, vulkan), the path-filter case that keeps
later PRs touching backend/cpp/audio-cpp/ from getting zero CI jobs, the
bump-bot entry pointing at the AUDIO_CPP_VERSION pin in the backend Makefile,
the gallery meta plus its -development variant and the image entries for every
variant, and the Makefile docker-build wiring.
The matrix entries carry base-image only, with no builder-base-image, unlike
the llama-cpp and privacy-filter blocks they sit next to. The prebuilt
quay.io/go-skynet/ci-cache:base-grpc-* images ship a from-source gRPC whose
protobuf v26 depends on abseil, and this backend's sentencepiece is built with
SPM_PROTOBUF_PROVIDER=package, so it sees real abseil's absl::lts_20240116::
internal alongside its own vendored plain absl::internal and every
absl::internal:: reference becomes ambiguous. Building against base-grpc-amd64
fails at sentencepiece-static.dir/error.cc.o with "reference to 'internal' is
ambiguous". Dockerfile.audio-cpp installs apt's gRPC/protobuf 3.21.12 itself,
which is also the pair every unit and end-to-end run of this backend has been
verified against, and the CUDA toolkit therefore has to come from base-image.
No Darwin matrix entry and no metal gallery entries: the Metal build needs
scripts/build/audio-cpp-darwin.sh, a backends/audio-cpp-darwin make target and
a routing step in backend_build_darwin.yml, none of which exist yet, so an
entry added now would be routed to build-darwin-go-backend and look for
backend/go/audio-cpp/. The inferBackendPathDarwin case and the
DARWIN_BESPOKE_BUILDERS membership are in place, inert, so that adding the
entry later is a one-line change that cannot be claimed by the generic Go path.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): pin the CUDA architectures, drop the vulkan variant
Upstream sets CUDA_ARCHITECTURES to `native` on the engine_runtime target
whenever CMAKE_CUDA_ARCHITECTURES is unset at root scope, and docs/build/
linux.md says so outright. ggml's own default does not rescue it: it
list(APPEND)s in the ggml subdirectory scope, which never reaches the root
scope where the engine_runtime property is decided. No CI runner has a GPU for
`native` to enumerate, so both cublas entries would have gone red on the very
commit that first turns a CUDA build on.
Pin the list in backend/cpp/audio-cpp/Makefile, selected by CUDA_MAJOR_VERSION,
which Dockerfile.audio-cpp now forwards from the CI build-arg it was previously
discarding. The values are copied from ggml's own version guards rather than
invented, so engine_runtime and ggml compile for the same set: CUDA 12 keeps the
Maxwell/Pascal/Volta virtual archs and stops at 120a-real, CUDA 13 drops them
and adds 121a-real. The `a` suffix is used rather than `f` because the latter
needs CMake 3.31.8 and Ubuntu Noble ships 3.28.3. Verified by driving CMake
3.28.3's own CUDA architecture validator over both lists, with 120f-virtual as
the rejected control.
Drop the vulkan matrix entry, its two gallery entries, the vulkan capability
key on both metas and the Vulkan tag. Every other vulkan backend gets its Mesa
ICD drivers from .docker/install-base-deps.sh, which package-gpu-libs.sh then
bundles; Dockerfile.audio-cpp calls neither and installs only libvulkan-dev and
glslc, so the image would ship a Vulkan loader that finds no GPU. No CI job runs
a vulkan image against real hardware, so that would have passed green and failed
in users' hands. BUILD_TYPE=vulkan stays supported for local builds.
Also note on the cublas entries that cuda-major-version now selects the
architecture list and that cuda-minor-version and the base-image tag encode the
same toolkit, and correct the stale entry counts on matrixEntryKey.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): build for Darwin Metal
Bespoke C++ Darwin path like ds4 and privacy-filter: an includeDarwin matrix
entry, a backends/audio-cpp-darwin make target, a gated workflow step, and the
metal image entries plus metal/metal-darwin-arm64 capability keys in the
backend gallery.
The build script deliberately does NOT reassemble the package the way
privacy-filter-darwin.sh does. It runs the backend's own `make package` and
copies the result, so the Darwin package keeps the root-level layout the Linux
one has: grpc-server, run.sh, the ggml objects and assets/ in one directory,
with lib/ for the dylib closure. Hand-assembling would drop assets/, and
assets/ is what makes the bundled: model paths resolve with nothing downloaded.
The dylib walk is a full transitive closure rather than the single level ds4
and llama-cpp do, because Homebrew's grpc++ pulls libgrpc, abseil, upb, cares
and OpenSSL that grpc-server does not link itself, and a level-1 walk ships a
package that only works on a machine that already has Homebrew grpc.
Two fixes folded in, both in the backend Makefile:
- an EMPTY CUDA_MAJOR_VERSION fell through to the CUDA 12 architecture list,
which contains 120a-real and so needs nvcc >= 12.8. A local
BUILD_TYPE=cublas build on a 12.0-12.7 host failed to compile where
upstream's documented default (native) worked. EMPTY now maps to native,
12 and 13 keep their lists, and any other non-empty value is an error on
cublas builds. CI always passes a major, so CI is unaffected.
- the Darwin branch now points CMake at Homebrew's keg-only libomp. AppleClang
ships no OpenMP runtime and nothing is symlinked into /opt/homebrew, so
FindOpenMP finds neither the library nor the header, and audio.cpp calls
find_package(OpenMP REQUIRED) whenever ENGINE_ENABLE_OPENMP is on. Without
the hint the macOS build would have died at configure time. If the keg is
absent the build disables OpenMP instead of failing.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): make the Darwin fallbacks loud and the rpath walk complete
Review follow-up on the Darwin Metal build.
The OpenMP fallback was silent. If brew --prefix libomp ever comes back empty,
CI produced a green Metal package with 108 #pragma omp directives across ~30
files compiled out, and clang says nothing about an ignored omp pragma without
-Wsource-uses-openmp, so the only trace was one absent flag inside a set -x
cmake line. That regression would have been blamed on Metal. It now warns.
The @rpath arm of the dylib walk had no live candidate when it was written, on
the reasoning that a Metal build links ggml statically. The OpenMP fix in the
same commit made libomp.dylib one, and whether Homebrew records it as an
absolute opt path or as @rpath/libomp.dylib is not observable from Linux. The
walk now expands @rpath, @loader_path and @executable_path against the object's
own LC_RPATH entries, and only fails when nothing on disk answers, printing the
rpath list with the error so a failure on a machine nobody can attach to
explains itself.
Also: ADDITIONAL_LIBS now go through the closure rather than a bare cp, so they
are deduplicated and their own dependencies bundled; build/darwin/lib is
created explicitly instead of relying on package.sh pre-creating it; the libomp
probe uses nested ifneq rather than $(and ...), which needs GNU make 3.81 and
would otherwise expand empty and take the OFF branch on an older make; and
-DOpenMP_ROOT is quoted like its CUDA sibling.
Verified with a Linux harness that runs the script verbatim against a stubbed
otool: a level-2 transitive dep, an @rpath dep reachable only through LC_RPATH,
and an ADDITIONAL_LIBS dep are all bundled, a dependency cycle terminates,
system libraries are skipped, the packaged tree has assets/ at the root beside
grpc-server with the dylibs in lib/, and both failure paths exit non-zero.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): make bundled: reachable from a model YAML
resolve_model_path() tested the bundled: prefix on `candidate`, which prefers
ModelFile and falls back to Model. LocalAI fills ModelFile by joining ModelPath
onto the configured model string (pkg/model/loader.go, LoadModelWithFile), and
only sets it from a managed artifact otherwise, so a model YAML saying
`model: bundled:silero_vad` arrives as ModelFile "/models/bundled:silero_vad"
and Model "bundled:silero_vad". The prefix therefore never matched through the
normal load path: it matched only for a hand-written LoadModel call that left
ModelFile empty, which is exactly how task 15 verified it, and every model YAML
using the form failed with "model path does not exist:
/models/bundled:silero_vad".
Both fields are now checked, Model first, so the zero-download VAD path the
package ships assets for is reachable the way it is documented. A caller that
puts the form in ModelFile still works, so task 15's verification stands.
Compiled clean; the runtime check could not run on this host, whose system
libprotobuf/libre2 have gone missing (the pre-existing grpc-server binary no
longer resolves its libraries either), so it wants a container run.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): advertise the backend and document its options
Registers audio-cpp as preference-only in /backends/known: the family lives in
GGUF metadata that an importer cannot read from a remote repo, and one repo
hosts thirty families, so there is no honest auto-detect signal. Modality is a
single string and the import form chips on a fixed key set, so it registers as
tts with the other modalities named in the description rather than under an
invented key the UI would bucket as "other".
Adds a features page covering the option namespacing, the routing table per
endpoint, the RPCs this backend declines and why, the bundled VAD path, the
separation stem behaviour, and the family gotchas (supertonic needs the orig
package; chatterbox advertises cloning and no plain tts; nemotron_asr defers
its whole decode to finalize so live transcription emits nothing until the
client half-closes, unlike higgs_audio_stt and voxtral_realtime). Every option
name and family capability in it was read off the pinned upstream checkout.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): test resolve_model_path, and correct the family names
The bundled: fix in 842443cd7 shipped without a test, which is how the bug got
there: task 15 verified the form with a hand-written LoadModel that left
ModelFile empty, and that is the one shape the server never produces. Four cases
in streaming_driver_ctest, which already links loaded_model.cpp, pin the
PRODUCTION shapes instead. The first fails against the pre-fix source (returns
the joined /models/bundled:silero_vad); the other three are the branches the
bundled: lookup now runs in front of and must fall through for.
Three family names in the docs were the source directory rather than the
registered family, on pages whose whole argument is that these names cannot be
guessed: demucs is htdemucs (demucs/loader.cpp:22), roformer is
mel_band_roformer (roformer/assets.h:15), and moss is TWO families,
moss_tts_local and moss_tts_nano. The hyphenated ASR names are underscored to
match, here and in the compatibility table.
The supertonic dtype note claimed more than the evidence carries. The f16 abort
is a local observation, identical through TTS and TTSStream; upstream's
docs/gguf.md leaves the 16-bit column untested and records q8_0 as "No
(unsupported weight dtype)", which says unusable rather than fatal. Both are
still refused, because the allow list is what the family can run. Corrected in
family_gate.h, family_gate.cpp and the docs together, since the docs inherited
the wording from the code.
The importers tripwire says in the file that it is a tripwire: it exercises no
audio-cpp behaviour, and the registration assertion lives in backend_test.go.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* gallery: add audio.cpp models covering every served RPC
One representative model per RPC group of the audio-cpp backend, plus the two
bundled VAD models, which need no download at all because the assets ship
inside the backend package.
Every hash was computed with sha256sum on the downloaded file. Quantizations
come from upstream's tested-status table in docs/gguf.md rather than a default
of q8_0: supertonic ships the orig package (its q8_0 is recorded as an
unsupported weight dtype and its f16 aborts in ggml_concat), and nemotron_asr
and htdemucs ship f16 because their q8_0 builds are recorded with drift while
16-bit is a clean pass.
Diarization and separation use the diarization and audio_transform usecases,
not transcript: /v1/audio/diarization and /audio/transform filter the default
model on FLAG_DIARIZATION and FLAG_AUDIO_TRANSFORM respectively, so a
transcript flag would have hidden both models from their own endpoints. The
forced aligner sets parameters.language, which the transcription endpoint uses
as the fallback when no language form field is sent, because the family
requires both a transcript and a language.
All ten entries were run twice: once against the raw gRPC server, and once
installed with local-ai models install and called through the HTTP endpoint.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* gallery: correct the audio.cpp entries' licenses
Swept all ten entries against the real upstream named in audio.cpp's
tools/model_manager.py rather than against the audio.cpp repo's own license.
Three were wrong:
supertonic apache-2.0 -> openrail weights come from
mlx-community/supertonic-3-mlx, and
both it and Supertone/supertonic are
openrail
citrinet apache-2.0 -> other pulled from NGC
nvidia/nemo/stt_en_citrinet_256,
governed by the NGC Terms of Use
sortformer other -> cc-by-nc-4.0 nvidia/diar_sortformer_4spk-v1 is
CC BY-NC 4.0, and the gallery already
uses that exact string, so there is no
reason to obscure a non-commercial bar
The license field is one word, so citrinet and sortformer also gained a
sentence saying why they are restricted. The other seven were confirmed
correct against their sources.
Also drops an unverified claim from the nemotron description. It said the
model drives the realtime transcription session; that endpoint actually calls
TranscribeStream, and the live RPC reaches LocalAI only through
realtime_semantic_vad.go. Neither path was exercised here, so the description
now states only the two calls that were.
MarbleNet gains the NeMo upstream under urls: for parity with silero.
No sha256, quantization, usecase or model choice changed.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(audio-transform): bound sample_rate, keep same-named uploads apart
Four defects the whole-branch review found on the Go side, plus two comment
corrections.
sample_rate is a disk-exhaustion hazard. The branch added the `form:` tag that
makes the field bind for the first time, so the resample path went from dead to
live, and utils.AudioResample interpolates the int straight into ffmpeg's -ar
with no bound. Measured with ffmpeg 7: -ar 999999999 on a 0.01 s clip writes
20 MB and exits 0, which scales linearly to the reported 3.9 GB for one second,
into a GeneratedContentDir nothing sweeps, and convertStems repeats it once per
separation stem. Clamped to 8000..192000 in the handler, before the temp dir and
before the model is touched, and rejected with a 400 outside it.
The low end was reported as "a 0-byte file". It is not: -ar 1 writes a 78-byte
header with no audio behind it, whose declared data size still claims 70 bytes,
so go-audio parses it as a 35 SECOND file and a size check does not see it. The
guard therefore compares the declared data chunk against the bytes actually on
disk, and AudioResample now fails rather than returning a WAV carrying nothing.
Both parts of a transform request land in one temp dir, and the raw copy was
named only after the client's basename, so `-F audio=@mic/clip.wav
-F reference=@loopback/clip.wav` wrote "raw-clip.wav" twice. Since
AudioToWavPreservingShape hardlinks an already-PCM16 WAV rather than copying it,
the reference part's os.Create truncated the inode audio.wav pointed at: mic and
reference came out identical, which makes an echo canceller null everything and
return near-silence with a 200. The raw copy now carries the form field name.
audio-cpp had no BackendCapabilities entry, so VoiceCloningForModel returned nil
before it ever consulted the model's tts.voice_cloning override and every
`voice: "profile:<id>"` request was refused with a 400, on a backend that ships
audio-cpp-chatterbox whose family serves cloning and not plain TTS. Registered
with its RPCs, usecases and the reference-audio contract, and deliberately
without the 16 kHz mono fold, which its separation families cannot survive.
GetBackendCapability was exact-match only, so every pinned gallery variant read
as an unknown backend: vulkan-localvqe lost the 16 kHz mono fold that used to be
unconditional and started failing inside LocalVQE, and the usecase gate does not
stand in for it because BuildFilteredFirstAvailableDefaultModel returns early
once the client names a model. Lookup now falls back to the meta name by
stripping the gallery's hardware prefix and release-channel suffix, exact match
first so nothing can be shadowed. Same class as #10945.
Also corrected: the AudioTransformRequest comment claimed echo's binder falls
back to the field name, which it does not in either direction (bindData binds
ONLY tagged fields and `continue`s otherwise; `model` arrives from
setModelNameFromRequest's c.FormValue). And the stable_audio `src` heap
corruption caveat now lives on ElevenLabsSoundGenerationRequest, where the Go
developer who would add the field can see it, instead of only in C++.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(audio-cpp): refuse a task pin the RPC cannot serve, and stop empty frames holding the lane
The model's `task:` option is copied into the request shape by all nine
handlers, which is correct, but resolve_route then replaced the RPC's candidate
list with the pin WHOLESALE and never asked whether the pin was something that
RPC routes to. One pin therefore bled across all nine surfaces, and because the
family still supported the pinned task the result was a wrong 200 rather than an
error. Reproduced live: nemotron with task:asr made Vad return 200 with zero
segments after a full ASR decode, so 14 seconds of speech was reported as
silence, and Diarize did the same; silero_vad with task:vad made
AudioTranscription return 200 with empty text and four segments whose spans were
VAD segments, which combined with response_format in {text,srt,vtt,lrc} building
the body solely from Segments[].Text yields a well formed SRT of four timed
EMPTY cues. It also contradicted the documented contract, that a family which
cannot serve a request is refused rather than rerouted.
A pin is now checked against the RPC's admissible task set before it is adopted,
and the refusal names both the pin and the RPC. The set is derived from
task_candidates with every shape flag set rather than restated, so a task added
to an RPC's candidates cannot become inadmissible by omission. Every legitimate
pin survives, and the test asserts all fifteen of them alongside the eight
crossings that must not.
The live watchdog was defeated by empty frames. idle.touch() ran on ANY message,
before the has_audio and pcm.empty() filters, so a peer writing unset-oneof or
zero-length frames faster than the window held the lane indefinitely while
feeding the decoder nothing. There is one lane per model and one model per
process, so that is a single client denying the whole backend, which is what the
watchdog exists to prevent, and the thrown text already said "no audio frame
arrived". The touch moved below the filters, which are now a named predicate so
the distinction is testable rather than a call order nobody can see.
Three comments corrected against measurement rather than reasoning:
- CMakeLists claimed zero google::protobuf:: definitions remain in the
executable. nm -C --defined-only reports 2515, and that is expected: they are
generated code, sentencepiece::ModelProto's own _InternalParse among them. The
claim that holds, and the one the ABI fix is actually about, is that no
vendored protobuf RUNTIME is linked and ParseContext::ParseMessage is
UNDEFINED in the executable, resolving to libprotobuf.so.
- refuse_cloning_without_a_clip's "cannot misfire" paragraph had its reasoning
backwards. Routing picks VoiceCloning as the FALLBACK when there is no clip,
which is the case being caught; chatterbox, which ships in the gallery,
advertises clon and no tts at all, so every voice-less request lands there.
- audio_units read "2.1 min at 96 kHz" for index 11289602, which is 1.96 min.
2.1 min is 96 kHz's OWN first failure at 12288002. Both were remeasured and
the note is now a per-rate table.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* build(audio-cpp): exclude the upstream checkout from the C++ gate, harden the darwin walk
run-unit-tests.sh pruned */llama.cpp/* but not */audio.cpp/*. It is safe today
only by luck: upstream's 44 tests all put "test" at the FRONT of the filename
(17 test-*.cpp, 27 test_*.cpp, zero *_test.cpp), so the glob misses every one of
them, and nothing enforces that. This gate runs on every PR for every backend
and compiles each match as a standalone translation unit with nothing but
nlohmann/json on the include path, so the day upstream adds or renames one test
the gate goes red repo-wide on an Apache-2.0 file nobody here wrote.
audio-cpp-darwin.sh now logs the raw otool -L output and the parsed LC_RPATH
list unconditionally, before the walk. Both awk filters in that script assume a
column layout nobody working on this can observe, since it runs only on the CI
Mac, and a green first Darwin run proves nothing about the assumption: an awk
that silently matched nothing yields an empty dependency list, which reads
exactly like "no non-system dependencies" and packages happily. Both filters
otherwise feed process substitutions, so their input never reached the log.
It also lists every symlink in the package and fails on one that cannot resolve
inside the image. A dangling link does not fail anything else here, because
every assertion tests with -e, which follows links; it fails at dlopen on a
user's Mac. Links are NOT banned outright, which the review suggested but which
would break the libggml.dylib -> libggml.0.dylib chain the `cp -a` above exists
to preserve. What is banned is a link that resolves on the build host and will
not resolve in the image: a broken one, or an absolute one pointing outside the
package.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* style(audio-cpp): drop em dashes from the audio-cpp capability entry
Follow-up to a84b3c4b9, no behaviour change.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(config): key the voice-cloning model rule on the resolved backend
Making GetBackendCapability strip the gallery hardware prefix and release
channel fixed pinned variants of /audio/transform, but VoiceCloningForModel
kept keying its per-backend switch on the caller's spelling. A pinned name
therefore resolved the capability by stripping and then missed every case in
the switch, falling through to the permissive default: cuda12-vibevoice-cpp
advertised voice cloning for the realtime 0.5B model, metal-coqui for
tacotron2, cuda12-crispasr for a pure ASR model, cpu-qwen3-tts-cpp for
CustomVoice. Each of those is a model that cannot clone, so /v1/audio/speech
accepted a profile: voice it had to fail on inside the backend rather than
rejecting it with a 400, and the UI advertised the capability too.
resolveBackendCapability now returns the key the entry was found under, and
callers that branch on backend identity use that key instead of the name they
were handed. The exact-match-first order is unchanged, so a backend genuinely
registered under a variant-looking name still keys on its own name.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* gallery(audio-cpp): declare audio_transform on the chatterbox entry
Chatterbox advertises VoiceCloning AND VoiceConversion
(src/models/chatterbox), and the entry's own description already said so, but
known_usecases listed only tts. /audio/transform selects its default model by
FLAG_AUDIO_TRANSFORM, so voice conversion was reachable only by naming the
model explicitly and was invisible to every usecase-driven surface. It is the
one audio.cpp task with a shipped gallery model and no way to find it.
Verified against the real model rather than inferred from the capability list:
AudioTransform with chatterbox-q8_0, speech as audio_path and a speaker clip
as reference_path, returns a 5.08 s 24 kHz mono WAV at -25.5 dB mean and zero
stems, which is the single-output shape voice conversion should have.
The description now says which endpoint reaches that half and warns that
installing this next to a source-separation model gives /audio/transform two
candidates, so the model should be named rather than defaulted.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* gallery(audio-cpp): add voice-design and singing-voice-conversion entries
Two of the three audio.cpp task kinds that had no gallery model now have one.
Both were driven end to end against the real weights through the backend
before being written, not inferred from the capability tables.
audio-cpp-irodori-voicedesign covers vdes. TTS carrying `instructions` routes
to the vdes task, so the voice is described in words rather than supplied as a
clip. Verified: "a calm elderly woman speaking slowly with a warm, gentle
tone" over an 8.76 s 48 kHz mono render at -16.8 dB mean, and a closed-loop
citrinet pass recovers the sentence with the accent drift expected from a
Japanese-first model read by an English recogniser.
audio-cpp-seedvc-singing covers svc, and pins task:svc because nothing else
can reach it. seed_vc advertises svc and ordinary voice conversion, no request
signal means "this input is singing", and auto-routing resolves the tie to
voice conversion every time. Verified with the pin: 5.04 s 44.1 kHz output
whose closed-loop citrinet transcription is exact.
s2s deliberately has no entry, and the reason is not effort. miocodec is the
only upstream family whose speech-to-speech route needs no text, and it
returned audio with correct duration and level but no recoverable speech in
four independent attempts: the stale build, v2 q8_0, v2 orig (the variant
upstream records as a clean Pass), both tasks, and matched 44.1 kHz inputs on
both sides. vevo2's route refuses with "Vevo2 text/prosody route requires
text_input or target_text", and session.cpp:897 fills target_text only from
request.text_input, which AudioTransform has no field to carry. The same
vevo2 weights convert voice correctly through the default route with an exact
ASR round trip, so the model and the plumbing are both healthy; it is the s2s
route specifically that this RPC cannot express.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* backend(audio-cpp): carry transform text through params, add the s2s entry
AudioTransform is audio-in / audio-out and its proto message has no text
field, but not every task it routes to is audio-only. vevo2's speech-to-speech
route is a text and prosody route: session.cpp:897 fills refs.target_text from
request.text_input and nowhere else, and the run refuses without one with
"Vevo2 text/prosody route requires text_input or target_text". The params map
is the only channel this RPC has that reaches the engine, so the text travels
through it and apply_transform_text_input unpacks it after the params have
been copied into task.options.
Before this, s2s was not awkward to reach through /audio/transform, it was
unreachable, and it was the last audio.cpp task kind with a real model and no
way to get to it.
target_text is canonical and text is its alias, the order vevo2's own option
table declares them in, so a request setting both gets the canonical one
rather than whichever the map happened to store first. An empty value falls
through to the next candidate instead of ending the search. language rides
along only when a text was found: on its own it conditions nothing, and
manufacturing a text_input for it would route a plain separation request
carrying a language hint through the text path. The keys are left in
task.options rather than erased, because vevo2's loader advertises target_text
as a request option and a family reading it there keeps working.
Nine tests, all confirmed failing on behaviour against a stub that returned
false before the implementation was written. Verified end to end afterwards:
vevo2-q8_0 with task:s2s and params[text] returns a 5.12 s 24 kHz output whose
closed-loop citrinet transcription is exact, and htdemucs separation with no
text param still returns its four stems, with and without params[stem].
audio-cpp-vevo2-speech-to-speech ships that route. Every audio.cpp task kind
with a loadable family now has a gallery entry; spk remains the only gap and
has no family upstream at all.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* docs(audio-cpp): document params[text] and the pinned transform tasks
The text channel and the two task pins are both invisible from the endpoint
contract alone: nothing in the AudioTransform form tells a reader that a
speech-to-speech model needs the line it is resynthesising, and nothing says
that asking for singing voice conversion without task:svc silently gets plain
voice conversion instead. Both are the kind of thing a user only discovers
from a refusal or, worse, from output that looks right and is not.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(utils): annotate the two G304 sites this branch introduced
gosec flags os.Open on a variable path, and both new call sites in ffmpeg.go
are its alerts on this PR. Neither is reachable by an outside caller: isPCM16Wav
opens the exact path it is about to hand ffmpeg as input, which in the upload
path is a server-created temp file named from path.Base of the client name so
no traversal survives, and wavAudioBytes opens AudioResample's own dst, a name
this package derives from src and has just had ffmpeg write.
Annotated in the repo's existing style rather than restructured, with the
reason spelled out, because a bare suppression is worth nothing to the next
reader. The three other G304 sites in this file, in passthroughWAV and
isTargetWav, predate the branch and are left untouched.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(audio-cpp): build arm64 with gcc-14 for the armv9.2 SME variants
The arm64 CPU image failed to build:
cc1: error: invalid feature modifier 'sme' in
'-march=armv9.2-a+dotprod+fp16+sve+i8mm+sve2+sme'
ggml's CPU_ALL_VARIANTS table includes armv9.2 variants compiled with +sme, and
Ubuntu Noble's default gcc-13 rejects that feature modifier. Every entry in the
table has to compile even though a host only ever dlopens the one its own CPU
supports, so a single unbuildable variant fails the whole image. gcc-14 accepts
it, which is exactly the fix llama-cpp already carries in
.docker/llama-cpp-compile.sh; this is the same problem reached by a different
Dockerfile.
Applied to every arm64 BUILD_TYPE rather than to the CPU one alone, and that
differs from llama-cpp on purpose. llama-cpp needs it only for its pure-CPU
image because its GPU builds run llama-cpp-fallback, which builds no variant
table. This backend's Makefile turns ENGINE_ENABLE_CPU_ALL_VARIANTS on for
every non-Darwin build, GPU included, so an arm64 GPU image would hit the
identical error. The matrix has no arm64 GPU entry today, which is precisely
why gating on an empty BUILD_TYPE would leave the trap armed for whoever adds
the first one.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(model): deterministic, file-type-filtered backend auto-detect (#9287)
When a model config declares no explicit `backend:`, Load() fell into a
trial loop built by ranging the external-backends Go map (random order)
with no filtering, returning the first backend whose gRPC LoadModel
succeeded. An unrelated installed backend - e.g. the "opus" audio codec -
could therefore win a GGUF/LLM model load, so a model that should run on
llama.cpp wrongly tried to use opus.
Extract the candidate selection into a pure, testable function
SelectAutoLoadBackends that:
- sorts the candidate list deterministically (no more map-order
nondeterminism), and
- for a `.gguf` model, filters to LLM-capable backends (via
core/config.BackendCapabilities) and puts llama-cpp first, so an
incompatible audio/codec/image backend can never win the trial loop.
If filtering would leave zero candidates, the full sorted set is returned
unchanged, so a previously-loadable model is never made unloadable.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(model): break core/config <-> pkg/model import cycle in backend auto-detect
The #9287 auto-detect change made pkg/model/autoload.go import core/config
for the backend capability table. core/config already imports pkg/model
(runtime_settings_registry.go uses model.DefaultWatchdogInterval), so this
closed a core/config -> pkg/model -> core/config import cycle and broke the
build and golangci-lint.
Invert the dependency so the lower-level pkg/model no longer imports the
higher-level core/config. pkg/model exposes RegisterLLMCapableBackendFunc and
uses the registered predicate; core/config (which owns the capability table)
registers it from an init(). The deterministic, GGUF-type-filtered selection
behaviour is unchanged. When the predicate is unwired the GGUF filter is
skipped, preserving the existing zero-candidate fallback.
The unit test now injects a fake capability predicate so SelectAutoLoadBackends
is exercised independently of the core/config table.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
* fix(grammars): reject cyclic $ref in JSON-schema grammar to prevent stack-overflow crash
JSONSchemaConverter.visit resolved $ref entries by recursively calling
itself with no cycle detection. A client-supplied grammar_json_functions
schema whose $defs contains a self- or mutually-referential $ref (e.g.
{"A": {"$ref": "#/$defs/A"}}) made visit recurse until the goroutine
stack was exhausted, producing a fatal "stack overflow" that kills the
whole process rather than failing the single request. The schema is
converted synchronously in the /v1/chat/completions handler before any
backend call, so this is an unauthenticated remote crash. Fixes#11020.
Track the $ref targets currently on the recursion stack and error out
when one is re-entered, while popping after each descent so sibling
(non-cyclic) reuse of the same $ref is still allowed.
Signed-off-by: Tai An <antai12232931@outlook.com>
* fix(grammars): add a bounded recursion depth and cover llama31 $ref cycles
Addresses the review on #11041. The stack-set approach catches cyclic
$ref chains, but a deeply nested yet acyclic client schema (thousands of
nested arrays/objects) can still recurse through visit until the
goroutine stack is exhausted, which is the same unauthenticated remote
crash surface as #11020.
- Add a bounded depth counter to JSONSchemaConverter.visit (incremented
with a defer-based cleanup, capped at maxSchemaDepth = 256, far above
any realistic schema) so an over-deep schema fails the request with an
ordinary error instead of crashing the process.
- Apply the same cyclic-$ref guard and depth bound to
LLama31SchemaConverter.visit, the other production grammar entry point
named in #11020, which previously had no cycle detection at all.
- Regression tests: a deeply nested acyclic schema is rejected while a
moderately nested one still builds, plus direct/indirect $ref cycle
and depth tests for the llama31 converter.
Signed-off-by: Tai An <antai12232931@outlook.com>
* test(grammars): make llama31 cycle fixtures valid function-call shapes
The two new llama31 $ref-cycle specs asserted on "cyclic $ref" but the
converter requires each top-level oneOf alternative to carry its
function-name property before descending, so both fixtures failed
earlier with "no function name found in the schema" and never reached
the cycle guard.
Give each fixture a valid llama31 shape: construct the converter with
NewLLama31SchemaConverter("function"), put "function": {"const": "test"}
on the top-level alternative, and hang the cyclic $ref under an
arguments property, so all 29 grammar specs pass and the assertions
genuinely observe the cyclic $ref error.
Signed-off-by: Tai An <antai12232931@outlook.com>
---------
Signed-off-by: Tai An <antai12232931@outlook.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
backend_pr.yml and test-extra.yml already filter themselves, so a gallery-only
or docs-only PR costs them about one job each. The image and Go workflows had no
filter of any kind, so a one-line gallery/index.yaml edit queued 20 jobs: 7
container image builds, 3 GoReleaser/darwin launcher builds, 3 unit test jobs, 2
golangci-lint, 1 e2e, 1 yamllint, plus the 3 that correctly stop after their
detect step. A docs-only PR queued the same.
This matters more than the job count suggests. Measured over the week to
2026-07-30, 97% of CI wall-clock is queueing and 3% is execution: a median
5-hour queue against a 4-20 minute median job. Cutting job count is the only
lever that shortens feedback time. The volume is there to cut, too: 13
gallery-only PRs merged that week with 10 open at once, and 78 of the 137 PRs
opened were bot-generated.
Add paths-ignore for gallery/**, docs/**, examples/** and **/*.md to the
pull_request trigger of image-pr.yml, build-test.yaml and tests-e2e.yml, and
add gallery/** to lint.yml, which already excluded the rest. That drops 13 of
the 20 jobs. None of the four can observe such a diff: gallery metadata is
parsed at runtime and never copied into an image, docs and markdown never enter
one at all, GoReleaser and the launcher take no such input, the e2e suite drives
backends over gRPC directly, and golangci-lint runs new-from-merge-base so a
diff with no touched Go lines is a no-op. The build-test exclusion also frees
macOS capacity, which is the scarcest runner class.
The two checks that do validate the gallery are deliberately left alone.
test.yml still runs core/gallery/variants_lint_test.go, which reads the real
gallery/index.yaml and asserts the index invariants, and yaml-check.yml still
lints the syntax.
paths-ignore skips a run only when every changed file matches, so a PR touching
the gallery and Go code still runs everything. master carries no branch
protection and no rulesets, so a skipped workflow reports no status and nothing
waits on it; .agents/ci-caching.md records that constraint for whenever required
status checks are introduced.
image.yml on master push is left unfiltered on purpose: skipping it would stop
the master and latest tags being republished for a gallery commit, which is a
publishing decision rather than a cost one.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* gallery: add Nanbeige4.2 3B GGUF variants
Add Q4_K_M and Q8_0 builds of the compact Nanbeige4.2 agentic and reasoning model, grouped as install-time variants.
Assisted-by: Codex:gpt-5
* gallery: simplify Nanbeige4.2 model name
Apply the maintainer-requested canonical model name while retaining the quantization variants under the entry.\n\nAssisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add the official Q4_K_M build and seven APEX quality and size variants for llama.cpp.
Assisted-by: Codex:gpt-5 [Hugging Face API]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
GenerateImage hardcoded TilingParamsSetEnabled(vaep, false), so tiled VAE
decoding was unreachable from a model config even though all four upstream
setters were already bound in main.go.
Sampling runs in latent space, but the final VAE decode expands to full
resolution and needs one large compute buffer. At 1024x1024 that buffer
exceeds 8GB, which fails on two kinds of device: cards without the VRAM
for a full-frame decode, and drivers that cap a single allocation
regardless of how much memory is free. Mesa RADV reports a 4GiB
maxMemoryAllocationSize, so a Radeon 8060S with 74GiB of device-local
heap still cannot serve that decode:
[INFO ] sampling completed, taking 251.82s
[INFO ] decoding 1 latents
ggml_vulkan: Requested buffer size exceeds device buffer size limit:
ErrorOutOfDeviceMemory
[ERROR] vae: failed to allocate the compute buffer
[ERROR] decode_first_stage failed for latent 1
Every sampling step completes and then the run is discarded at the last
stage, so the whole generation is wasted.
Add three options, parsed in Load and applied per generation:
vae_tiling:true enable tiled decoding (bare flag also works)
vae_tile_size:512 tile size, or 512x384 for a rectangle
vae_tile_overlap:0.25 overlap between tiles
Tiling stays off unless requested, so existing models are unaffected. Tile
size and overlap only reach the library when the operator set them, which
keeps upstream's defaults rather than pushing a zero, and an unparseable
value is treated as absent for the same reason.
Truthy spellings match what load_model already accepts for its own bool
options, and the bare-flag form matches diffusion_model, so no new
convention is introduced.
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
Add the CAJAL GGUF gallery template and gallery index entry for local llama-cpp installs.
Assisted-by: Codex:gpt-5
Signed-off-by: Ching Kao <0980124jim@gmail.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
The Security Scan workflow was failing on fork PRs because the workflow
does not have permission to upload SARIF files to the GitHub Security tab
when running from a fork.
This change adds '!github.repository.fork' checks to all steps
to prevent the workflow from running on fork repositories.
This fix should be applied to the main repository so that
all forks inherit the correct configuration.
Fixes#10322, #10318, #10320, #10321
Co-authored-by: ghshhf <ghshhf@users.noreply.github.com>
fix(chatterbox): pin cublas12 torch/transformers and setuptools so the backend loads
The cuda12-chatterbox gallery backend fails to load on a fresh install
because several deps in requirements-cublas12.txt are unpinned:
- torch/torchaudio: unlike requirements-cublas13.txt and
requirements-cpu.txt, this file has no --extra-index-url, so pip pulls
a wheel whose CUDA runtime (cu130) is newer than the host driver
supports ("NVIDIA driver on your system is too old"). Add the cu124
index and pin torch/torchaudio 2.6.0+cu124.
- transformers: resolves to 5.x, which dropped LlamaConfig.rope_theta
that chatterbox-tts 0.3.1's T3 config still reads. Cap to <5.
- setuptools: 81+ dropped pkg_resources, which perth imports under a
bare try/except and silently sets PerthImplicitWatermarker=None,
making ChatterboxTTS.__init__ raise 'NoneType' object is not callable.
Cap to <81 in requirements.txt.
Fixes#11070
Signed-off-by: Tai An <antai12232931@anaiguo.com>
Co-authored-by: Tai An <antai12232931@anaiguo.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
response.create accepts a metadata map and ResponseCreateParams has carried
the field all along, but triggerResponse never copied it onto the Response it
emits, so both terminals went out with metadata omitted.
That field is the only thing tying a terminal event back to the
response.create that asked for it. Our own doc comment on ResponseCreateEvent
says so — "the metadata field is a good way to disambiguate multiple
simultaneous Responses" — and it is what makes an out-of-band response
(conversation: "none") usable at all: a client running one alongside the
spoken conversation has no way to tell its own answer from the conversation's,
so it waits for a reply it already received and gave away.
Found from the client side: a headless text turn injected into a live session
was answered correctly in about a second, and the caller still blocked until
its own two-minute timeout because it could not recognise the answer.
Carry the map on liveResponse so all three terminals (in_progress, cancelled,
completed) report it, and leave it omitted when response.create sent none.
Assisted-by: Claude:claude-opus-5 gofmt
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat: add Valkey Search vector store backend
Add a new built-in Go gRPC store backend 'valkey-store' that implements the
four Stores RPCs (Set/Get/Delete/Find) against the Valkey Search module (FT.*)
using the pure-Go github.com/valkey-io/valkey-go client. It is selected via the
existing per-request 'backend' field on /stores, so there is no proto or HTTP
API change, and it mirrors the in-memory local-store while adding persistence
across restarts and opt-in HNSW.
Each vector is a Valkey HASH keyed by hex(little-endian float32); the index is
created lazily on first Set (FLAT+COSINE by default), cosine similarity is
derived as 1-distance, and namespaces get a collision-resistant token. Includes
unit tests (valkey-go mock) and env-gated integration tests against
valkey/valkey-bundle, plus build/matrix/gallery wiring and docs.
Assisted-by: Kiro:claude-opus-4.8 golangci-lint
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* Address review feedback: recover persisted index dimension, harden Find
- Load now recovers the persisted vector DIM from FT.INFO (not just index
existence), so a post-restart Set/Find validates against the real DIM
instead of silently re-learning a wrong one and dropping mismatched
vectors from the index. This also restores Find's dimension check after
a restart.
- StoresFind treats a dropped/missing index as an empty store (empty
result, no error) and clears the stale indexCreated flag, matching
local-store's empty-store behaviour.
- StoresSet reuses checkDims for its per-key length check so the four RPCs
share one dimension-guard implementation.
- Add unit tests for FT.INFO dimension recovery, loadIndexState, and the
dropped-index Find path.
Assisted-by: Kiro:claude-opus-4.8
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* Address review feedback: TLS ServerName/CA, Find nil-check, config fail-fast
Addresses external review comments on the valkey-store backend:
- StoresFind now rejects a nil/empty query Key before dereferencing it,
so a malformed gRPC request can no longer panic the backend.
- TLS: derive ServerName (SNI) from the VALKEY_ADDR host so certificate
verification works for IP-addressed endpoints, and add VALKEY_TLS_CA_CERT
(custom CA bundle) and VALKEY_TLS_SKIP_VERIFY (testing-only) knobs.
- Config integer parsing now fails fast on a malformed value (e.g.
VALKEY_HNSW_M=1x6) instead of silently defaulting, matching the
fail-fast behaviour of the index-algo/distance-metric validation.
- Add VALKEY_DB (SELECT n) support for logical-DB isolation.
- Cap the human-readable part of a namespace token at 64 chars so a very
long model name cannot produce an unbounded key prefix / index name
(the appended short hash keeps distinct namespaces collision-free).
- Document the KNN-query injection-safety invariant (fields are constants)
and why StoresGet uses a single aggregate DoMulti deadline for reads.
- Unit tests for the Find nil/empty-key guard, fail-fast HNSW parsing,
and VALKEY_DB parsing/validation; docs + .env updated for the new vars.
Assisted-by: Kiro:claude-opus-4.8 golangci-lint
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* Address review feedback: configure valkey-store via model config
richiejp asked that the valkey-store backend take its configuration from
a model config rather than process-wide VALKEY_* environment variables,
so multiple stores can each have their own Valkey config within one
LocalAI process. This removes every env access from the backend and
routes config through the model-config seam every other backend uses.
- config.go: loadConfig(opts *pb.ModelOptions) now parses the model
config `options:` list (key:value strings, split on the first ':')
instead of os.Getenv. Option keys mirror the old VALKEY_* names without
the prefix (addr, index_algo, distance_metric, ...). Defaults, fail-fast
validation and the mandatory client name are unchanged.
- store.go: Load threads opts into loadConfig; TLS comments/errors renamed
off the VALKEY_* names.
- core/backend/stores.go: StoreBackend and NewVectorStore take a
*config.ModelConfigLoader, resolve the per-store ModelConfig by store
name, and pass its Options (and Backend when unset) to the backend via
WithLoadGRPCLoadModelOpts. No config -> default backend + built-in
defaults, preserving the zero-config experience.
- Endpoints/routes/application: thread the config loader to StoreBackend.
- Unit + integration tests: configure via options; the integration test
passes addr through the model-config path (VALKEY_ADDR is now only the
test harness locating the server).
- docs + .env: document the model-config options, drop the env var table.
Assisted-by: Kiro:claude-opus-4.8
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* Remove valkey-store informational comment from .env The backend is configured via model config, not env vars — the comment was unnecessary noise in .env. The configuration is already documented in docs/content/features/stores.md.
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* feat(valkey-store): gate Load on NamespacePrefix to refuse autoload probing Mirror local-store's pattern: reject model names without store.NamespacePrefix so the model loader's greedy autoload probe cannot bind an arbitrary model name to the vector store backend (the #9287 failure mode). Also adds unit tests for the gate covering: prefixed namespace, prefix alone, unprefixed model name, empty model, and nil opts.
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* feat(valkey-store): add username_env/password_env credential indirection Add support for resolving Valkey credentials from environment variables named in the model config, mirroring cloud-proxy's api_key_env pattern. This keeps secrets out of model YAML files and lets distinct store configs each reference their own credentials. Options: username_env / password_env name the env var holding the value. The direct username / password options still work and take precedence when both are set (backward compatible). Includes 5 unit tests and updated stores.md documentation.
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* fix: correct rebase artifacts in backend-matrix.yml and Makefile Fix two issues introduced by the conflict-resolution script during the rebase onto master: 1. .github/backend-matrix.yml: valkey-store entries were merged INTO the cloud-proxy entries (duplicate keys in same YAML map items) instead of being separate list items. This broke cloud-proxy Linux builds and the cloud-proxy darwin entry lost its build-type/lang. Fixed by making them standalone entries and restoring cloud-proxy exactly as on master. 2. Makefile: duplicated .NOTPARALLEL and docker-build-backends lines. Collapsed to single lines that are master's current content plus the valkey-store additions. Also adds the three optional pickups from #10801: - /valkey-store in .gitignore (the built binary) - valkey-store row in docs/content/reference/compatibility-table.md - valkey-store line in backend/README.md
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
---------
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
Co-authored-by: Daria Korenieva <daric2612@gmail.com>
* docs(gpu): add gfx1151 / ROCm 7.x and fix ROCm section
- Fix typo: "deditated" → "dedicated", "ROCm6" → "ROCm"
- Add ROCm 7.x to requirements (alongside ROCm 6.x)
- Add Ubuntu 24.04 to tested OS list
- Add AMD Strix Halo / gfx1151 section with kernel params,
required env vars (HSA_OVERRIDE_GFX_VERSION, ROCBLAS_USE_HIPBLASLT),
and Docker Compose example
- Add gfx1151 to the list of compiled GPU targets
- Add ROCm version column to verified devices table
- Add gfx1151 / Radeon 8060S (ROCm 7.11.0) as verified device
* fix(docs/gpu): correct gfx1151 section — env vars, image tag, safety warning
- Add all 4 required env vars (HSA_OVERRIDE_GFX_VERSION, ROCBLAS_USE_HIPBLASLT,
HSA_XNACK=1, HSA_ENABLE_SDMA=0) with descriptions in a table
- Fix Docker Compose example to use the ROCm 7.x image tag (-gpu-hipblas-rocm7),
not the ROCm 6.x image
- Add explicit warning: GGML_CUDA_ENABLE_UNIFIED_MEMORY must NOT be set
(even =0 activates hipMallocManaged due to getenv != nullptr check)
- Add --force-recreate note (docker restart does not update container env)
- Add tested hardware note (Geekom A9 Mega / Ryzen AI MAX+ 395)
* docs(gpu): single ROCm image — drop -rocm7 tag suffix
Per maintainer feedback on PR #9229: there is only one ROCm/hipblas
main image, and it ships with ROCm 7.x by default — no separate
-rocm7 tag.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Document the reverse-proxy settings needed for long-running and multimodal requests, and distinguish edge-generated 504 responses from the optional LocalAI busy watchdog.
Assisted-by: Codex:gpt-5 [Codex]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(3d): add Generate3D RPC, FLAG_3D capability, and /v1/3d/generations endpoint
Adds the plumbing for image-conditioned 3D asset generation (binary
glTF / GLB output), modeled on the video generation path:
- backend.proto: Generate3D RPC + Generate3DRequest (staged image src,
glb dst, seed/step/cfg_scale/texture_steps, quality and background
enums, params map for backend-specific extras)
- pkg/grpc: thread Generate3D through client, server, embed, base and
the backend interfaces; connection-evicting and distributed-node
wrappers (in-flight tracking + file staging) included
- core/config: FLAG_3D usecase (guessed only for the trellis2cpp
backend), '3d' canonical usecase string mapped to the Generate3D
method, and a '3d' output modality
- REST: POST /v1/3d/generations (+ unversioned alias) returning
OpenAIResponse with a /generated-3d URL or b64_json; conditioning
image accepted as URL, base64, or data URI; quality/background
validated at the edge; .glb served as model/gltf-binary
- auth: '3d' route feature (default ON); /api/instructions entry
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(trellis2cpp): add the trellis2.cpp image-to-3D backend
Wraps localai-org/trellis2cpp (C++/GGML port of Microsoft TRELLIS.2,
pbr-textures branch) as a Go+purego backend, following the
stablediffusion-ggml pattern:
- backend/go/trellis2cpp: purego bindings to the flat C ABI (v9,
asserted at startup), eager pipeline load with model-set validation
(refuses non-trellis GGUFs; degrades coarse/geometry-only/textured
exactly like the upstream demo), Generate3D via t2_generate +
t2_bake_glb writing a binary glTF to dst. Weight-free unit tests
cover resolution/validation/param mapping — CI never downloads the
multi-GB GGUF set or runs inference.
- CPU SIMD variants build into per-variant directories (the shared
libggml sonames collide across variants, unlike sd-ggml's flat
renamed-.so scheme); run.sh picks one via /proc/cpuinfo.
- CI wiring: backend-matrix entries (cpu, cuda12/13, vulkan
amd64+arm64, l4t, l4t-cuda13, darwin metal), index.yaml meta +
latest/master image entries, bump_deps tracking of the pbr-textures
branch, changed-backends.js mapping, top-level Makefile targets.
- Importer: auto-detects trellis GGUF repos/URIs (registered before
llama-cpp so the .gguf match isn't stolen) and expands any trellis
URI to the full 10-file component set spanning the three LocalAI-io
HF repos.
- Gallery: trellis2-4b (full PBR + 1024 cascade) and
trellis2-4b-geometry (512 untextured) with verified sha256s.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(ui): 3D generation page with native GLB viewer and IndexedDB history
Adds a Studio tab + /app/3d page for the new image-to-3D endpoint:
- GlbViewer ports the trellis2cpp demo's dependency-free WebGL2
renderer (quaternion trackball, metallic-roughness PBR, ACES,
hidden-line wireframe with a bounded index budget) and pairs it with
a minimal GLB parser for the two forms t2_bake_glb emits — dense
vertex-PBR (linear COLOR_0 + _METALLIC_ROUGHNESS, uploaded as
normalized integers) and the opt-in UV-atlas textured form. Parsing
happens before any GL so stats and errors render without WebGL2.
- use3DHistory stores past generations (params, input thumbnail, and
the GLB blob itself) in IndexedDB with keep-newest-20 eviction —
GLBs are multi-MB binaries localStorage can't hold — and the page
offers a download button for the active GLB.
- Wiring: CAP_3D capability constant (FLAG_3D — the exact string
/api/models/capabilities serves), threeDApi, router entries, Studio
tab, vite dev proxy, en locale keys.
- e2e: render-smoke entry plus a focused spec that feeds a real
one-triangle vertex-PBR GLB through the parser/viewer and exercises
IndexedDB persistence, selection, deletion, and API errors.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(3d): address API correctness and UX issues
Keep 3D generation on the LocalAI-specific /3d/generations route and ensure authentication and permissions cover it.
Propagate distributed transfer failures, publish a portable ARM64 backend image, honor importer overrides, and align discovery, upload validation, and touch controls.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(3d): add previewable print remeshing
Add a single-detail CGAL Alpha Wrap workflow for existing Trellis GLBs, including PBR reprojection, API documentation, tracing, and an in-browser preview before download.
Allow the remesh route to enforce its 512 MiB upload cap independently of the smaller global default so generated high-resolution meshes can be processed.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* build(trellis2cpp): centralize remesh dependency pins
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(kokoros): implement Generate3D stub for new proto RPC
The Generate3D RPC added to backend.proto for the trellis2cpp backend
made tonic's generated Backend trait require generate3_d, breaking the
kokoros-grpc build. Return unimplemented like the other unsupported
modalities.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
backend/backend.proto is consumed by every language, so its SHARED_BUILD_INPUTS
rule could only ever be always/always: 417 Linux plus 56 Darwin builds. It fires
on ~1.3% of commits (10 of 767 over six months), which made it the single
largest CI cost driver in the repo.
On 2026-07-29 the queue reached 2178 jobs against 8 concurrent runners. Four
runs totalling 935 of those jobs were triggered by nothing but a proto edit. The
largest, 378 jobs on master, came from PR #11158, whose entire proto diff was
six lines adding `bool cache_prompt = 8;` to one message. No backend that does
not read that field behaves any differently for it.
Make the rule content-aware. changed-backends.js resolves backend.proto at the
base revision (the contents-API pattern already used for backend-matrix.yml) and
hands both texts to protoChangeIsAdditive(), which compares them structurally so
a comment reflow, reindent or field reorder does not read as a change. An
additive-only edit (new field with an unused number, new message, new enum
value, new RPC) suppresses the rule and rebuilds nothing; a removed, renumbered,
retyped or renamed field, a dropped RPC or a changed option still rebuilds
everything, as does an unresolvable base revision.
Every other matched rule is untouched, so a PR that edits the proto and
scripts/build/ is still a full rebuild, and the weekly full-matrix cron remains
the backstop for stale wheels.
Verified against all ten proto commits of the preceding six months: the nine
with a resolvable parent all classify as additive, and controls covering a
retyped-and-renumbered field, a deleted RPC, identical revisions and a
reindent-plus-comment-reflow all classify correctly.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
A new capability deserves its own page under `docs/content/features/`, plus cross-links from related features and an entry in `docs/content/whats-new.md`. See the pattern used by `face-recognition.md` / `object-detection.md`.
A new capability deserves its own page under `docs/content/features/`, plus cross-links from related features. See the pattern used by `face-recognition.md` / `object-detection.md`.
Announcing it is the release's job, not this page's: the capability gets covered in the release blog post under `website/content/blog/`. See [preparing-a-release.md](preparing-a-release.md). `docs/content/whats-new.md` is only a pointer at the blog and GitHub Releases, so there is nothing to add there.
## Path protection rules
@@ -334,7 +336,7 @@ When adding a new endpoint:
- [ ] Swagger block on the handler: `@Summary`, `@Tags`, `@Param`, `@Success`, `@Router`
- [ ] If new capability area (new swagger tag): entry in `instructionDefs` in `core/http/endpoints/localai/api_instructions.go` + test count bumped in `api_instructions_test.go`
- [ ] If new `FLAG_*` usecase flag: matching `CAP_*` symbol exported from `core/http/react-ui/src/utils/capabilities.js`
- [ ]`docs/content/features/<feature>.md` created; cross-links from related feature pages; entry in `docs/content/whats-new.md`
- [ ]`docs/content/features/<feature>.md` created; cross-links from related feature pages; capability covered in the release blog post (see [preparing-a-release.md](preparing-a-release.md))
**Quality**
- [ ] Error responses use `schema.ErrorResponse` format (or `echo.NewHTTPError` with a mapped gRPC status — see the `mapBackendError` helper in `core/http/endpoints/localai/images.go`)
@@ -122,18 +122,89 @@ The per-backend prefix match only sees files under a backend's own directory, so
| Changed path | Rebuilds |
|---|---|
| `backend/backend.proto` | everything (all languages compile or copy it) |
| `backend/backend.proto` | nothing if the edit is additive-only, otherwise everything (see below) |
| `backend/Dockerfile.<x>` | the Linux entries whose `dockerfile:` names it |
| `backend/python/common/` | Python, Linux + Darwin |
| `scripts/build/package-gpu-libs.sh` | Python, Linux only |
| `scripts/build/package-gpu-libs.sh` | every Linux entry (Python, Go and C++ all run it) |
| `scripts/build/<lang>-darwin.sh` | the Darwin entries that build target routes to |
| `.github/workflows/backend_build[_darwin].yml` | everything on that OS |
| anything else under `scripts/build/` (except `*_test.sh`) | everything — conservative default for unclassified packaging inputs |
Deliberately excluded: `backend/index.yaml` (gallery metadata, never enters an image), `.github/backend-matrix.yml` (adding a backend would rebuild all of them), `backend/Dockerfile.base-grpc-builder` (owned by `base-images.yml`), and the root `Makefile` (touched in ~11% of commits, and its backend-relevant edits arrive alongside the backend directory anyway). `make test-ci-scripts` pins all of this.
#### `backend/backend.proto` is content-filtered, not path-filtered
Every language consumes the proto, so a path rule for it can only ever say "rebuild all 473 images". It changes in ~1.3% of commits, and that was enough to make it the single largest CI cost driver in the repo: on 2026-07-29 four runs totalling 935 queued jobs traced to nothing but a proto edit, one of which (#11158) was a six-line diff adding `bool cache_prompt = 8;`.
An additive proto edit cannot change how a backend that never references the new symbol behaves, so `filterMatrix()` suppresses the rule for one. `changed-backends.js` fetches `backend/backend.proto` at the base revision (same contents-API pattern as `.github/backend-matrix.yml`) and hands both texts to `protoChangeIsAdditive()`, which compares them structurally rather than textually:
- **Additive, rebuilds nothing**: a new field with an unused number, a new message, a new enum value, a new RPC. Comment, whitespace and ordering changes also land here.
- **Breaking, rebuilds everything**: a removed, renumbered, retyped or renamed field, a dropped RPC, a changed `option` or `package`. So does an unresolvable base revision, matching the run-all posture used for a truncated diff.
Checked against every proto commit in the preceding six months, all nine resolvable ones classify as additive. Note the tradeoff this accepts: generated stubs do change for an additive edit, so image bytes would differ on a rebuild even though behavior does not. That is the same standard already applied when the filter declines to rebuild on unrelated `pkg/` changes, and the weekly cron remains the backstop.
The Sunday 06:00 UTC cron on `backend.yml` exists specifically because path filtering can leave Python backends frozen on stale wheels. `DEPS_REFRESH` (below) only fires when the build actually runs, so an untouched Python backend would never re-resolve its unpinned deps. The weekly cron is the safety net.
## Content-blind PRs skip the workflows that cannot see them
`backend_pr.yml` and `test-extra.yml` filter themselves (matrix generation and a `detect-changes` job), so a gallery-only or docs-only PR costs them about one job each. The Go and image workflows had no filter of any kind, so a one-line `gallery/index.yaml` edit queued 20 jobs, and a docs-only PR queued the same.
This is worth more than it looks. Measured over the week to 2026-07-30, **97% of CI wall-clock is queueing, 3% is execution** (median queue ~5h against a 4-20min median job). Cutting job count is therefore the only lever that shortens feedback time; making individual jobs faster moves 3%.
The volume is real: 13 gallery-only PRs merged that week with 10 open at once, and 78 of the 137 PRs opened were bot-generated.
`paths-ignore` on the PR trigger of `image-pr.yml` (7 jobs), `build-test.yaml` (3), `lint.yml` (2) and `tests-e2e.yml` (1) drops 13 of those 20. The excluded set:
| Path | Why no image or Go build can see it |
|---|---|
| `gallery/**` | Model-gallery metadata, parsed at runtime, never copied into an image |
| `docs/**`, `examples/**`, `**/*.md` | Never enter an image or a binary. `lint.yml` already excluded these before gallery was added |
### `backend/{cpp,go,python}/**` on `image-pr.yml` and `build-test.yaml` only
Version-pin bumps dominate PR volume: 48 `update/*` PRs in the week to 2026-07-30, from 16 pins, each a two-line diff. Most edit nothing but one `backend/*/<name>/Makefile`.
Neither of those two workflows can observe such a change. `make build` is `go build ./cmd/local-ai`, GoReleaser builds the same plus `./cmd/launcher`, and the core image's final stage ships only `entrypoint.sh`, `healthcheck.sh` and that binary. The per-backend trees are copied into the builder but nothing in them reaches the output.
What still triggers a full run, because none of it lives under those prefixes:
-`backend/backend.proto` — feeds `protogen-go`, so it does change the binary.
-`go.mod` / `go.sum` — the `go mod tidy` before-hook.
-`backend/Dockerfile.*` and anything else directly under `backend/`.
Deliberately **not** applied to:
| Workflow | Why it must keep seeing `backend/**` |
|---|---|
| `test.yml` | `TEST_PATHS` explicitly includes `./backend/go/cloud-proxy/...`, `./backend/go/local-store/...` and `./backend/go/valkey-store/...` |
| `lint.yml` | `.golangci.yml` carries `backend/`-scoped rules, so golangci-lint covers that tree |
| `tests-e2e.yml` | The e2e suite drives real backends over gRPC |
| `backend_pr.yml` | This is the workflow whose entire job is to rebuild the changed backend |
What still runs, and why it has to:
| Workflow | Why it keeps running |
|---|---|
| `test.yml` (`tests`) | `core/gallery/variants_lint_test.go` reads the real `gallery/index.yaml` and asserts the index invariants (no duplicate entry names, no build claimed by two parents). This is the only schema-level check the gallery has. |
| `yaml-check.yml` (`Yamllint`) | Lints `gallery/` for syntax. |
| `backend_pr.yml`, `test-extra.yml` | Already self-filtering; they stop after the detect step. |
Two properties this relies on:
-`paths-ignore` skips a run only when **every** changed file matches, so a PR touching the gallery *and* Go code still runs everything. That is what makes the exclusion safe rather than a hole.
-`master` carries no branch protection and no rulesets, so a skipped workflow reports no status and nothing waits on it. If required status checks are ever introduced, these four entries must be excluded from the required set or PRs will hang on "Expected — Waiting for status to be reported".
### `image.yml` on master push is gated too, by a job rather than a path filter
The same reasoning applies to master pushes, and the volume is larger there: on 2026-07-30, **12 of the 23 queued `image.yml` runs** were commits like "add 1 new model to gallery" or a docs fix, each rebuilding all 18 container images.
`image.yml` now has a `changes` job that decides once whether the push can affect any image; the other 11 jobs carry `needs: changes` plus an `if:` on its output. Verified against the shipped `Dockerfile`: the final stage copies only `entrypoint.sh`, `healthcheck.sh` and the `local-ai` binary, there is no `go:embed` of `gallery/` or `docs/`, and the gallery is fetched at runtime from `github:mudler/LocalAI/gallery/index.yaml@master`. A gallery-only commit therefore produces byte-identical images, and the gallery change reaches users through GitHub immediately whether or not an image is rebuilt.
Two properties to preserve if you touch it:
- **It is a job gate, not `paths-ignore`.** `paths-ignore` on `push` also applies to tag pushes, and a tag created on an existing commit carries an empty commits list, which would silently skip the release image build. The gate short-circuits to "build" for `refs/tags/*`, and for any push whose base commit is missing, zero, or unresolvable.
- **The merge jobs must name the gate explicitly.** They use `if: ${{ !cancelled() && ... }}`, and `!cancelled()` is true when a dependency is *skipped*, so without the extra condition they would run and try to merge manifest lists for images that were never built.
## The `DEPS_REFRESH` cache-buster (Python backends)
Every Python backend goes through the shared `backend/Dockerfile.python`, which ends with:
@@ -169,15 +240,38 @@ RUN --mount=type=cache,target=/root/.ccache,id=<backend>-ccache-${TARGETARCH}-${
bash /usr/local/sbin/compile.sh
```
The compile script exports `CMAKE_C/CXX/CUDA_COMPILER_LAUNCHER=ccache` so CMake threads ccache through gcc/g++/nvcc. `cache-to: type=registry,mode=max` exports the cache mount data into the registry cache, so subsequent builds restore it.
The compile script exports `CMAKE_C/CXX/CUDA_COMPILER_LAUNCHER=ccache` so CMake threads ccache through gcc/g++/nvcc. Cache scope is per `(TARGETARCH, BUILD_TYPE)` so e.g. cublas-12 doesn't share with cublas-13 (their CUDA headers differ; cross-pollination would just be cache misses anyway).
On a `LLAMA_VERSION` bump, most translation units are byte-identical to the previous version's preprocessed source — ccache returns the previous `.o` and skips the real compile. Same for LocalAI source changes that don't actually touch llama.cpp's CMake inputs. Cache scope is per `(TARGETARCH, BUILD_TYPE)` so e.g. cublas-12 doesn't share with cublas-13 (their CUDA headers differ; cross-pollination would just be cache misses anyway).
### ⚠️ This ccache does nothing in CI today
This section previously claimed that `cache-to: type=registry,mode=max` "exports the cache mount data into the registry cache, so subsequent builds restore it". **That is not true.** BuildKit does not export the contents of a `--mount=type=cache` to a registry cache export. A cache mount lives in the builder's local state, and every CI job gets a fresh runner with a fresh builder, so `/root/.ccache` starts empty on every single build.
Measured on 2026-07-30 from the `ccache -s` output the compile script already prints (it runs `ccache -z` first, so the numbers are per-build):
| Job | Commit touched | Build time | ccache |
|---|---|---|---|
| 89766266951 (llama-cpp, cublas 13) | `backend/go/magpie-tts-cpp/Makefile` only | 6369s | **0 / 889 hits**, and 0 / 1778 |
The first two are the decisive control: commit `90355cd44` changed exactly one file, `backend/go/magpie-tts-cpp/Makefile`, nowhere near llama.cpp. The engine source was byte-identical to the previous build, which is precisely the case this section says ccache should serve, and the hit rate was still **0.00%**. A cache that was being restored but merely matching poorly would show partial hits; 0-of-N is the signature of an empty cache.
So the paragraph above about `LLAMA_VERSION` bumps reusing previous `.o` files describes an intended design that is not in effect. `Dockerfile.{llama-cpp,ik-llama-cpp,turboquant,bonsai,ds4,privacy-filter}` pay the ccache wrapper overhead and get nothing back. Multi-hour C++ rebuilds are recompiling identical translation units from scratch.
**Do not "fix" this by adding cache mounts to more Dockerfiles.** Wiring the same mount into `Dockerfile.golang` (215 of the 434 matrix entries) was measured locally at 18% faster on a rebuild after a source edit, with a 71.5% ccache hit rate — but only because the local test reused one builder across both builds. In CI it would be a no-op for exactly the reason above.
Making this actually work needs the cache to live outside the builder. The options, none of them free:
- **ccache `remote_storage`** (ccache ≥ 4.4, HTTP or Redis backend) or **sccache** with an S3/GCS/Redis backend. Genuinely works across runners; needs a cache service to point at. quay.io is a registry, not a blob store, so the existing infra does not cover it.
- **Round-trip the cache dir through `actions/cache` on the runner**: restore it, pass it in, and export it back out via a build stage output. No external infra, but clunky, and the repo already sits at GitHub's 10 GB cache ceiling while the llama-cpp ccache alone is capped at 5 GB.
Until one of those lands, treat C++ backend builds as always-cold and spend the effort on not running them instead (path filtering, see above).
## Composite actions
Two composite actions handle runner-side prep:
- **`.github/actions/free-disk-space/action.yml`** — wraps `jlumbroso/free-disk-space@main` plus an explicit apt purge of dotnet/android/ghc/mono/etc. Reclaims ~6–10 GB on `ubuntu-latest`. No-op on self-hosted runners. Used by `backend_build.yml`, `image_build.yml`, `test.yml`, `tests-aio.yml`, etc.
- **`.github/actions/free-disk-space/action.yml`** — wraps `jlumbroso/free-disk-space@main` plus an explicit apt purge of dotnet/android/ghc/mono/etc. Reclaims ~6–10 GB on `ubuntu-latest`. No-op on self-hosted runners. Used by `backend_build.yml`, `image_build.yml` and `base-images.yml` — the jobs that actually build images. Deliberately **not** used by `test.yml`, which runs no buildx step.
- **`.github/actions/setup-build-disk/action.yml`** — relocates Docker's data-root to `/mnt` on hosted X64 runners. GHA hosted `ubuntu-latest` ships ~75 GB of unused space at `/mnt`; combined with the free-disk-space cleanup this gives ~100 GB working space — enough for ROCm dev image + vLLM torch install + flash-attn intermediate layers. No-op on self-hosted and on non-X64 hosted runners. Used by `backend_build.yml`, `image_build.yml`, `base-images.yml`.
Both actions run before any docker buildx step.
@@ -218,10 +312,20 @@ Eviction is rarely needed in normal operation — `DEPS_REFRESH` handles weekly
## What the cache does **not** cover
- The `free-disk-space` and `setup-build-disk` composite actions run on every job — these reclaim runner-state, not Docker layers, so BuildKit caches don't apply.
- The `free-disk-space` and `setup-build-disk` composite actions run on every job — these reclaim runner-state, not Docker layers, so BuildKit caches don't apply.`test.yml` deliberately does **not** use `free-disk-space`: it runs no buildx step, and the multi-GB fixture downloads that once justified it left `make test` in the test-suite reorg.
- Intermediate artifacts of `Build (PR)` are not pushed anywhere — PRs only build for verification.
- Darwin builds (see below) — macOS runners have no Docker daemon, so the registry-backed BuildKit cache cannot apply.
### The Linux Go workflows set `cache: false` on purpose
`test.yml`, `lint.yml`, `tests-e2e.yml` and friends pass `cache: false` to `actions/setup-go@v5`, unlike the darwin jobs. This looks like an oversight and is not.
Measured over the week to 2026-07-30, the `Set up Go` step has a **median of 11 seconds** on these runners. There is essentially nothing to win: the module download is not where the time goes. The expensive steps are compilation and test execution (`Test (with coverage gate)` at ~18.6min, `Test Backend E2E` at ~14.5min), and Go's build cache would have to survive across runners to touch those.
Enabling it also has a real cost. GitHub caps Actions cache at **10 GB per repo and the repo already sits at that ceiling** (31 entries), so every `setup-go` entry written by a branch with a distinct `go.sum` (222-375 MB on Linux, up to 1.4 GB on macOS) evicts something else. See the darwin cache budget below.
Before re-enabling this, measure `Set up Go` again and confirm it has actually become slow. If room is needed in the 10 GB budget, the cheapest evictions are the `docker.io--tonistiigi--binfmt` entries (~30 MB each, trivially re-fetched).
## Darwin native caches
`backend_build_darwin.yml` runs natively on `macOS-14` GitHub-hosted runners — there is no Docker, no BuildKit, no cross-job registry cache. Instead, the reusable workflow uses `actions/cache@v4` for four native caches that mirror the spirit of the Linux cache (warm by default, weekly refresh for unpinned Python deps, PRs read-only).
@@ -255,6 +359,26 @@ GitHub Actions caches are limited to 10 GB per repo. Steady-state worst case: ~8
One residual self-hosted reference remains in `test-extra.yml` (`tests-vibevoice-cpp-grpc-transcription` uses `bigger-runner` for the 30s JFK-decode timeout headroom). That's a separate concern.
### Small always-on jobs routed to `arc-runner-set`
The hosted pool is shared across the whole *account*, not per repo, so a burst in one repo starves the others. On 2026-07-31 it went to **zero scheduled jobs for 35 consecutive minutes** with 39 jobs queued, while `arc-runner-set` completed 12 jobs without interruption over the same window. Actions was healthy globally at the time (other public repos were scheduling normally), so this is an account-level throttle, not an outage.
`gh-pages.yml` (`build` + `deploy`) is therefore routed to `arc-runner-set` when `github.repository == 'mudler/LocalAI'`. It needs no fork-safety clause because it only triggers on push-to-master and `workflow_dispatch`, so it never executes pull-request code. The repository guard keeps forks (which have no such runner label) from queueing forever. It fetches its own toolchains via `setup-go` / `actions-hugo` and uses no `sudo`/`apt`.
#### What the `arc-runner-set` image actually contains
Measured 2026-07-31 on run `30637392862` by a preflight step, not assumed:
That is why `lint.yml` is **not** on the self-hosted pool. Both of its jobs were routed there and both failed in one second: `golangci-lint` needs `make` (for `make protogen-go`, itself needing `curl`+`unzip` to fetch protoc, and for `make lint`), and `build-scripts` additionally needs a C toolchain because the packaging-script tests compile a throwaway binary and inspect it with `ldd`. Both jobs are back on `ubuntu-latest`.
The preflight steps were deliberately left in place. They cost about a second on the hosted pool and mean that whenever the runner image gains `make` + `gcc`, re-routing is one `runs-on:` line per job and any remaining gap reports itself by name rather than as an opaque mid-build failure.
Note for any future re-route: `lint.yml` also triggers on `pull_request`, and a fork PR runs untrusted contributor code. That must never reach a persistent self-hosted runner, so any re-route has to stay push-only, e.g. `${{ (github.event_name == 'push' && github.repository == 'mudler/LocalAI') && 'arc-runner-set' || 'ubuntu-latest' }}`.
## Touching the cache pipeline
When changing `image_build.yml`, `backend_build.yml`, any of the `backend/Dockerfile.*` files, `Dockerfile.base-grpc-builder`, `.docker/install-base-deps.sh`, `.docker/<backend>-compile.sh`, or `scripts/changed-backends.js`:
@@ -70,3 +70,37 @@ The project documentation is located in `docs/content`. When adding new features
- **Configuration**: If you modify configuration options, update the relevant sections in `docs/content/`.
- **Examples**: providing concrete examples (like YAML configuration blocks) is highly encouraged to help users get started quickly.
- **Shortcodes**: Use `{{% notice note %}}`, `{{% notice tip %}}`, or `{{% notice warning %}}` for callout boxes. Do **not** use `{{% alert %}}` — that shortcode does not exist in this project's Hugo theme and will break the docs build.
## React UI styling
The React UI ships a design system in `core/http/react-ui/src/App.css`: design
tokens, form grids, data tables, stat cards, callouts, plus a small semantic
A release is not finished when the tag is pushed. The GitHub release, the blog post and the demo clips ship together, because the changelog says what moved and the post and the clips are what make anyone care.
## What a release must include
1.**Labels on the merged PRs.** GitHub generates the raw notes from PR labels, so label first, generate second. Wrong labels mean a miscategorised changelog that has to be edited by hand.
2.**`RELEASE_NOTES_vX.Y.Z.md`** at the repository root, in the house style: what changed, why it matters, PR numbers so people can read the diffs.
3.**A blog post under `website/content/blog/`.** One post per release, front matter with `title`, `date`, `author`, `category: "Release"`, `tags`, `summary` and `extracss: ["blog.css"]`. Cover the two or three changes that alter what a user does day to day, not the whole changelog, and link the PR numbers. See `website/content/blog/what-landed-in-localai-4-8.md` for the shape.
4.**Demo clips for the notable features.** Anything visible (a new backend, a UI change, a new endpoint, a measured speedup) gets a short screen recording. Put the file in `website/static/media/`, reference it from the blog post, and reuse it on the marketing pages where it fits.
A release without a post and without clips is incomplete, in the same way a user-facing code change without a docs update is incomplete.
## Clip conventions
- MP4, H.264, no audio track unless the feature is about audio. Keep them short (10 to 30 seconds) and loopable.
- Record the real thing. A clip from the engine's own benchmark suite or a real session, never a mockup.
- Where the change is a speedup, record both sides on the same machine on the same input, so the comparison is honest.
- Name the file after the feature, not the release (`vllm-race.mp4`, not `v4-8-demo.mp4`), so it stays reusable once the release is old.
- The marketing site plays clips with `muted loop playsinline preload="none"` and a `data-lazy` attribute, which the site's IntersectionObserver uses to play and pause them on scroll. Follow that pattern for anything you add.
## Order of work
Label the PRs, generate and edit the release notes, cut the draft release, record the clips while the branch is still fresh in your head, then write the post against the notes and the clips. Publishing the release and merging the post should happen on the same day.
The `creating-localai-releases` skill drives steps 1 to 3 and captures the React UI screenshots that go into the notes.
Beyond the parser names above, `Options[]` carries `--` prefixed engine flags (`--enable-prefix-caching`, `--kv-cache-dtype:fp8_e5m2`). `apply_options_to_engine_args` in `backend/python/common/vllm_utils.py` maps them onto `AsyncEngineArgs` fields, and it must run **before**`AsyncLLMEngine.from_engine_args()` - applying them afterwards is a silent no-op, which is exactly what issue #11130 was.
Things to keep straight when touching this:
- Precedence is typed proto fields → `options:` → `engine_args:`. `applyEngineArgDefaults` in `core/config/hooks_vllm.go` therefore skips seeding a production default whose key the user already set as an option, otherwise the later `engine_args:` pass would silently override them.
- Only `--` prefixed entries are engine flags; `tool_parser:`/`reasoning_parser:` and friends keep their meaning. Parser lookups accept both spellings via `normalize_option_key`.
- Unknown or uncoercible flags warn and are skipped, unlike `engine_args:` which is strict - `Options[]` is a shared bag and knows entries this mapping doesn't.
- Field types come from the annotation's *base* (`Literal["auto","float16"]` is not a float). The helper's tests are stdlib-only: `make test-python-helpers`.
Auto-defaults for known model families live in `core/config/parser_defaults.json` and are applied:
- at gallery import time by `core/gallery/importers/vllm.go`
- at model load time by the `vllm` / `vllm-omni` backend hook in `core/config/hooks_vllm.go`
| [.agents/preparing-a-release.md](.agents/preparing-a-release.md) | Cutting a release: PR labels, `RELEASE_NOTES_vX.Y.Z.md`, the blog post under `website/content/blog/`, and the demo clips under `website/static/media/` |
## Quick Reference
@@ -42,6 +43,7 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
- **Docs (docs-with-code rule)**: When you change user-facing behavior (API endpoints, CLI flags, config keys, or features), update the corresponding page under `docs/content/` in the SAME change, not as a follow-up. A user-facing change without a matching docs update is incomplete. See also the documentation conventions in [.agents/coding-style.md](.agents/coding-style.md).
- **New API endpoints**: LocalAI advertises its capability surface in several independent places — swagger `@Tags`, `/api/instructions` registry, auth `RouteFeatureRegistry`, React UI `capabilities.js`, docs. Read [.agents/api-endpoints-and-auth.md](.agents/api-endpoints-and-auth.md) and follow its checklist — missing any surface means clients, admins, and the UI won't know the endpoint exists.
- **Admin endpoints → MCP tool**: every admin endpoint that an admin would manage conversationally (install/list/edit/toggle/upgrade) MUST also be exposed as an MCP tool in `pkg/mcp/localaitools/`. The LocalAI Assistant chat modality and the standalone `local-ai mcp-server` consume that package; drift between REST and MCP is a real risk. Read [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) — the `TestToolHTTPRouteMappingComplete` test fails until you wire the new tool and update the route map.
- **Releases ship with a post and clips**: a release is not done at the tag. It needs labelled PRs, `RELEASE_NOTES_vX.Y.Z.md`, a blog post under `website/content/blog/`, and a short demo clip in `website/static/media/` for each notable feature. See [.agents/preparing-a-release.md](.agents/preparing-a-release.md).
- **Build**: Inspect `Makefile` and `.github/workflows/` — ask the user before running long builds
- **Backend OS coverage**: a new backend must target every OS it can build for, not just Linux. `.github/backend-matrix.yml` has two matrices — `include:` (Linux) and `includeDarwin:` (macOS / Apple Silicon). Most C/C++/GGML and many Python backends build on Darwin too — wire the `includeDarwin` entry + `backend/index.yaml``metal:` entries, or say in the PR why an OS is unsupported. See the darwin checklist in [.agents/adding-backends.md](.agents/adding-backends.md).
- **Gallery variant ranking**: a gallery entry can declare `variants` (alternative builds of the same weights), and LocalAI ranks the ones a host can run by engine preference first, size second. A new backend that should be preferred on some hardware must be listed in `engineNamePreferenceRules` in `pkg/system/capabilities.go`; the sibling `backendBuildTagPreferenceRules` speaks build tags rather than engine names, and using the wrong table matches nothing without erroring. See [.agents/adding-backends.md](.agents/adding-backends.md).
@@ -161,7 +161,7 @@ local-ai run https://gist.githubusercontent.com/.../phi-2.yaml
local-ai run oci://localai/phi-2:latest
```
To test a running LocalAI server from the terminal, open an interactive chat session from another shell. Inside the prompt, `/models` lists installed models and `/model <name>` switches between them.
To work with a running LocalAI server from the terminal, start the built-in agent from another shell. It answers questions, reads your files and runs commands on your machine, asking you to approve anything that changes state. Inside a session, `/models` lists installed models and `/model <name>` switches between them. See the [Terminal agent](https://localai.io/docs/features/terminal-agent/) docs.
```bash
# Terminal 1
@@ -195,7 +195,7 @@ For more details, see the [Getting Started guide](https://localai.io/basics/gett
- **August 2025**: MLX, MLX-VLM, Diffusers, llama.cpp now supported on Apple Silicon
- **July 2025**: All backends migrated outside the main binary — [lightweight, modular architecture](https://github.com/mudler/LocalAI/releases/tag/v3.2.0)
For older news and full release notes, see [GitHub Releases](https://github.com/mudler/LocalAI/releases) and the [News page](https://localai.io/basics/news/).
For older news and full release notes, see [GitHub Releases](https://github.com/mudler/LocalAI/releases) and the [blog](https://localai.io/blog/).
## Features
@@ -238,13 +238,14 @@ Most backends wrap a best-in-class upstream engine. A handful of them are native
| [magpie-tts.cpp](https://github.com/mudler/magpie-tts.cpp) | C++/GGML port of NVIDIA's Magpie TTS Multilingual 357M: 22.05 kHz mono text-to-speech in 5 voices and 9+ languages, with the NanoCodec neural codec and tokenizer/G2P embedded in a single GGUF |
| [ced.cpp](https://github.com/localai-org/ced.cpp) | C++/GGML port of the CED audio-tagging models: sound-event classification (527-class AudioSet) over REST and the realtime API for live recognition |
| [voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp) | Speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion), replacing the Python speaker-recognition backend |
| [voxtral-tts.c](https://github.com/mudler/voxtral-tts.c) | Voxtral Realtime 4B speech-to-text in pure C |
| [voxtral-tts.c](https://github.com/mudler/voxtral-tts.c) | Mistral Voxtral-4B-TTS text-to-speech in pure C: 20 preset voices across 9 languages, 24 kHz WAV output, no dependencies beyond libc |
| [vibevoice.cpp](https://github.com/mudler/vibevoice.cpp) | Native port of Microsoft VibeVoice for TTS (voice cloning) and long-form ASR with speaker diarization |
| [face-detect.cpp](https://github.com/mudler/face-detect.cpp) | Face detection, recognition, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace), replacing the Python insightface backend |
| [free-splatter.cpp](https://github.com/localai-org/free-splatter.cpp) | Pose-free 3D reconstruction (FreeSplatter): turns a handful of plain photos into 3D Gaussians, no camera poses or GPU required |
| [trellis2.cpp](https://github.com/localai-org/trellis2cpp) | C++/GGML port of Microsoft TRELLIS.2: single-image to textured 3D mesh (GLB with PBR materials) |
- [Integrations & community projects](https://localai.io/docs/integrations/)
- [Installation video walkthrough](https://www.youtube.com/watch?v=cMVNnlqwfw4)
- [Media & blog posts](https://localai.io/basics/news/#media-blogs-social)
- [Blog: release write-ups, benchmarks and engineering notes](https://localai.io/blog/)
- [Examples](https://github.com/mudler/LocalAI-examples) — including the [realtime voice assistant demo](https://github.com/localai-org/localai-realtime-demo) (Go client for the Realtime API with tool calling)
# Gated on cublas because the variable means nothing to any other build, so a
# stray CUDA_MAJOR_VERSION in the environment must not break `make clean` or
# a CPU build. It does still error for `BUILD_TYPE=cublas make clean`, which
# is the right trade: that invocation is asking about a CUDA build tree.
$(error CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) has no architecture list here (12 and 13do). Leave it empty for a native build, or pass CUDA_ARCHITECTURES explicitly.)
$(warning audio-cpp: libomp not found at '$(LIBOMP_PREFIX)'; building without OpenMP (single-threaded host DSP). Install it with `brew install libomp`, or set LIBOMP_PREFIX.)
CMAKE_ARGS+= -DENGINE_ENABLE_OPENMP=OFF
endif
else
# Portable Linux CPU. Upstream wires this to GGML_BACKEND_DL +
# GGML_CPU_ALL_VARIANTS + $ORIGIN rpath, so one build serves every CPU
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
