Resolve the existing model directory before choosing a missing file. On
macOS, resolving the missing leaf fails and otherwise compares an aliased
path with a canonical allowed directory. Keep this test on the Upload
error branch instead of the containment rejection branch.
Assisted-by: Codex:GPT-6
Resolve allowed directories before comparing them with resolved files.
Otherwise staging rejects valid files under macOS temporary paths.
Cover aliased roots, sibling paths, and symlinks escaping the root.
Assisted-by: Codex:gpt-6
Virtual model names have no primary file to anchor the worker path.
Companion assets still stage successfully, but relative options retain
an incorrect model directory and fail to load.
Derive the worker root from successfully staged option assets when the
primary path is absent. Cover Buffalo packs, files, directories,
overrides, and failed transfers. Document the frontend upgrade.
Assisted-by: Codex:gpt-6 golangci-lint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Merge current master and adapt incoming tests to the distributed APIs.
The sound mock now implements NewClientForNode and uses WorkerLocalAddress.
The upload finalization test supplies the existing direct test dialer.
Assisted-by: Codex:gpt-6
A worker can retain all model bytes with an unfinished-upload marker.
Retries then start at zero and repeatedly fail with HTTP 416.
Verify the existing bytes and finalize same-file retries at full size.
Reuse the normal integrity checks so corrupt content cannot be accepted.
Add regression coverage and document worker recovery.
Assisted-by: Codex:gpt-6 golangci-lint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The general e2e target builds a container, not the host binary that
cluster specs require. Run only its own package; dedicated distributed
and cluster targets retain the full subpackage coverage.
Assisted-by: Codex:GPT-6
Accept original embeddings and timestamps so clients can restore faces
when the in-memory store restarts. Derive stable IDs from exact vectors
to make registration retries preserve identity without duplicate entries.
Assisted-by: Codex:GPT-6 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): stage sound detection audio
Sound detection passes frontend temporary paths directly to remote
workers, unlike transcription. Stage the WAV before classification so
CED can read it without a shared temporary directory.
Preserve the original request for retries and propagate staging errors
without calling the backend. Cover staging, request preservation, and
error handling with regression tests.
Assisted-by: Codex:GPT-6 golangci-lint
* test(distributed): verify routed sound staging
Call sound detection through the client returned by SmartRouter.Route.
This checks interface dispatch through both routing wrappers, rather
than constructing FileStagingClient directly.
The test fails without the sound-staging override and passes with it.
Assisted-by: Codex:GPT-6 golangci-lint
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Community-maintained packagings that currently track releases —
Homebrew, ALT Sisyphus and the Gentoo local-ai overlay — with a note
that versions may lag. Placement and scope as discussed in the issue.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
* ⬆️ Update leejet/stable-diffusion.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(stablediffusion): adapt streaming options
Upstream now selects segmented weight streaming automatically and removes the stream_layers field. Keep the old LocalAI option as a no-op for existing model configurations.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update antirez/ds4
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(ds4): link upstream image helpers
The ds4 bump adds vision calls to the engine object. Link the new image preprocessing object into every backend target.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Preserve heartbeat checkpoints and backend readiness across the tunnel
transport changes. Update incoming tests for the renamed worker address
fields and health monitor arguments.
Assisted-by: Codex:gpt-6
* Update containers.md to fix podman image qualification
Signed-off-by: Alex Mazzariol <alex@alex-maz.info>
* docs(containers): clarify Podman image names
Podman can reject short image names when no registry is configured. Explain why the examples use fully qualified Docker Hub names.
Assisted-by: Codex:gpt-5.6
---------
Signed-off-by: Alex Mazzariol <alex@alex-maz.info>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The mutex round the realtime transport double added a snapshot accessor for each
recorded slice. Only the event one has a caller, so make lint refuses the build:
realtime_doubles_test.go:64:25: func (*fakeTransport).recordedAudio is unused (unused)
No spec has ever read the audio log, before the mutex or after it, so the
accessor is deleted rather than nolinted and the struct comment says where the
next one comes from. audioLog stays written, because a double that silently
discarded what a coordinator sent it would be a different double.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Probing every candidate before handing it out is right, and it made sixteen
specs that were never about the kernel depend on it. They build a supervisor
directly, name the ports they expect literally, and those literals sit inside
Linux's default ephemeral range, so with the real probe each one asks this host
whether 50051 is bindable at that instant.
The first full -race run over ./core/... and ./pkg/... after the probe landed
went red on four of them, and holding 50051, 50052, 50060 and 50061 from
another process turns eleven red deterministically. Nothing was wrong with the
allocator in either case: something else on the machine held a port, which is
the situation the probe exists to survive.
So the specs that assert bookkeeping now inject a probe that always says yes,
and say why once. The two specs that are about the probe leave the field unset
and keep asking real sockets, which is what still fails if the probe is removed.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
None was introduced by this branch and all three are in test code, which is
what made them survive: every suite passed on every run and only the race
detector said otherwise. A known-failing -race run is worse than a noisy one,
because a real race raised by production code lands in the same report and is
read as one of these.
galleryop: gatedModelManager guarded the recorded names and not the gate
channel itself. A spec frees the parked worker by closing the gate and
installing a fresh one, on the spec goroutine, while the worker goroutine reads
the field to park on it. The channel is now read and replaced under the same
mutex, and cleanup closes idempotently.
pkg/model: two specs swapped xlog's package logger to capture output and
swapped it back on cleanup. xlog.SetLogger writes an unsynchronised global, so
the restore raced with the backend process watcher, which logs while a process
is stopping; the captured bytes.Buffer was written by that goroutine and read
by an Eventually at the same time. SetLogger is now called once for the whole
test binary, from init, before a goroutine exists to race with, and a spec
swaps the DESTINATION under a mutex through a routing slog.Handler. Per-spec
level filtering is preserved deliberately: one of these specs asserts that a
debug emission is filtered OUT and would pass vacuously against a handler that
recorded everything.
openai: fakeTransport appended to its event and audio logs from the response
and turn coordinators' goroutines while a spec ranged over them. Both are
behind a mutex and are read through snapshot accessors; the fields are renamed
so a raw read from another spec file does not compile.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The claim queue's attempts counter grew without bound and nothing read it. At
the default two-second poll a permanently undispatchable row cost about 43000
UPDATEs a day, and it cost more than writes: rows are claimed oldest first, so
the oldest stuck row was re-claimed ahead of every newer one on every tick and
held a dispatch slot while it failed. One poison row starved the queue behind
it.
No dead letter, and that is the decision rather than the omission. Read
settleClaim: the only outcome that releases a claim is one where NOTHING was
learned about the work. No agent worker was connected, the tunnel broke, a peer
could not be reached, the stream was refused before the request body left this
replica. Not one of those is a worker saying it ran the job and it failed, and
an attempt ceiling would turn "the fleet was away long enough" into a job
failure nobody reported, which is the collapse this whole design exists to
prevent pointed at work instead of at nodes. The one verdict available here,
that no build of any worker serves this kind, is already settled as an answer.
So the retry stays unbounded and the RATE does not. Each release stamps the row
with the earliest it may be claimed again, doubling from two seconds to a cap
of sixty, computed in the release statement from the row's own attempts count
and stamped on the DATABASE clock, because that is the clock competing replicas
order the queue on. Queued work becomes claimable again within one cap of the
fleet returning, and a stuck row no longer holds the head of the queue. A claim
released by the reap carries no delay at all: that work was never handed to
anyone, so there is nothing to back off from.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A frontend keeps two caches of one http.Client per worker: the control
client's, built on the first verb issued to a node, and the HTTP file stager's,
built on the first file staged to it. Both are keyed by node ID and neither was
ever pruned. Their own comments said so and named what a fix would need, a
signal that a node has left, which did not exist when they were written and
does now.
The map slot is the smaller half. Each entry holds an http.Transport whose idle
connections are streams on that worker's tunnel, kept until IdleConnTimeout
even after the tunnel is gone, so ForgetNode closes them rather than leaving
them to the collector. Both are registered on the deployment's one departure
notifier, as two subscribers and not one: a node can be in either cache without
being in the other, and a single hook would say only that some client was kept.
ForgetNode is on the FileStager interface rather than on the one implementation
that has state to drop, so a stager that grows a per-node map later cannot be
added without answering the question, and so registerDepartureEvictions can
take a FileStager and still fail to compile if the registration is deleted. The
S3 stager's is a documented no-op that deliberately does not forward to the
control client, which registers itself: forwarding would evict a cache it does
not own, twice per departure, and the second drop would not appear in the
subscriber names the wiring spec reads.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
HealthMonitor.misses holds one consecutive-failed-probe count per (node,
model, replica) and nothing ever removed an entry whose row had gone. It is
the only per-node state in a frontend that grows on model churn rather than on
fleet size, so a deployment that loads and unloads models for months
accumulates an integer per tuple it ever probed and gives none back.
There are four ways a row stops being visible to the pass, not one. A node
departs and the pass skips its probes; a node goes offline or unhealthy on a
stale heartbeat and the pass skips it entirely; an operator sets a node
draining; or the row is removed by an unload, a scale-down or an eviction, and
nothing tells this monitor.
So the bound is the pass itself, and not a subscription on the departure
notifier. The notifier evicts the caches a DEPARTURE invalidates and it keeps
that one meaning; this reads a different fact, that there is no longer a row
to count misses against, and covers all four cases with one rule. A row the
pass could not probe is marked seen before the probe, so an unreachable worker
still leaves its streak exactly as it was rather than having it forgiven; a
pass that could not list the fleet prunes nothing, since it observed nothing.
Forgetting only ever delays a reap by up to the miss threshold and can never
cause one.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The backend port allocator allocated from its own bookkeeping alone. That
bookkeeping records what this worker did with a port, and the collision it
cannot see is with something this worker never did: the default base port is
50051, inside Linux's default ephemeral range of 32768 to 60999, so the kernel
hands ports in this range to outbound connections and to anything that binds
port 0. A backend handed one of those dies on bind, and the frontend sees a
backend that will not start.
Every candidate is now probed by binding the exact address the backend will
listen on, in all four allocation branches: the key's own port, the free pool,
a grown port and a stolen one. Probing the free pool matters as much as
probing a grown port, because a port this worker released is exactly as
available to the kernel as one it never used.
A candidate that fails the probe is quarantined rather than blacklisted, since
whatever holds it is usually an ephemeral connection that gives it back, and
its affinity claim is dropped so an unbindable port does not stay reserved for
the key that last held it. Exhaustion now says how many candidates were
skipped, which is what tells an operator "something else is in my range" from
"my range is too narrow".
This does not remove the race and cannot: between the probe and the child's
bind the kernel can still give the port away. It removes the far larger window
in which the allocator hands out a port the kernel gave away minutes ago,
which was the whole of the observed one-in-three harness flake. The e2e
harness comment that recorded the missing check is corrected, and the docs say
how to move the range out of the ephemeral one entirely.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Skills and RAG collections had no cross-replica invalidation, and the two
builders that would have published one were deleted earlier in this branch
because nothing called them. Wiring one now would be wrong, not merely late.
Both features are derived entirely from the frontend's own state directory.
A skills.Service indexes <state dir>/skills, a collections backend enumerates
<state dir>/collections and holds one handle per collection it found there,
and no replica reads or writes another replica's copy of either. In
distributed mode PostgreSQL carries a skill's NAME and description in
skills_metadata, and nothing else: Get, Search, Export and the resource verbs
all read local files. So a peer told to drop a cache entry would rebuild it
from a directory that does not hold the change. For a postgres-engine
collection it would be worse than a no-op, since re-deriving one on a replica
with no local index file yields a collection that answers with an empty file
list against a populated vector store. What is missing is shared storage, not
a broadcast.
Recorded rather than left silent: the two cache fields say why nothing
invalidates them, a distributed frontend logs the limitation once at startup,
and the docs name the two deployments that avoid it. The new spec pins the
premise, so a change that moved either directory onto storage every replica
mounts reddens and the decision gets taken again.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
GET /api/cluster/peer authenticated with the deployment's shared
registration token and took the dialling replica's id from ?id= on trust.
Every worker holds that token, so anything holding it could open a peer
link as any replica: relay through it to every worker tunnel that replica
owns, displace a real replica's inbound link by declaring its id, and
point the roughly 31 GiB per-session receive window at one replica.
Validating the id against the instances table does not fix this, because
the attack declares a real replica's id. So the route now checks two
credentials and needs both. The shared token still says the dialler
belongs to this deployment; a new per-replica credential says which
replica it is.
The credential follows the per-node worker credential rather than
inventing a second mechanism: crypto/rand.Text, stored only as a hex
SHA-256, compared in constant time, with no fallback to the shared token.
It differs in the stronger direction. A worker's credential is minted by
the frontend and handed over once; a replica writes its own instances
row, so it mints its own secret, publishes only the hash in the same
statement that publishes its address, and never sends the plaintext
anywhere but the peer dial.
A peer that presents no credential is refused, not waved through. An old
replica and an attacker holding the shared token send the same request,
so accepting the first accepts the second; there is no safe downgrade
here, only a quiet one. The refusal is made loud instead, on both sides,
naming the upgrade rather than the network. On the documented
frontend-first order a new replica still dials an old one; an old replica
cannot dial a new one, which costs relayed requests that land on a
not-yet-restarted replica and surfaces as no route, never as absence.
A rejected peer gets its own sentinel, ErrPeerRejected, whose unwrap
chain carries ErrPeerUnreachable as well and no absence sentinel at all.
Keeping the older sentinel means no existing consumer changes behaviour;
the cause stays out of the chain, so absence cannot escape through it and
nothing can read an authorization failure as a worker that went away.
One consequence beyond the fix: a replica with no advertised address has
no instances row, so it now cannot dial out either. It was already
unreachable inward. The startup error and the docs say so.
Registry.Register, NewMembership, NewPeerPool, PeerHandler and
RegisterClusterRoutes all gained required arguments, so the identity
cannot be dropped without a compile failure.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Removing the broker left one thing carrying every broadcast family in the
product: PostgreSQL LISTEN/NOTIFY, in core/services/pgbus. It is covered
thoroughly in process by test-e2e-distributed, and it was covered nowhere at
all by real binaries: grepping the six Cluster spec files for pgbus,
bus_messages, LISTEN and NOTIFY returned zero hits. Registration, model staging
over the tunnel and inference through both the owner and the relay paths were
already proven by real processes; the carrier that now carries everything else
was not, so a deployment whose replicas each published to themselves and heard
nobody would have left every suite green.
Two specs, both on two frontends and no workers against one PostgreSQL,
publishing at frontend 0 and reading at frontend 1.
1. A gallery operation admitted at one replica, read out of the other, with the
queued state observed before the terminal one.
2. A broadcast of about 9.3 kilobytes, which PostgreSQL refuses as a
notification payload, making the round trip byte for byte through the
bus_messages spill table.
The family is a gallery operation for one property nothing else on this carrier
has: the answer a peer gives is held in memory ALONE. GET /models/jobs/<id>
reads galleryop's statuses map, which on a peer is filled by the
gallery.*.progress subscriber and by nothing else, because the only other
filler, Hydrate, runs once at startup and every operation here is created long
afterwards. Every other family has a durable table behind it that a peer would
converge through anyway, and a spec on one of those cannot separate "the
broadcast arrived" from "the row was read".
That is then made checkable rather than argued. The gallery_operations row is
written when the gallery worker DEQUEUES an operation, so an operation still
waiting in the queue has NO row, and both specs assert zero rows while the peer
is already answering with the operation's own bytes. Both also read the
instances table and require the reading replica to be a different live instance
from the publishing one, so "the other replica" cannot decay into a spelling of
"this replica".
Holding the queue is what cluster.Options.Galleries is for. The gallery worker
runs one operation at a time on an unbuffered channel, so an install parked
inside a gated index fetch parks everything behind it; without that the
admission broadcast and the terminal one are separated by two database round
trips and no HTTP poller could see between them. The option also turns the
startup estimate warmer off, because a second fetcher filling the process-wide
index cache would leave the operation never blocking and the spec passing on an
ordering nothing enforced.
The spill spec is written against a failure this branch has shipped three
times: a size-limit spec that cannot fail. The oversized body is an ordinary
element name that the real consumer decodes and surfaces, so it is not a body
the decoder would have refused at any size. The size is ABSOLUTE at 9000 bytes
rather than derived from the cap, and a one-byte control operation in the same
run is required to leave no spill row, so moving the 8000-byte cap in either
direction reddens the spec. pgbus.FitsInline, which shares its encoder and its
comparison with Publish, is asked about both payloads and must answer
differently. The spilled row is then decoded and its element name compared byte
for byte against what frontend 1 answers.
The terminal assertion in spec 1 does not re-check the element name: a terminal
status does not carry one, because updateError in galleryop.Start builds a
fresh OpStatus holding only the error. It asserts the two fields that status
does carry, in the relation that one place writes them.
Attacks run, each alone, each reverted, each behaving as predicted. Neutering
the pg_notify in pgbus.Publish so every replica knows only what it did itself
reddens both specs at frontend 1, which answers 500 for an operation it was
never told about; the bus_messages row assertion still passes under it, which
is right, since the row is written before the notification. Releasing the queue
gate reddens spec 1 at the gallery_operations count, because the operation is
dequeued and the row appears. Shrinking the oversized name to 100 bytes reddens
spec 2 at FitsInline; inverting that guard so the run reaches the row check
reddens it there instead, with no bus_messages row written, which is what makes
the row a statement about size.
test-e2e-cluster is 26 specs in 933.8 seconds of Ginkgo time, 15m37s wall. The
two additions cost 7.0 seconds together, 5.0s and 2.0s: they start no workers,
so they pay for no registration, and what they wait on is a broadcast rather
than a threshold. test-e2e-distributed is unchanged at 223 plus 8 specs, 130.6
seconds. The budget comment and .agents/building-and-testing.md move from 24
specs at 897 to 907 seconds to 26 at 933.8.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Qwen3-style chat templates append the opening <think> tag to the *prompt*
when thinking is enabled. The model therefore never generates it and emits
only the reasoning text plus the closing </think>.
sglang's ReasoningParser keys off the opening tag:
in_reasoning = self._in_reasoning or self.think_start_token in text
if not in_reasoning:
return StreamingParseResult(normal_text=text)
so with such a template the entire completion — reasoning and answer, the
raw </think> in between — is returned as content and reasoning_content
stays empty, no matter how reasoning_parser is configured.
sglang's own OpenAI server handles this via
force_reasoning = (self.template_manager.force_reasoning
or self._get_reasoning_from_request(request))
This backend has no template manager, so derive the same signal from the
rendered prompt: if it ends with the detector's think_start_token, the tag
was prefilled and the parser is constructed with force_reasoning=True.
Structured decoding is the exception, and it matters: a grammar applies
from the first token, so the model cannot emit the closing tag even though
the template opened the block. The whole completion is schema output and
belongs in content — forcing there files it as reasoning and returns an
empty answer. Measured against a JSON-schema code audit: 10107 characters
of "reasoning", zero content. sglang's own server keeps the two apart for
the same reason; its grammar backend owns the reasoning prefix when a
reasoning parser is configured.
force_reasoning is only passed when it is meant to be True, so detector
defaults (DeepSeek-R1 already defaults to True) are untouched, and a
prompt without a prefilled tag behaves exactly as before — which matters,
because forcing unconditionally makes an answer generated with thinking
off disappear into reasoning_content.
The construction is factored into _new_reasoning_parser() so the streaming
and non-streaming paths, which previously built the parser separately,
cannot drift apart.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
With `template.use_tokenizer_template: true` the sglang and vllm backends
render the prompt themselves via `tokenizer.apply_chat_template()`, and they
hand it plain string content. A chat template only emits the model's own media
tokens when the content is a list of parts, so the rendered prompt carries no
`<|vision_start|><|image_pad|><|vision_end|>`. The pixels do reach the engine
(`image_data` / `multi_modal_data`), but both engines locate them by scanning
the prompt for that token, so they are discarded silently: HTTP 200, no
warning, and the model answers as if no image had been attached.
Add `attach_media_parts()` to the shared `python_utils` helper and call it in
both backends: the last user turn is rebuilt as
`[{"type": "image"} * n, {"type": "video"} * n, {"type": "text", ...}]` before
templating, which makes the template emit the placeholders. The pixels keep
travelling out of band exactly as before.
Text-only requests are untouched - with no media the helper returns None and
the original string-content path runs unchanged. If a template cannot iterate
content parts (a text-only model), the parts render is caught and the request
falls back to the previous string-content prompt instead of failing.
Signed-off-by: Tai An <antai12232931@outlook.com>
vLLM's engine-based reasoning parsers derive their initial state from the
chat template kwargs. Qwen3Parser:
chat_kwargs = kwargs.get("chat_template_kwargs", {}) or {}
self.thinking_enabled = chat_kwargs.get("enable_thinking", True)
Constructed as ReasoningParser(tokenizer) the flag defaults to True, so the
parser starts in the REASONING state. A completion produced with thinking
disabled contains no tags at all, and every reasoning parser shape then
reports the whole answer as reasoning:
- engine-based parsers classify it by initial state;
- BaseThinkingReasoningParser hits its documented "may not generate start
token" fallback and returns (model_output, None).
Either way `content = c if c is not None else generated_text` turns that
into a duplicate: a Qwen3 model answering "391" with thinking off comes back
as reasoning_content="391" AND content="391".
Measured against Qwen3.5-MoE on vLLM 0.28, non-streaming:
before thinking on reasoning=202 content="391"
thinking off reasoning="391" content="391" <- duplicated
after thinking on reasoning=192 content="391"
thinking off reasoning="" content="391"
Forward the kwargs the prompt was rendered with, which is what vLLM's own
OpenAI server does; parsers that do not accept the argument keep the plain
constructor.
_split_reasoning() covers the older parser shape, which has no initial state
to set. It only reclassifies when the parser exposes a start/end token pair
and neither the completion nor the prompt ever opened a reasoning block.
Truncated reasoning (block open, end token never arrived) stays reasoning,
and parsers without that token pair are left untouched.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
#11772 exempted Temperature from the zero-filter in both backend adapters,
because proto3 has no field presence and an explicit 0 is indistinguishable
from "unset". Seed has exactly the same property and is still filtered:
if proto_field != "Temperature" and value in (None, 0, 0.0, [], False, ""):
continue
A caller pinning `"seed": 0` for a reproducible run therefore gets a random
seed instead, with no error and no log line — the one case where the failure
is invisible precisely because the request looked deliberate.
Both adapters now share a named tuple of fields whose zero is meaningful, so
the next one is added in one place rather than as a second special case.
Deliberately left filtered: top_k, top_p, min_p and the penalties. Their zero
is not a value a caller means — sglang disables top_k with -1, not 0, so
forwarding 0 there would turn a default into an invalid argument.
Verified on the sglang backend (Qwen3.5-MoE, arm64): with the temperature fix
alone, two identical requests at temperature 0 are byte-identical, but pinning
seed 0 has no effect until this change.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
A quantization job that runs with no client on its progress stream stays
"queued" forever, in the API and in state.json, while the finished artifact
sits on disk. state.json was written once by StartJob, and the only code that
advanced a job afterwards lived inside the stream callback of StreamProgress,
so job state depended on somebody watching it.
The backend's progress stream cannot simply gain a second reader: each job owns
one queue.Queue and QuantizationProgress pops from it, so two consumers split
the updates rather than both seeing them. The stream has to be opened exactly
once per job.
StartJob now starts watchProgress on the application context (the request
context is done as soon as the handler returns). That goroutine is the single
reader: it applies each update to the job -- in the cross-replica store and in
state.json, terminal statuses still winning over late updates -- and republishes
it in-process. StreamProgress becomes a pure reader over that fan-out and no
longer loads a backend or opens a stream. A client attaching to a job that has
already finished, including one hydrated from disk after a restart, gets a final
event built from the stored job instead of blocking.
Two paths used to end a client's stream by breaking the gRPC connection and now
release it explicitly: StopJob kills the backend, so it publishes the stopped
event itself; and a stream that ends without a terminal update means the backend
is gone, so the job is recorded as failed rather than left running forever.
Signed-off-by: Tai An <antai12232931@outlook.com>
The Intel backend installs PyTorch XPU wheels, but Qwen ASR only
checked CUDA and MPS. Every Intel model therefore loaded on the CPU.
Select XPU when available and place the model on xpu:0. Keep the
existing CUDA, MPS, and CPU placement behavior.
Assisted-by: Codex:GPT-5 [apply_patch] [gh]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
WhisperX silently returned a plain transcript when diarization lacked
the Hugging Face token required to load pyannote. Reject that request
clearly so callers do not mistake missing speaker labels for a
successful diarization.
Convert WhisperX seconds to the nanosecond duration unit used by the
transcription API.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
importNpmLock turns the same-version hono override into a file: tarball
that conflicts with the direct dependency (EOVERRIDE). Pass
--legacy-peer-deps so the flake build can proceed without
touching package.json (open #11633).
Fixes#11804
Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
LLM-jp 4 provides a recent Japanese and English reasoning model on the supported Llama architecture. Add its official Q4 and BF16 GGUF builds so hosts can select the fidelity that fits.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add Q4_K_XL and Q8_K_XL llama.cpp builds with the shared vision projector. Enable the preserved MTP head for speculative decoding.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the text-only Q4_K_M build for private red-team, blue-team, and security operations workloads. Configure the supported Gemma 4 model for llama.cpp with its verified Hugging Face checksum.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add Q4_K_M and Q8_0 llama.cpp builds plus the official F16 GGUF for Mixedbread mxbai-embed-large-v1.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The new importance-matrix builds reduce Laguna S 2.1 from the existing 96 GB default to 73.9 GB or 54.4 GB. Add both as selectable llama.cpp variants.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
DFM Mimir is a new permissively licensed Danish and English HRM-Text model. LocalAI already ships a vLLM version with native support for its architecture.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the artifact-backed VoxCPM2 model for multilingual speech synthesis, voice design, and controllable voice cloning.
Assisted-by: Codex:gpt-5.6 [Codex]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the Q4_K_M and Q8_0 GGUF builds for a compact Qwen3.5-based reasoning and tool-use distillation. The variant pair lets LocalAI choose higher fidelity when the host has enough memory while keeping the Q4 build as the installable fallback.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a compact NVFP4 build with its embedded MTP head and vision
projector. This gives Blackwell-capable llama.cpp hosts a speculative
option alongside the existing Q4 and Q8 builds.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the original-dtype GGUF for LocalAI\x27s audio.cpp backend. The entry exposes multilingual voice cloning and records the model license.\n\nAssisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the compact LightOnOCR model to the OCR section with Q8_0 and F16 builds. Each build uses its matching vision projector.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the official F16 GGUF build of the compact Apache-2.0 Qwen3-family chat model with its native 2K context limit.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
- api-errors.md documented LOCALAI_SUBTLEKEY_COMPARISON (missing the
KEY underscore); the code defines LOCALAI_SUBTLE_KEY_COMPARISON, so
the documented variable silently did nothing
- cli-reference.md documented a --csrf flag / $LOCALAI_CSRF env that
do not exist, with inverted semantics; the actual flag is
--disable-csrf (LOCALAI_DISABLE_CSRF), 'Disable CSRF middleware
(enabled by default)'
- the PR template's 'Signed commits' anchor pointed at a CONTRIBUTING
section that does not exist; repointed at the Commit messages
section
- the longcat-video backend README linked a docs page that was never
committed; replaced the dead link with plain text
- formal-verification/README.md used ../../../ for five in-repo
packages (escaping the repo root); fixed to ../
Tasks 1 to 17 are proven by unit and integration specs and by two e2e passes
taken mid-flight. This is the pass that boots the real binaries with every
carrier in place and none of the old one, and it does so on the topology the
feature was built for rather than on the one-worker shape the rest of the
cluster suite uses.
Two frontends and two workers is the configuration that matters. With each
worker's tunnel landing on a different replica, the owner path and the relay
path are live at the same instant against one roster, one scheduler and one
health monitor, so a routing mistake has somewhere to show up instead of
hiding. It is also the only shape in which "killing a replica re-homes only ITS
worker" can be stated at all.
Three scenarios, all 2x2:
1. Both workers served from both replicas. No broker as a property of the
ARTIFACT (debug/buildinfo reports no github.com/nats-io module, with the
module count asserted non-zero so a stripped binary cannot pass vacuously),
no broker in either worker's live /proc environment, and no advertised
address on either worker. One completion over the owner path and one over
the relay, plus the mirror image through the other replica, plus four
control-plane listings covering both paths for both workers.
2. The replica owning worker 0's tunnel is killed with that tunnel blocked.
Leg 1 asserts nothing and only waits for the killed instance to leave the
live set, because before that it still reads as a live owner and the
scenario is not yet about absence. Leg 2 then holds a window inside the
reconnect grace requiring that nothing acted on the absence. Leg 3 requires
the re-home and inference again. Worker 1 keeps serving throughout.
3. The suite-wide negative control. Both tunnel dials refused while
registration and heartbeats flow, both workers refused at both replicas
naming the routing fact and not a departure, nothing reaped and both
heartbeats fresh. Then ONE tunnel is restored and exactly one worker
recovers while the other stays refused.
Which worker served is read back from node_models rather than assumed: the two
models are pinned to one worker each through PUT /api/nodes/:id/labels and
POST /api/nodes/scheduling, and every assertion requires the model to be on the
expected node AND absent from the other. That the relay hit a non-owner is read
from the production Owner query before the request and re-read after it.
Attacks run, each alone, each reverted, each behaving as predicted:
hand a worker a broker URL reddens scenario 1's environment leg; point the
module check at gorm.io reddens its artifact leg; start one worker instead of
two reddens all three at the topology guard; delete the relay in
WorkerDialer.Dial reddens scenario 1 on exactly the request sent to the
non-owner while 2 and 3 stay green; a one-nanosecond reconnect grace reddens
scenario 2's leg 2 on the demotion while 1 and 3 stay green; lifting both
blocks at scenario 3's differential reddens its "still unreachable" half. The
brief's "restore the NatsURL validation" attack cannot be applied:
DistributedConfig has no such field left to validate.
Label-orphan arithmetic, counting non-skipped It nodes from --dry-run:
all 256, dist 231, cluster 24, vllm 1, and 231 + 24 + 1 = 256, so no spec is
orphaned by the label filters.
Three test-e2e-cluster runs: 897.0s, 897.8s and 906.6s of Ginkgo time, 24 specs,
15 minutes wall. The only failure across the three was a pre-existing spec dying
at cluster.Start with frontend-1 exiting status 2, which passed in the other two
and is reported as a port-allocation flake rather than a regression.
test-e2e-distributed is 223 plus 8 specs in 131.7s. The budget comment and
.agents/building-and-testing.md move from 21 specs at 800 to 830 seconds to 24
specs at 897 to 907.
The harness gains ProcessEnviron, which reads /proc for any of the three process
families; WorkerEnviron and FrontendEnviron become wrappers rather than being
deleted, so the specs that call them are not re-aimed for a rename.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
- middleware.md: the 'default detector' link used #instance-wide-defaults;
the heading is 'Instance-wide default detector'
- the advanced/reference landing pages linked an ../installation/
directory that does not exist in docs/content; dropped the dead
bullets (deployment content lives under getting-started)
* docs(llama-cpp): clarify multimodal speculative decoding
Update the speculative decoding guidance now that modern llama.cpp backends can combine mmproj-based vision with speculative decoding, including MTP. Document compatibility checks, draft acceptance statistics, VRAM tradeoffs, and a combined configuration example.
Assisted-by: Codex:GPT-5.6-Sol [gh] [OpenStack] [Docker]
Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com>
* docs(llama-cpp): clarify multimodal MTP references
Distinguish the upstream change that removed the general multimodal speculative restriction from the later change that added MTP with explicit vision compatibility.
Assisted-by: Codex:GPT-5.6-Sol [gh] [Docker]
Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com>
---------
Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com>
Distributed mode has not dialled a message broker since the control plane
moved onto the workers' own outward tunnels and every fan-out family moved
onto PostgreSQL LISTEN/NOTIFY. What was left was the dependency itself, and
the code that existed only to feed it.
Dropped from go.mod: nats-io/jwt/v2, nats-io/nats.go, nats-io/nkeys,
nats-io/nuid and testcontainers-go/modules/nats, along with the fourteen
indirect requires that only the NATS testcontainer pulled in. go.sum carries
no nats line either, so the removal is not the partial kind where the require
goes and the checksum stays.
Deleted with them: pkg/natsauth in full, the broker client's remaining
options and TLS files, the per-node JWT minting on both the register and the
approve path, and the natsauth.Config parameter threaded through the node
routes. The credential manager is renamed and stripped rather than deleted,
because it still holds the tunnel token that every re-registration rotates.
The bus flags stay accepted and ignored, and are now hidden, on every command
that had them, so an existing unit file, compose file or Helm values file
still starts on the day of the upgrade. What is not kept is the validation
that REQUIRED one: a distributed frontend started with no bus URL is no
longer fatal. The TLS paths lose type:"existingfile" deliberately, so a
certificate deleted along with the broker cannot fail a startup.
One operator-visible behaviour change: --nats-require-auth no longer makes an
agent worker wait through admin approval. Ask for that wait with
--distributed-require-auth, which already implied it. It is documented in the
migration section and pinned from both sides.
A deployment now needs PostgreSQL and the frontends' own HTTP listener, and
nothing else.
coverage-baseline.txt moves from 54.2 to 62.0.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Every carrier had already moved and no process opened a bus connection, but
the surface an operator reads still described a deployment with a broker in
it: a compose service, a 220-line credential-generation script, two CI steps
pulling a container nothing started, two flag tables offering --nats-url, an
architecture diagram with a NATS box wired to the workers, a join-command
generator in the Nodes page that emitted --nats-url for agent workers, and a
test suite that stood a NATS server up for specs that no longer used it.
That is the one way this programme could still fail invisibly. Every test
passes, every binary works, and every production deployment goes on running
and paying for infrastructure that carries nothing.
Nothing in this repository starts a NATS server any more. The compose file is
four services, the docs say to shut the broker down and what to keep, and the
e2e suite runs on one PostgreSQL container.
The three LOCALAI_NATS_*_TIMEOUT env vars are KEPT, and are now documented
twice as being kept. They were never broker settings: each names a control-RPC
budget the frontend applies to a worker, still read and still enforced. They
carry the prefix only because they arrived with the bus, and renaming them
would break every existing deployment for cosmetics.
The agent worker's join command was the last surface still emitting the flag,
two tasks after the agent worker stopped dialling. The Playwright spec that
covered it asserted the opposite of what is now true, so it is inverted rather
than deleted, and it reads the rendered command string rather than the
component's variables: the variables are what the fix removes, so a spec
reading them would have stopped compiling instead of failing, and a compile
error is not evidence about what an operator is shown.
nats_jwt_test.go and its helpers are deleted. They pinned a real server
ENFORCING the minted permissions. The CONTENT of those allow lists is still
pinned, untouched, by pkg/natsauth's own suites, including the spec that
refuses to let the agent lists go empty, since an empty allow list in NATS
means unrestricted. The enforcement half is retired rather than moved:
enforcement is a property of a connection, and nothing opens one.
The suite's own NATS container goes with them, which the brief left for the
next task. Removing the pre-pull while BeforeSuite still ran the image would
have defeated the step rather than cleaned it up, and this change removes the
last reader of TestInfra.NC. agent_native_executor_test.go and
mcp_ci_job_test.go are moved onto infra.Bus() instead of deleted: they were
the last two specs building a bridge and a dispatcher on a client nobody uses,
which is exactly the drift TestInfra.Bus's own comment warns about.
cluster.Options.NatsURL is now fed a deliberately dead address rather than a
live container's. Frontends and agent workers still receive LOCALAI_NATS_URL,
because that is the coverage for the promise that an existing command line
still starts; sourcing it from a running server would have let a regression
that actually dialled it pass. The control in cluster_control_test.go keeps
its assertion and loses its explanation, which claimed the deployment had a
bus and no longer could.
One latent spec race surfaced and is fixed: the background-run spec waited for
a COUNT of events and then read a snapshot for the terminal status, which is
the last event of a run and therefore always arrives after the count is met.
Its immediate twin had already been fixed this way. Nothing in production
changed.
pkg/natsauth keeps its files. It is reachable from production only through the
natsauth.Config parameter thread, and that thread is the next task's.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Task 4 gave agent workers tunnels and deliberately left the NodeType skip in
HealthMonitor.tunnelDeparted, with a spec asserting that an agent node whose
presence reader answers PresenceGone is NOT marked unhealthy. That spec was
scaffolding. It was true while an agent worker took its jobs and its verbs over
the message bus: a departure row for one said nothing about whether it could
work, and an early bug in the new tunnel client could otherwise have demoted a
fleet of healthy agent workers.
There is no bus. An agent worker is reachable through its tunnel and through
nothing else, so a departed agent tunnel means exactly what a departed backend
tunnel means: no live replica holds it, the departure has outlived the reconnect
grace, and that is a routing fact the scheduler and a reaper may act on. The
skip would now hide the only symptom an unreachable agent worker has. This is
the deliberate removal Task 4's M6 predicted, and task-4-report.md is where that
mutation already stands recorded red against the spec this commit deletes.
The skip existed at ONE site. router_liveness.go has none: its candidates come
from queries that already filter node_type = 'backend'. The two skips in
managers_distributed.go stay, because an agent worker still runs no backend
processes, so it has no backend to list and no backend op to apply.
Two node types can depart now, which is why the second half exists. Before this,
one type could depart and every per-node cache a departure left stale was
dropped from wherever its owner happened to notice, so a reader could not tell
which caches a demotion invalidated by reading the demotion path. Departure gets
ONE notification point. DepartureNotifier is edge triggered, because the monitor
runs on a ticker and a departed node stays departed; its subscribers are NAMED,
because what has to be caught is a forgotten cache and a count can say only that
one of four is missing; and NewHealthMonitor takes it as a required positional
argument, so a caller that does not pass one fails to compile.
Four caches subscribe: prefix-cache affinity in every model, probe freshness at
every address, in-flight staging operations, and the per-node breakdown of every
open gallery operation. The prefix-cache one is registered only when
prefix-cache routing is enabled, so --distributed-prefix-cache=false stays a
true no-op. The notification carries the node's name as well as its id, because
the staging tracker keys on the name and the other two key on the id, and a
subscriber should not have to read the registry from inside an eviction hook.
A departure notification is an act on absence, so it fires only on the routing
fact. A tunnel lost inside the grace, a worker that never dialled, a presence
query that failed and a stale heartbeat all announce nothing, asserted per node
type. The stale-heartbeat branch is excluded on purpose: it already marks the
node offline, which deletes its rows and runs the registry's replica-removed
hooks, so firing there too would double-evict and make the notification mean two
different things at its subscribers.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
agent.<name>.cancel was the last family on a message bus, and the only
reason an agent worker dialled one. Its subscriber is the worker running
the execution, and a worker has no database, so the family could not move
to the PostgreSQL fan-out carrier: a cancel published there would reach no
worker while reporting that it had been sent.
It is a control verb now. An agent worker mounts workerctl.PathAgentCancel
on the loopback control plane behind its tunnel and applies the cancel to
the same registry the executor registers a run on. The frontend issues it
through nodes.AgentControlClient.CancelAgentRun.
That call is a FAN-OUT and not a pick, because nothing records which worker
holds a given execution: the claim row names the claiming replica, and it
is deleted when the run ends. Every agent worker a live replica can reach
is asked over its own tunnel, relayed by the peer mesh when a peer holds
it, and each worker answers only for itself.
The answers stay apart, which is why this family was held back. A cancel a
worker made is nil. A cancel some worker could not be asked is
ErrAgentCancelUndelivered, which is neither a refusal nor a missing run. A
cancel every reachable worker declined to own is ErrAgentRunNotOnAnyWorker.
A deployment with no agent worker is ErrNoAgentWorker. Neither new sentinel
wraps ErrWorkerUnroutable and neither is a worker answer, so nothing is
reaped, demoted or evicted because of a cancel.
A worker in the ABSENT CONNECTION condition, one whose tunnel was lost
inside the reconnect grace, counts as undelivered. It is not retried in the
call and not queued: a retry would spend a budget the caller did not
choose, and a queue would need durable state whose only consumer is a run
whose control stream went with the tunnel. A worker whose departure has
outlived the grace is the one routing fact a caller may act on and is
excluded, or a single retired agent node would make every cancel
undelivered for ever.
The fan-out reads a different node set from the pick. A draining worker
takes no new work but is still finishing what it holds, so it is offered
the cancel; a pending one is refused by the tunnel route on every dial and
is not.
With that, nothing in LocalAI connects to NATS. The agent worker's dial,
its credential ladder and its refresh loop are gone, and so is the
frontend's cancel carrier. LOCALAI_NATS_URL is accepted and ignored
everywhere, and distributed mode no longer requires it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Nothing in the tree publishes, subscribes, queue-subscribes or requests through
the MessagingClient interface any more, so it is deleted rather than shrunk to
Broadcaster: two exported names for one method set in one package is an
invitation for the next author to pick whichever the surrounding file already
imported.
$ grep -rn 'messaging\.MessagingClient' --include='*.go' .
core/services/syncstate/syncstate.go:54: // It is messaging.Broadcaster rather than messaging.MessagingClient because
(one hit, a comment; no live referent. The naive grep in the plan also matches
prose and the local test type names fakeMessagingClient and
countingMessagingClient, so it can never be empty.)
*messaging.Client is shrunk to exactly Broadcaster plus its own lifecycle.
QueueSubscribe, QueueSubscribeReply, SubscribeReply, Request, Conn and the
package helpers QueueSubscribeJSON and RequestJSON go with it; none had a
production caller. Deleting the methods rather than only the call sites is what
makes putting a family back on this carrier a build error instead of a line that
compiles, publishes successfully, and is delivered onto a carrier the deployment
is being taken off. Conn is in that list because while it existed every other
name was one c.Conn().X() away; the flush-and-verdict that its real consumers
needed is now ConfirmRoundTrip, which keeps the NATS JWT permission specs armed.
The client, its options and its TLS plumbing are NOT deleted, and both processes
stay on the bus. agent.<name>.cancel is the one fan-out family that could not
move: its only subscriber is the agent worker, which has no database and cannot
join the PostgreSQL carrier at all, so a cancel published there would reach no
worker and be reported as sent. The frontend passes the client to
newFanoutBridges as its cancelCarrier and the worker subscribes on it, so
--nats-url stays required on agent-worker. Both go with the tunnel cancel verb.
The struct field is renamed Nats -> CancelCarrier to say what it is for, and
agentpool loses the messaging.Publisher it held only to be non-nil: it never
published on it, and it was gating whether a frontend runs agents distributed or
in an in-process pool. Retiring the bus would have flipped every replica back to
the in-process pool silently. The gate now reads the agent store, which is the
dependency the mode actually requires.
Also deletes four subject builders with no production publisher
(SubjectFineTuneProgress, SubjectFineTuneCancel, SubjectCacheInvalidateSkills,
SubjectCacheInvalidateCollection), the queue and request/reply halves of the
shared test double, and the e2e specs that were their only callers. Every
surviving subject is now pinned to its exact literal, because a subject is a
cross-version wire format and a rename that looks internal stops half a fleet
hearing the other half.
Docs: distributed-mode.md and cli-reference.md no longer claim NATS carries the
agent-worker job subjects, the frontend's cross-replica events, or an agent
worker's real work.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Gallery progress and cancel, the operation cache's start and end, the model
and backend cache invalidations, staging progress, and the prefix cache's
observations and invalidations now travel on the LISTEN/NOTIFY carrier. No
subject is published or subscribed on messaging.Client anywhere in the tree,
which is what makes retiring that package a deletion rather than a migration:
$ grep -rn 'natsClient\.Publish\|nats\.Publish\|\.Nats\.Publish\|QueueSubscribe\|SubscribeReply\|\.Request(' \
--include='*.go' core/ pkg/ | grep -v _test \
| grep -v 'c\.Request()\|ctx\.Request()\|Request()\.Context' \
| grep -v 'core/services/testutil/fakebus.go'
core/services/messaging/client.go:168,170,172,227,234,236,250,252,254,268,269,287
core/services/messaging/interfaces.go:21,22,23
Every remaining hit is inside core/services/messaging itself. The production
reads of the NATS client are now three, all of them the documented agent-worker
exception: Close on shutdown, the agent pool's publisher, and the agent-cancel
carrier passed to newFanoutBridges.
Prefix-cache observations publish like every other family rather than through a
method that refuses a message too large for a notification. The plan proposed
such a refusal on the reasoning that a long prompt makes a chain of thousands of
entries; ExtractChain caps a chain at Config.MaxDepth blocks, MaxDepth is a
constant with no operator knob, and the chain reaching Sync.Observe has one
source, the router's own extraction hook. A worst-case observation is a few
kilobytes against an 8000-byte cap, so the hot-path spill the refusal was
designed to avoid cannot occur, and shipping it would have added the programme's
only deliberate message drop to guard a condition that cannot arise. pgbus gains
FitsInline instead, a predicate that shares one size decision with Publish and
decides nothing, and core/application refuses at startup to wire a prefix cache
whose configured depth would put every observation over the cap.
The carrier choice is no longer stated at four sites. StagingTracker.SetPublisher
and SubscribeBroadcasts become one SetBroadcaster, so a tracker that publishes
where its peers are not listening cannot be spelled; prefixcache.Sync gains
SubscribeBroadcasts, which reads the carrier it publishes on; and the gallery
service and the operation cache are wired by methods on DistributedServices that
name no carrier at all, so the NATS client beside it cannot be handed over.
OpCache.SetMessagingClient and GalleryService.SetNATSClient are renamed to
SetBroadcaster so a missed call site fails to compile.
Two pre-existing defects that the two-real-carrier specs surfaced are fixed. A
progress tick published before a cancel and delivered after it cleared Cancelled
and left the operation reading as still running on that replica; mergeStatus now
drops a stale tick rather than merging it. GetStatus and GetAllStatus handed out
the stored OpStatus pointer while the broadcast subscribers mutated it in place,
so an /api/operations response could be marshalled mid-write; both now copy.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Five of the six families whose subscriber is an open HTTP response rather
than a process-lifetime cache now travel on pgbus: jobs.<id>.progress,
jobs.<id>.result, jobs.<id>.cancel, agent.<name>.events.<user> and
responses.<id>.cancel. Both ends of each move together, so there is no
state where a publisher is on one carrier and its subscriber on the other.
agent.<name>.cancel does NOT move, and the plan was wrong about why. Its
only subscriber in the tree is the agent worker, which has no database and
so cannot join the PostgreSQL carrier at all. Publishing that cancel on
pgbus would have lost every cancel of a worker-run agent while returning
nil, which reports a cancel that reached nobody as a cancel that was sent.
EventBridge now names its cancel carrier separately, a frontend replica
sets it to the carrier the worker reads, and it stays there until a cancel
rides the worker's tunnel like every other verb addressed to a worker.
The carrier drops at 256 rather than blocking, which is not safe on its own
for a result: a lost result has no successor message. It is not the only
path. The claiming replica persists the terminal line before it releases
the claim, and an open progress stream re-reads the job row once after
subscribing and then periodically, so a dropped terminal broadcast costs
promptness and never the answer.
Both per-request subscriptions close in a defer instead of on one return
path, and pgbus grows Subscribers() so the leak they would otherwise cause
can be asserted. It has no other symptom: only the first subscriber of a
channel issues a LISTEN, so a leaked filter just adds one closure per
notification for every stream the replica has ever served. Subscribe now
issues its LISTEN before it registers, which makes that count a readiness
signal rather than a figure to compare against itself.
Two rules that were stated at several sites and pinned at none are now one
each. The re-broadcaster is built beside the dispatcher and the bridge and
handed to the dispatch loop, so no line is left that can point it at a
carrier nobody subscribes to while every spec stays green. The set of
statuses a job never leaves is one exported set that the SSE bridge and the
store both read. The last hand-written subject filter in production code
became messaging.SubjectAgentEventsWildcard.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
syncstate.Config held one carrier field typed as the NATS client, so a
pgbus.Bus could not be handed to a SyncedMap at all: it satisfies
messaging.Broadcaster and not MessagingClient. The durable re-hydration
path built for the responses map therefore had a NATS-only consumer and
nothing in the build said so.
The field becomes Bus messaging.Broadcaster, SubscribeJSON moves to its
own file and relaxes its parameter to Broadcaster, and the four adopters
fan out over PostgreSQL LISTEN/NOTIFY: fine-tune jobs, quantization jobs,
agent tasks with their per-tenant children, and Open Responses metadata.
A new spec proves it on a real database, over two Bus instances on two
pinned listener connections: a Set and a Delete carry, a payload past the
8000-byte notification cap comes back byte identical through the spill
row, two families sharing one LISTEN channel stay separate, and a
terminated listener re-hydrates a row written while it was gone.
The five sites that each chose a carrier for an adopter are collapsed
into one DistributedServices.Broadcast() accessor. Five field reads were
five chances to leave one family on NATS with nothing failing, because
messaging.Client satisfies Broadcaster too. The accessor also refuses to
hand out a nil pgbus.Bus wrapped in a non-nil interface, which every
adopter would read as "broadcast" and dereference on the first Set.
SetTaskSyncNATS and SetJobSyncNATS are renamed to SetTaskSyncBus and
SetJobSyncBus so a missed wiring site fails to compile.
The response metadata table gains a retention of its own, defaulting to
24 hours. It inherited the Open Responses store TTL, which defaults to 0
meaning no expiration. Zero is defensible for a map that dies with the
process and is not for a table: the table grew for the life of the
deployment and a restarting replica re-hydrated every response the
cluster had ever created. A row that names its own expiry is still judged
on that column alone, and "this row is dead" now has one SQL spelling
that PurgeExpired deletes by and ListUnexpired is the negation of, so a
hydrate cannot resurrect what a sweep has already retired.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Every AgentJobService built its tasks SyncedMap with the name "agent.tasks",
and there is one service per user. So every tenant published on and subscribed
to the same subject, state.agent-tasks.delta, and SyncedMap.apply scopes
nothing: a task tenant A created was written into tenant B's in-memory map on
every replica, and ListTasks reads that map. Nothing repaired it short of a
process restart.
The subject now carries the tenant in a token of its own,
state.<name>.<tenant>.delta. Four tokens where the unscoped builder makes
three, deliberately: SubjectMatches compares token count before anything else,
so a tenant's subject and the cluster-wide one cannot cross-match, and neither
can two tenants. Putting the tenant inside the name token would not do that,
because the sanitizer folds '.' to '-' and the only filter that could then span
tenants is state.*.delta, which spans every other family too.
The rule is stated once. subscribeFilters calls publishSubject rather than
restating the subject, so a map cannot end up publishing scoped and subscribing
unscoped, which would leak exactly as before while every publish assertion
passed. The one case that decides on its own is the cluster-wide administrative
view: it hydrates from every tenant's rows, so it also takes the per-tenant
wildcard, or it would be stale the moment any tenant wrote. A tenant hydrates
from its own rows and applies only its own deltas. PerTenant defaults to false,
so finetune, quantization and the responses store keep the subject they have.
The second half of the same defect was the delete. taskStoreAdapter.Delete
called DeleteTask(id) and JobStore deleted by primary key with no user
predicate, reachable from DELETE /api/agent/tasks/:id, which takes the id off
the URL. A tenant who learned another tenant's task id destroyed that tenant's
row. The user id now travels with the id and lands as a user_id predicate.
Empty stays the administrative any-owner scope, the same thing an empty id
already means for ListTasks and ListJobs. A foreign delete removes nothing and
returns no error: not yours and not there are the same answer to the caller,
and neither is a store failure.
SetUserID rebuilds the tasks map for the same reason SetTaskSyncNATS does.
GetJobs happens to set the user id first, nothing enforced it, and with the
order reversed the map would be built with an empty tenant and put that user's
tasks back on the cluster-wide subject.
Both halves predate this programme; they are surfaced here rather than caused.
Neither is fully closed for a deployment with the agent pool off, where the
task routes are still served by one cluster-wide service that every
authenticated caller shares; that is a separate gap and it is documented.
testutil.FakeBus grew a real defect this was the first change to trip:
Unsubscribe matched on the filter string, so with two subscribers on one
filter, closing one deafened the other. Subscriptions now carry an id.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The responses.metadata SyncedMap had no durable Store, so its reconnect
re-hydrate replaced nothing. That was survivable while responses converged
through deltas on a broker that mostly stayed up. It is not survivable on a
carrier whose listener is one pinned PostgreSQL session: every response created
while the subscription was down stays invisible on that replica forever, and the
symptom is a 404 from one replica and a 200 from another for the same
response_id.
State that must survive a gap now lives in a response_metadata table, and the
notification only says it changed. The map writes through on a Set and reads the
table on hydrate, on reconnect and on reconcile, so the gap closes instead of
becoming permanent.
The row carries the whole projection as JSON rather than one column per field. A
column-per-field schema would be a second definition of what a peer may act on,
and the two would drift the first time syncedResponse gained a field: the map
would broadcast the new field and hydrate without it, so a replica that had
reconnected would serve a different response body from one that had not, with
nothing failing anywhere. Only PayloadJSON is ever decoded; owner_replica and
owner are indexed copies for an operator reading the table by hand.
A missing row and an unreachable database are different facts. Every store and
adapter method returns a driver failure as an error and never as an empty
result, and syncstate replaces nothing when its source errors, so an outage
leaves the map holding what it had rather than blanking it into a cluster-wide
404.
Liveness is the database's clock, spelled expires_at IS NULL OR expires_at >
now(), because every replica hydrating from this table must agree on which rows
are live and a Go-side cutoff makes that a property of whichever process asked.
The test container shares the host clock, so no behavioural spec can tell the
two apart; the statement shape is pinned instead. The constructor refuses a
non-PostgreSQL handle, because an unguarded now() on the single-binary path
reads as a missing migration.
A ticker sweeps expired rows every five minutes on each replica, and Close waits
for it rather than racing it. Note that the sweep removes nothing while
LOCALAI_OPEN_RESPONSES_STORE_TTL is 0, which is the default: with no TTL nothing
ever expires and the table grows for the life of the deployment. The docs say so
plainly.
EnableDistributed takes the store positionally and last, so a call site that
forgets it fails to compile rather than silently restoring the deltas-only map
this change exists to replace. A nil store there is refused by name: it is
reached only from the distributed branch of route registration, so it is a
wiring bug and not a deployment shape.
What still never leaves the owning replica is unchanged: the resume buffer and
the CancelFunc. The write-through is one row per response state change, not one
per generated token.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The AI-assistant policy says an AI agent must never add a Signed-off-by
trailer, because only a human can certify the DCO. That is right for the
case it was written for: an assistant helping a contributor who then
signs off themselves.
It does not fit automation a maintainer runs. Those pull requests have no
human submitter, so nothing ever signs and the DCO check blocks them
permanently. Sixty-one open pull requests from the maintenance bot are in
exactly that state, every one of them correctly following the documented
rule.
Carve out the case: automation a maintainer operates signs off with that
maintainer's identity. The maintainer certifies the DCO, as they do for a
commit they typed by hand, because they configured the automation, own
its output, and take responsibility on merge. The Assisted-by trailer
still records that a model wrote the code, so provenance is unchanged.
Keep the exception narrow. An assistant helping an outside contributor
still must not sign off, and a bot must not sign for anyone but its
operator, including on a contributor's branch it pushes to.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The three NATS queue groups jobs.new, jobs.mcp-ci.new and agent.execute are
gone. Dispatching work is now a row in a work_claims table, taken by one
frontend replica with SELECT ... FOR UPDATE SKIP LOCKED and driven on an agent
worker as a streaming control RPC over that worker's tunnel. Exactly-one
delivery among competing consumers is a database problem, not a broker feature.
An agent worker has no database, so it never claims; it executes what the
claiming replica hands it.
A claim must not outlive the replica that took it. The reap releases a claim
whose owner is no longer a live replica in the instances table, on the database
clock, and never asks how long the claim has been held. A job that legitimately
runs for an hour on a heartbeating replica is left alone, while a claim whose
owner stopped heartbeating becomes claimable again within one liveness window.
A replica with no advertised address has no instances row at all, so it refuses
to claim rather than have its work reaped out from under it mid-run.
The settle rule is stated once, in settleClaim, and every exit path calls it. A
transport failure releases the claim and never completes or discards it; only a
decoded reply line completes it. That line is deliberately not
cluster.IsWorkerAnswer, which accepts the stream refusals a worker's tunnel
writes before any request body reaches its control server: completing on those
would discard work that never ran.
The terminal line is persisted before the claim is completed, so a store that
refuses leaves the claim standing rather than leaving the job running for ever.
That is the dropped-result defect fixed structurally rather than by retry.
This also surfaces a pre-existing gap rather than causing one: no worker has
ever served plain task jobs, and publishing them into an empty queue group left
them running with no trace. Such a claim is now failed with a reason.
Removes QueueWorkers, --agent-subject and --agent-queue, and narrows an agent
worker's minted JWT by agent.execute and jobs.mcp-ci.new.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A worker has no database and cannot NOTIFY, and it does not need to: every
message it sends is produced inside a handler the frontend invoked, so
there is always an open control response to write on. This adds the two
ends of that, and the authorization decision that sits between them, and
nothing that dispatches yet.
workerctl.Envelope gains Subject, a REQUEST and not an instruction. Empty
means the line is for this caller alone, which is what every pre-existing
progress line is, and omitempty keeps those lines byte-identical for an
older reader. It qualifies a progress line and never a reply line: a reply
is the worker's verdict about the work, and there is no version of
"publish my verdict for me" this control plane has to carry.
nodes.MayBroadcast is the replacement for pkg/natsauth's allow list, and
the inversion is the point. NATS read an EMPTY allow list as NO
RESTRICTION, which is why phase 3 refused to delete the backend branch and
spelled it {"_INBOX.>"}. This one reads an empty list, and an absent node
type, as DENY EVERYTHING, and a table-driven spec pins that by emptying
the agent entry and asserting all three of its subjects are then refused.
Matching goes through messaging.SubjectMatches, the one definition in the
tree, so a filter that fires here fires on the carrier.
nodes.Rebroadcaster.Handle returns a bool and never an error. A refused or
failed re-broadcast is logged and the RPC continues, because the RPC's
outcome is the worker's verdict about the work and a publish failure says
nothing about it. The return shape is asserted at compile time in the file
that states the rule, so changing it to an error does not compile.
ControlClient.CallStreaming's progress callback now takes the line's
subject alongside its raw bytes. The client no longer decodes progress at
all: what a line means is a question about the verb and whether a named
broadcast may be made is a question about the node, and it knows neither.
Both moved into installProgressBridge, which every one of the three
streaming call sites in unloader.go goes through. A line naming a subject
is dropped there rather than delivered as install progress, because
backend.install and backend.upgrade are a backend worker's verbs and a
backend worker is allowed no subjects.
agents.StreamPublisher is the producing end, a messaging.Publisher writing
NDJSON envelopes onto an in-flight control response and flushing each one,
so a tick reaches the frontend while the handler is still running.
Serialized, because two concurrent encodes on one http.ResponseWriter
interleave bytes and tear the framing.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Two things made every end-to-end result on this branch unreliable.
make test-e2e-cluster did not rebuild ./local-ai. It only checked that a
file by that name existed, so an edit that was never rebuilt ran the whole
suite against an older build while reporting on the working tree, and a
missing binary skipped every spec and exited 0 with "Test Suite Passed".
The target now depends on a new e2e-binary target, which is the plain go
build CI already used rather than make build, since that one also builds
the React UI this suite never touches. The harness carries the other half:
localAIBinary now FAILS, locally as well as under CI, when the binary is
older than the newest non-test Go source in the tree, which covers a run
started with LOCALAI_E2E_BINARY or by invoking ginkgo directly. Test files
are excluded from that scan because they compile into the ginkgo suite and
never into local-ai. The CI job drops its own build step and the env var so
that one place owns the build and it happens after protogen-go.
test-e2e-distributed was audited for the same hole and has none: ginkgo
compiles that suite from the working tree on every run and it execs no
prebuilt binary. build-mock-backend already rebuilt unconditionally.
"Worker tunnel under load" bounded the worst probe inside the bulk transfer
window against the worst probe under the empty-load window. A max over n
samples is a biased estimator when the two n differ, and here they always
do: the bulk window is by construction longer and draws several times as
many chances at an unrelated scheduling outlier. Anything loading the box
widens that gap, so the spec reddened on what else was running: 255ms
against a 161ms budget with make lint beside it, 57ms alone. It now bounds
the probe COMPLETION RATE instead, which is the statistic a serialised
session actually moves and a mean over dozens of samples in both terms, so
a uniform slowdown cancels in the ratio. Measured with make lint running:
0.97 direct and 0.60 relayed against a floor of 0.125.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
nodes.<id>.backend.stop was the last worker-facing NATS subject, and it existed
only because ONE publisher had not moved. An agent worker already mounted
workerctl.PathBackendStop on the tunnel it holds, and a backend worker already
took its stop there, so RemoteUnloaderAdapter branched on NodeType to pick a
carrier for a verb both kinds of worker served the same way.
The branch is gone, and with it nodeTypeOf and its NodeTypeBackend default,
which removes one of the ten NodeType branches left to sweep. The adapter loses
its messaging.MessagingClient outright rather than keeping an unused field: it
now holds no publisher, so re-routing any verb back onto the bus is a change to
the struct and to every caller of the constructor, and does not compile until
all of them agree. messaging.SubjectNodeBackendStop and subjectNodePrefix are
deleted, the agent worker's subscription with them.
pkg/natsauth drops the per-node backend.stop grant from the agent SUB list. That
is a narrowing of eleven entries to ten, never to nothing: NATS reads an EMPTY
allow list as unrestricted, so the coverage spec asserts both that the retired
subject is no longer covered and that the queue subjects an agent worker lives
on still are. The e2e half proves it against a real enforcing server: one spec
subscribes successfully on an agent-minted JWT, the next is refused the retired
subject on a JWT minted the same way.
Both halves of the old split were pinned, so both pins are re-aimed rather than
deleted, and the two node types are asserted separately rather than as one
parameterised case, because only two cases can show that the two used to differ.
Three assertions that the adapter published nothing are deleted instead: with no
publisher to hold, no change could ever redden them.
The CLI's handler set moves into agentWorkerControlHandlers so a spec can stand
it up and post to it. That wiring was a bare literal no spec pinned, and
deleting the subscription made it the ONLY carrier for backend.stop: a dropped
field would have been a 404 the frontend reads as a worker too old to serve the
verb, and nothing in the repo would have noticed.
Mutations: the agent branch restored off the control route reddens two specs;
the backend branch restored, separately, reddens five; PathBackendDelete in
place of PathBackendStop reddens nine across both node types; dropping the CLI
wiring line reddens the new wiring table; re-adding the allow-list entry reddens
the unit spec and the JWT e2e spec; and restoring the publisher for real does
not compile.
Four comments this change falsified are fixed, in core/cli, pkg/model and the
distributed-mode docs, which now say both kinds of worker serve
POST /v1/control/backend/stop and what each does with it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Two independent breakages on master make every open pull request red,
for reasons unrelated to the changes under review.
The e2e backend suite stopped compiling. Reply.message is `bytes` in
backend.proto, so res.GetMessage() returns []byte, and strings.ToUpper
wants a string. Every other call site in the file already converts.
tests/e2e-backends sits behind a build tag, so `go build ./...` never
compiled it and the breakage reached master unnoticed.
The darwin vllm build stopped resolving. Upstream vllm-metal deleted
its old dev tags and re-versioned to track the vLLM release it targets,
so the pinned wheel 404s. The coupled vLLM release also moved out of
upstream's install.sh into .github/vllm-release-tag.commit, and the
wheel's platform tag moved from macosx_11_0 to macosx_15_0.
Read the wheel name from the release's own asset listing rather than
composing it from a hardcoded platform segment, so a platform-tag
change cannot silently 404 again, and resolve the vLLM version from
the new metadata file with a fallback to the legacy installer. The
bump script and the extractor learn the same two-source lookup, so the
next nightly run converges on the pin checked in here instead of
reintroducing the break.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(downloader): make file:// installs reachable again
DownloadFileWithContext already has a branch that copies from a local
file, but it could never run. Before reaching it the function decides
whether the destination is fetchable with
} else if !os.IsNotExist(err) || !URI(url).LooksLikeHTTPURL() {
and LooksLikeHTTPURL is http(s) only, so any URI resolving to a local
path is rejected there. Falling through requires the destination to be
missing AND the source to be an HTTP URL, which a file:// source never
is -- leaving the local-source branch below unreachable.
A first import always has a missing destination, so importing
file:///path/to/model.gguf always failed, with an error that listed
file:// among the supported schemes (#11701).
Name the local-source condition once as URI.hasLocalSource and use it
both to admit the destination and to pick the source, so the two cannot
drift apart again.
Signed-off-by: Tai An <antai12232931@outlook.com>
The recommended sampling parameters for a model family were applied at
install and then never took effect. Two things went wrong on the way to
disk.
They were written as top level keys. ModelConfig embeds PredictionOptions
under the "parameters" yaml key, so temperature, top_p, top_k, min_p,
repeat_penalty and presence_penalty are only read from there. At the top
level they parse without error and are then ignored for the life of the
model.
They were also merged in after the YAML had already been marshalled. The
only re-marshal sat behind the artifact binding, which an entry carrying
files: never reaches, so for those entries the defaults were computed and
then dropped before anything was written.
Neither failure was visible in normal use. ApplyInferenceDefaults runs
again at load time and fills the same values from the same table, so the
model ends up tuned correctly while the file on disk pins nothing. It
surfaces when someone edits one of those values expecting it to win, or
when a family is absent from inference_defaults.json and there is nothing
to refill from.
Both install paths are covered: an entry carrying files:, and one that
binds a primary artifact instead.
The empty base spec asserted that the authored parameters block landed
verbatim. It now checks the authored keys individually, because the family
defaults are merged into that same block.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
Two specs in the distributed e2e suite have been red since 1cf847f29, which
stopped workers advertising an address and removed every fall-back to the
node's own endpoint. After that commit a replica row must name the loopback
endpoint of its own backend process: DistributedModelStore.Range refuses to
list a replica whose backend process is unnamed, and SmartRouter treats an
unnamed warm row as naming no process and cold-loads instead.
1cf847f29 updated the unit specs under core/services/nodes for the new
contract but not tests/e2e/distributed, and the phase's closing verification
ran test-e2e-cluster rather than test-e2e-distributed, so nothing reported it.
The task brief named six later commits as candidates and called 671785621
known good; both are wrong, and 671785621 fails these two specs as well.
The fix is the scripted input, not the guard. Both specs wrote an empty
address, which is a row this release cannot produce, since installBackendOnNode
refuses an install that names no address.
The dedup spec is repaired rather than merely un-reddened. With an empty
address its DB row was dropped by the unnamed-replica guard before Range ever
consulted the seen-set, so deleting the dedup check left it green: it asserted
nothing. With the endpoint named, removing that check reddens it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
mcp.tools.execute and mcp.discovery were the only NATS subjects that
combined a queue group with a reply, and no carrier in this design
provides both. They never needed one: a queue group is a way of choosing
a subscriber, and choosing is a query.
The frontend now lists the approved, non-draining agent nodes, asks the
node_connections table in one joined statement which of those tunnels a
live replica holds, prefers one this replica holds so the call skips the
relay hop, and issues an ordinary control RPC on the path task 4 already
mounted. A peer-held tunnel is reached through the relay. That is a
choice a broker's hidden balancing could not make.
The selection reads presence and nothing else. It is filtered only on
node type and on the two statuses an operator controls, never on a health
verdict written on another clock, because refusing a worker that is
connected and answering is the same defect as picking one that is gone.
An empty fleet answers ErrNoAgentWorker, which is deliberately neither
ErrWorkerUnroutable nor anything cluster.IsWorkerAnswer accepts: nothing
was asked of any worker, so no reap guard may act on it.
A reply carrying an Error is the worker's own answer and is returned
unchanged; it is never offered to a second worker, which would turn "this
MCP server rejected your arguments" into "the fleet is broken" and could
run a tool twice. A call that never reached a worker is retried against a
different pick, at most three times, and whatever error is finally
returned is returned unwrapped so its identity survives the loop.
MCP prompts and resources now answer 501 in distributed mode instead of
an empty 200. They are served only from sessions the frontend holds, and
in distributed mode it holds none. That gap predates the removal of the
bus and is not closed by it; this only stops it being silent.
Agent workers keep every other subject, including nodes.<id>.backend.stop.
Their minted JWT loses the two MCP subjects and keeps a non-empty allow
list, because NATS reads an empty one as no restriction at all.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Phase 2 gated agent nodes out of tunnel credentials at the mint site.
That was right while nothing dialled into an agent worker: a credential
would have replaced nothing, and the gate was structural rather than a
second check that could drift. It is wrong now that the frontend needs to
reach an agent worker by RPC.
attachTunnelToken mints for backend and agent nodes and CLEARS for
anything else, through one tunnelEligible predicate rather than two
conditions that can be widened separately. ConnectHandler still never
reads NodeType, so an empty hash is still what refuses an ineligible
node.
An agent worker now starts a loopback control server behind the same
bearer check a backend worker uses, and holds one tunnel whose only
stream tag is http: it runs no backend processes, so the grpc tag has
nothing to route to and is not offered. Its MCP tool, MCP discovery and
backend.stop verbs are served from ONE implementation reached by both the
bus and the tunnel, so a frontend cannot get different bytes depending on
which carrier delivered.
The tunnel is an ADDITION. --nats-url is still required, and agent jobs,
MCP execution, MCP CI jobs and nodes.<id>.backend.stop all still travel
on the bus.
Absence semantics are unchanged. An agent node now has a real
node_connections row whose departure ages past the grace, so the node
type check in HealthMonitor.tunnelDeparted stopped being an optimisation
and became the rule; its comment says so, and the spec that pins it is
shown red under a mutation that deletes the check. The scheduler needed
no change: every placement query already filters node_type = backend, so
an agent node never reaches nodeMayTakeWork.
Shared rules moved to one site each. The request bounds, the POST-only
check and the unknown-path 404 live in workerctl and are called by both
worker packages; the bearer check that guards every extra route is one
function in core/services/nodes used by both server constructors.
workerctl.AllPaths splits into BackendPaths and AgentPaths, with AllPaths
as their deduped union, because a backend worker does not mount the agent
verbs and asserting otherwise would fail a correct worker.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ds4): build CUDA kernels for the target architecture
The ds4 backend compiled its CUDA objects with no -arch. Upstream's Makefile
leaves CUDA_ARCH empty and its `cuda` target refuses to build without one,
offering `cuda-spark` (sm_121) and `cuda-generic` (native) instead. We invoke
its object targets directly, which bypasses that guard, so nvcc fell back to
its default architecture and the kernels ran as JIT'd PTX on the real GPU.
On GB10 (sm_121) that silently corrupted inference: any prompt over roughly 128
tokens produced text unrelated to the input and never closed its thinking
block, so content came back empty and the chat showed only reasoning; longer
prompts failed with "cuda decode failed". It also cost close to two orders of
magnitude of prefill throughput. Measured on one box, same model, same prompt,
same GPU, upstream ds4 at the pinned commit, differing only in the nvcc flags:
make -B ds4 (archless, as we build it) garbage output 4.21 t/s
make cuda-spark (compute_121a/sm_121a) correct output 325.70 t/s
Select an architecture list from CUDA_MAJOR_VERSION, which the backend matrix
already declares for both ds4 cublas entries but Dockerfile.ds4 never forwarded.
Upstream's CUDA_ARCH takes a single value, so it cannot express the fat binary
these images need; NVCC_ARCH_FLAGS is overridden instead, since a command-line
assignment wins over its `:=`. The lists are copied from vllm-cpp rather than
invented so the two CUDA images cover the same GPUs, with l4t/arm64 covering
Orin, Thor and GB10. An empty CUDA_MAJOR_VERSION keeps upstream's `native`
behaviour for local developer builds, and no CI runner has a GPU to enumerate.
DS4_CUDA_HAVE_MXF4 is deliberately left unset: upstream defines it only for
single-arch sm_120/sm_121 builds and guards it with a plain #ifdef rather than
__CUDA_ARCH__, so it cannot be combined with older archs. It gates an optional
MXFP4 indexer fast path whose #ifndef branch returns 0 and falls back cleanly,
so omitting it costs speed on GB10, not correctness.
Assisted-by: Claude Code:claude-opus-5
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
* test(ds4): cover the multi-batch prefill regression
The architecture fix has no automated guard: every existing e2e spec uses a
short prompt, and the miscompiled backend answered short prompts correctly.
The corruption only appears once a prompt spans more than one prefill batch,
so the whole suite passed against a backend that produced garbage in normal
use.
Add an opt-in "long_prefill" capability to the backend e2e suite that sends a
prompt well past one batch with a known needle and asserts the answer still
reflects it, and document in the ds4 guide why the build must never omit an
nvcc architecture, how to check which flags a configuration resolves to
without compiling, and how to run the new spec.
Assisted-by: Claude Code:claude-opus-5
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
---------
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
Propagate gRPC cancellation into DS4 prompt synchronization and poll it at decode boundaries.
Stop on failed stream writes and skip parser finalization and KV persistence for abandoned partial requests.
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
PostgreSQL holds undelivered notifications in a queue it shares with every
session on the server, and it kills a listener that stops draining. Two failure
modes follow, and both are silent: a carrier that blocked on a slow resolver
would lose its connection and with it every later broadcast, and a carrier that
reconnected without re-registering would be connected and deaf.
The receive and dispatch halves were already separate. What was missing is
everything around them. The listener path moves into listener.go and gains a
carrier-level Dropped() so a replica that is behind can be seen; the queue depth
and the spill retention become Config fields with exported defaults; the LISTEN
session gets an application_name so an operator can count listeners in
pg_stat_activity and a spec can drop exactly one of them; and OnReconnect fires
after the re-LISTEN, on a goroutine of its own, because a callback re-hydrates
from a database and must never run on the path whose only job is to drain.
That callback is reached through an optional interface assertion, so deleting
its invocation compiles and every adopter silently stops converging. The spec is
the only guard, and it is named in a comment at the site.
The slow consumer is proved through the transport rather than a seam: an ACCESS
EXCLUSIVE lock on bus_messages stalls the resolver's spill SELECT for exactly as
long as the spec holds it, and the listener is shown still draining and dropping
while it does. The dropped connection is a pg_terminate_backend matched on the
carrier's own application name.
Neither Dropped nor IsConnected is on messaging.Broadcaster, and a spec asserts
that over the interface type. Both are facts about a frontend; the conditions a
scheduler acts on are facts about a worker, and no consumer holding the
interface can read one as the other.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Unsubscribe decided a channel had lost its last subscriber under one
lock and issued the UNLISTEN after releasing it. A Subscribe on the same
root could decide to LISTEN in that window, and the two reached the
connection in the wrong order: the root ended up not listened with a
live subscription on it. It does not heal, because the next Subscribe
sees the registration already there and never re-LISTENs, so the whole
root stays deaf on that replica until the connection drops.
The decision and the statement it implies now happen under one lock,
held across both, at both call sites. A second lock and not the
registration lock: issuing waits on the listener goroutine, delivery
takes the registration lock, and holding that across the wait deadlocks
the carrier.
The race is spec'd through a barrier seam rather than by racing
goroutines. The natural window is microseconds wide, and a spec that
waits for it to open passes by luck; the seam scripts the interleaving,
so the spec decides in both directions.
Resolving a spilled message moved off the listener. PostgreSQL keeps
undelivered notifications in a shared, fixed-size queue, so a listener
that stops draining it can block COMMIT for every publisher on the
server, not only this one. The listener now only drains; one resolver
goroutine reads the row back and dispatches, which also keeps a spilled
message and an inline one on the same subject in the order they were
published.
Three wiring lines that could be deleted with the suite staying green:
the sweeper's start is now pinned by a Config interval, and the two
lines that carry the bus into the deployment now refuse to boot when
either is missing. A subscription can also report what it dropped, so
the party that missed a message is the party that can see it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Distributed mode needs an operator to run a NATS cluster. This adds the
carrier that replaces its fan-out half, so a deployment eventually needs
PostgreSQL and its own HTTP listener and nothing else.
pgbus holds one PostgreSQL session per replica, pinned for the life of
the process because LISTEN registrations belong to one backend session
and a pooled handle would lose them on the next checkout. Publishes go
out on the pool with pg_notify.
Subjects map onto a channel by their first token, from a closed set of
roots. A subject outside the set is refused at publish AND at subscribe
rather than mapped to a channel of its own: a channel name is capped at
63 bytes, and one LISTEN per job id would be unbounded. Refused rather
than dropped, because a subject that goes nowhere and reports nothing is
the class of defect this work exists to remove.
PostgreSQL refuses a notify payload of 8000 bytes or more, and several
subjects on this bus exceed that in normal operation: a job result
carries a whole LLM output, a gallery progress event carries one entry
per node. Those are written to a row and the notification carries the
id. What is measured against the cap is the ENCODED notification, not
the caller's payload, because the subject and the envelope travel too.
The filter grammar is not respelled here. Subscribe asks
messaging.ValidFilter and delivery asks messaging.SubjectMatches, which
makes this the first production caller of a matcher that had only test
doubles. New refuses a DSN that names a different database from the
pool: that pairing publishes successfully, delivers nothing, on every
replica, and reports no error anywhere.
Nothing publishes on it and nothing subscribes yet. The construction is
wired anyway, because the DSN has exactly one legitimate source and a
setting that decides whether any broadcast is delivered should not be
invented by whichever call site is migrated first.
Delivery is at-most-once, like NATS core. Nothing downstream may read a
message it did not receive as evidence about a node: a carrier that
cannot deliver is not a worker that is gone.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Three copies of "does this filter match this subject" lived in the tree: one
in testutil.FakeBus, a byte-identical second inside galleryop's own private
fakeBus, and a third in pkg/natsauth with different semantics. The first two
are doubles the specs publish through, and the carrier the pgbus work is about
to add needs the same rule in production. Two spellings drift, and the drift
reads as a peer that receives an event on one replica and not on another.
messaging.SubjectMatches is now the only definition either double uses. The
natsauth copy stays: it matches a NATS server allow list, so it has to
implement the '>' tail wildcard this one deliberately refuses, and Task 16
deletes that package anyway.
'>' is refused rather than implemented because no surviving subscription uses
it, and a caller who writes one must get no messages rather than silently
getting every message on the prefix. The refusal is checked BEFORE the
filter == subject fast path: a verbatim port checks equality first, and then
the filter "a.>" matches the literal subject "a.>", which is the contract
leaking. One table row pins that ordering and it is the only row that does.
messaging.ValidFilter refuses an empty filter, a '>' filter and an empty token
so a subscriber is told at subscribe time instead of staying silently empty for
the life of the process. FakeBus.Subscribe calls it, which is what keeps the
double honest about what the carrier will do, and three new testutil specs pin
that wiring: the previous state of the tree had no spec at all that failed when
the double's wildcard routing was replaced by exact matching in any package the
plan named.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Five cluster specs that run the binaries an operator runs, plus the repair
of eighteen specs phase 2 left red.
The eighteen were router_tracking and full_flow, failing since 1cf847f29 on
"reported backend installed but named no address for the process". Two
contracts had changed under them: an install reply that names no
worker-local address is refused rather than substituted, and a frontend with
no worker dialer reaches no backend at all. Nobody noticed for a phase
because phase 2 verified with --label-filter='Cluster', which excludes both
suites. ServeBackendLifecycle and tunnelBackendClients state both facts once
for every spec.
The transport double is the part that matters. It translates a refused
connect into cluster.ErrStreamTargetUnavailable, which is what a real worker
answers when its backend process has died and what IsWorkerAnswer lets a
reap guard act on. A bare ECONNREFUSED reaches those guards as "no route" and
reaps nothing, so a double returning the raw syscall error could never fail
the way production fails; putting it back reddens the stale-record spec and
nothing else.
The new specs cover: a backend worker with no bus URL in its /proc environ
registering, being scheduled onto and serving inference; a backend install
and a backend listing driven through the replica that does NOT own the
worker, with the owner read through the production Owner query and re-read
after; that install's progress proven to arrive before its terminal reply,
made deterministic by a gallery server that holds the worker's fetch open so
a reply cannot exist yet; a worker whose tunnel is genuinely gone, waited for
rather than assumed, losing nothing inside the reconnect grace and re-homing
after; a heartbeating worker with a permanently dead tunnel losing its
healthy status while an agent worker in the same cluster keeps it; and the
suite's negative control, where a control RPC to a tunnel-less worker fails
naming the missing route, reaps nothing, and succeeds the moment the tunnel
returns.
Every scenario was attacked. The churn one was WRONG on the first attempt
and only the attack found it: its hold window sat entirely inside
cluster.InstanceLiveness, so a killed replica still read as a live owner
throughout, presence was "connected", and the spec passed with the reconnect
grace set to a nanosecond. It now blocks the tunnel before the kill and waits
for the ownership row to actually empty. Attacks that redden the rest:
posting at the owner, writing the install reply before the work, collapsing
PresenceReconnecting into PresenceGone, removing the non-backend node-type
guard, and not blocking the tunnel. Agent workers turn out to be protected
twice over; no single mutation reaches them.
Harness: Options.AgentWorkers and Options.ReconnectGrace, WorkerEnviron
(read from /proc, because Cmd.Env is the harness agreeing with itself),
NatsURL, FrontendBackendsDir, AgentWorkerName, PostJSON, and a node String()
so a failing roster assertion is readable instead of several hundred bytes
rendered as numbers.
Budget: 20 specs at 787 to 808 seconds over three runs, up from phase 2's 591
to 612. --timeout goes to 30m so a loaded runner reports a cause rather than
a spec name.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A local-ai worker no longer opens a bus connection. connectNATS and its
spec are gone; Run registers once, starts its tunnel, arms /readyz on that
tunnel, and heartbeats. The worker's bus credential flags (--nats-jwt,
--nats-user-seed, --nats-require-auth, the three TLS flags) and
Config.NatsAuthRequired go with it. --nats-url stays, accepted and
ignored, so an existing worker command line still parses.
/readyz was the thing most likely to wedge a tunnel-only worker: it
required a live NATS link, so a worker with no bus would have reported
itself unready forever. nodes.NATSReadiness becomes nodes.TunnelReadiness
over a local interface{ Connected() bool }, and worker.Tunnel gains
Connected(), backed by a mutex-guarded session field the loop publishes
and clears. A closed-but-not-yet-cleared session reads as disconnected:
the loop waits for every in-flight stream before it clears the field, and
the probe must answer not-ready through that wait.
The heartbeat gate is DELETED rather than re-pointed at the tunnel. The
heartbeat is the worker's own answer that its process is alive; whether
the frontend can reach it is a separate fact the frontend already holds
and ages against LOCALAI_WORKER_RECONNECT_GRACE. Withholding the
heartbeat would report an unreachable worker as an absent one on the one
path with no grace, where the health monitor marks it offline and its
pending backend ops are deleted behind it. heartbeatLoop is given no view
of the tunnel, so a gate cannot be added back without changing its
signature.
Removing the NATS credential manager from this path also removes a defect
it carried: its refresh loop re-registered on a timer to renew a JWT, and
Register CLEARS a node's NodeModel rows. Any backend worker running on
frontend-minted credentials had its replica rows deleted roughly every
18 hours.
Of core/cli/workerregistry, everything survives. The manager is still
used in full by core/cli/agent_worker.go, which still needs NATS: Acquire,
Provider, RefreshLoop, HasCredentials and TunnelToken are all untouched.
The backend worker simply calls RegisterFullWithRetry directly now.
WorkerPermissions is documented as serving agent nodes, and its non-agent
branch narrowed to _INBOX.> on both sides. It is NOT deleted: NATS reads
an empty allow list as no restriction, so returning nil would upgrade
every JWT the frontend still mints for a backend node from its own inbox
to the whole account.
Agent workers keep the bus everywhere: their CLI flags, their
subscriptions, the agent branch of WorkerPermissions, and the compose
service with its LOCALAI_NATS_URL and depends_on: nats.
Also corrected two flags the Nodes page advertised that do not exist
(--distributed-nats, --distributed-db), and a log line plus several
comments that still named a bus the code no longer touches.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Removing "Presence: clusterRegistry" from the options literal in
initDistributed left all seven suites and tests/e2e/distributed green.
The predicate was right and its input was silently nil, which returns the
deployment to absence being decided by nothing, with no log line and no
failing request. That is the fourth finding of this exact shape in this
phase.
The two assignments move out of a twenty-field literal into
distributedSchedulerOptions, a named function a unit spec can reach.
Deleting either is now red. The health monitor takes its presence reader
and grace as a required positional pair instead, so deleting those does
not compile at all. requireAbsenceWiring then refuses to start a
distributed frontend whose scheduler or health monitor has no source of
absence, because refusing to boot is the only symptom either failure has.
With a fresh heartbeat and a permanently gone tunnel there was no reaper
at all. A heartbeat says the worker's supervisor is alive; it says
nothing about whether anything here can reach that worker's backends,
because those are reached over the tunnel. A proxy that stops upgrading
WebSockets, a rotated registration credential or a reconnect loop longer
than the grace left a node listed healthy forever while every request for
a model already loaded on it failed "no route to that worker", and every
reaper keyed on the heartbeat. The health monitor now reads presence from
the same place and against the same window as the scheduler and demotes
such a node. That also ends the 15s re-promotion: the demotion arm
returns before the recovery arm, so the scheduler's demotion is no longer
undone on the next tick, and recovery needs the tunnel back rather than
just the heartbeat.
The demotion is status-only. MarkOffline would DELETE the node's rows,
and deleting rows on a presence read would give any future defect in that
read the widest blast radius in the system for nothing the demotion does
not already deliver.
LRU eviction is the third path that commits work to a node, and it read
only the stored status. A node full enough to be an eviction target is
exactly the node the VRAM and idle selectors never offer, so
pickReachableNode structurally cannot cover it. It now runs its chosen
node through the same nodeMayTakeWork predicate, demotes it and evicts
again rather than handing back an install that cannot land. Presence is
read after the transaction and not inside it: reading it inside would
hold a FOR UPDATE lock across a query needing a second pooled connection,
which is how concurrent evictions deadlock a pool.
Also: a router built with a presence reader and no grace now has its
documented default pinned by a spec rather than only claimed by a
comment; ageDeparture asserts RowsAffected, since an UPDATE matching
nothing succeeds and the inside-the-grace spec returned the same verdict
either way; the scheduler comment that still described the bus is
corrected; the docs stop conflating heartbeat recovery with tunnel
recovery and name the third reader; and an overlong rewrapped line in
membership.go is folded.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The scheduler decided whether a worker had gone away from
nats.ErrNoResponders: one frontend's observation that nobody answered IT
within a request budget. Two replicas asking in the same moment could
disagree and demote each other's workers, and a worker re-homing its
tunnel between replicas looked identical to one that had died.
SmartRouter now reads cluster.Presence instead. Only PresenceGone -- no
live replica holds the tunnel AND the departure has outlived the
reconnect grace -- excludes a node from placement, and it is a fact every
replica reads identically from the database. PresenceReconnecting,
PresenceUnknown and a failed presence query are all non-verdicts and
place work as normal: excluding on a database hiccup would cost the fleet
its capacity for a reason that has nothing to do with any worker.
nodeAnswersOnBus is deleted. It excluded on a sentinel no control RPC can
produce, so it decided nothing while PingNode cost a relayed round trip
per scheduling decision to feed it. PingNode goes with it, from the
adapter and from NodeCommandSender. isRequestTimeout drops nats.ErrTimeout:
every verb this adapter sends now travels over the worker's tunnel.
The predicate is named nodeMayTakeWork rather than nodeHasRoute. "Route"
is ErrWorkerUnroutable in this package, the condition nobody may act on;
PresenceGone is the one a scheduler may. Spelling them the same way is
the collapse this work exists to prevent.
Also folds in ReapStale's return rename: it counts connection rows
CLEARED, never rows deleted, and reading it as a delete count would make
a worker that is re-dialling right now look forgotten.
The spec pinning that a message merely quoting "nats: timeout" is not a
timeout was scripting a SUCCESSFUL reply carrying the phrase, which comes
back with a nil error and never reaches the classifier. Restoring the
string match left it green. It now scripts a 5xx whose body carries the
phrase, and asserts that the phrase reaches the classifier as a
precondition.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The rule "a body this worker could not parse is a non-2xx, never the
worker's answer" is written at three exits in control_files.go and only
ensure was pinned. Turning stage's or listdir's decode exit into a
200-with-error left worker and nodes entirely green, and what that
converts is a frontend's malformed request into the worker's own verdict
about a file, which passes cluster.IsWorkerAnswer and reaches a reap
guard. The production code was already right; nothing held it there.
The e2e NATS JWT spec was asserting the opposite of the code and passing.
It published nodes.<id>.files.in and called it an allowed subject after
that grant was deleted, and it could not tell: a permission violation
does not close the connection, so FlushTimeout and IsConnected both stay
happy. It now reads LastError, the way its sibling always has, and
asserts the denial plus the one publish right a backend worker has left.
Also pinned, each mutation-verified alone: the CreateTemp branch (an
existing staging-tmp at 0500 reaches it without a seam), the walk's
context check (a caller that gave up must fail the listing, never be
answered with a short one), and the cache and data directory layout.
The data directory was derived twice, once in worker.go and once for the
listdir verb; worker.go now reads the same helper, so a move cannot leave
a verb listing files the file server does not serve. The per-verb RPC
ceiling moves from an argument at five call sites into fileRPCBudget,
so no site can name the wrong one, and the two values are asserted.
The body-cap table now holds both directions locally and with two
different claims: a body exactly at the cap proves the bound is a
ceiling and not an off-by-one, and an absolute megabyte proves the cap
stays above real gallery traffic. Only the second notices a cap shrunk
to 64 KiB.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The four nodes.<id>.files.* subjects were the last commands a
serve-backend worker took off the bus. They are now HTTP routes under
workerctl.Prefix, on the same loopback server and behind the same bearer
check as the ten lifecycle verbs, so the frontend reaches them through
the worker's tunnel.
files.listdir is the verb this matters most for. Its reply had to fit a
payload the bus would carry, which put a wide model directory close to
the limit; a response body has no such ceiling, so nothing truncates the
listing at either end. A short listing reads to the frontend as files
the worker does not have.
S3NATSFileStager becomes S3FileStager and calls ControlClient, which
means every failure now lands in the bucket phase 3 exists to keep
straight: a route this frontend could not use is unroutable and nothing
may act on it, while the worker's own answer, including "that file is
not there", is evidence a caller may act on. Each RPC's deadline is
DERIVED FROM the caller's context rather than started fresh, at every
one of the five call sites, so a caller that gave up stops the RPC too.
A worker started without an object store mounts no file verb at all and
answers 404, which is the same answer a build too old to know them
gives. The subjects and the backend worker's files.> publish grant go
with them; a backend worker now publishes nowhere but its own inbox.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review fix round 1. Two blocking findings and seven non-blocking; both
blocking ones are M12's shape again, and this time on the invariant
itself. No production behaviour changes here: everything below was
already correct and merely unpinned, so re-inserting the defect left all
679 specs green. The only non-comment edits are one struct-field comment
and one log message.
"A failed control RPC no longer demotes a node" is stated three times in
this package and was pinned once, at ListBackends. Putting MarkUnhealthy
back at either op-drain site passed. What that buys in production is the
fleet-wide eviction this phase exists to prevent: MarkUnhealthy removes a
node from ListDuePendingBackendOps AND from scheduling, so a frontend
replica that has just lost its tunnels demotes every node it holds an op
for, for a reason that is about the frontend. The reconciler's is the
worse of the two, being a background loop nobody is watching. Both now
have a spec, each with the recorded op failure as its negative control so
"still healthy" cannot pass by nothing having happened.
The sweep the review asked for found four more rules stated at more call
sites than they were pinned at, and two the review had not:
The still-installing surfacing at the manager layer has two call sites
and was pinned at InstallBackend. Dropping it from UpgradeBackend
reported a spent budget as GREEN SUCCESS: the admin sees the upgrade
finished while the worker is still re-pulling gigabytes.
The agent-node skip has two call sites and was pinned at ListBackends.
Without it the fan-out enqueues a row for every agent node, and an agent
worker serves no control plane, so that row can never drain: it retries
until the dead-letter cap.
The still-installing conversion has three call sites and was pinned at
two; the legacy force-install fallback was the gap. Its budget was
unpinned too, so the new spec asserts both, on the upgrade budget rather
than the install one, since the fallback re-fires an install as part of
an upgrade.
The carrier split has two call sites and was pinned at one. Hardcoding
NodeTypeBackend in UnloadRemoteModelContext passed, and an agent node
holding a node_models row would then have its stop sent over a tunnel it
does not hold, fail, and leave the row behind. The new spec unloads a
model held by one node of each kind and asserts each stop went to that
node's own carrier and to no other.
router_nats_liveness_test.go asserted demote-on-absence, which production
can no longer produce, and its header described the pre-cutover world.
The exclusion is unreachable by construction rather than by argument:
cluster, the package supplying every control-path dial error, does not
link nats.go at all. The file now says that, and gains the assertion that
IS load-bearing, a table naming each sentinel a control RPC can answer
with and requiring that none of them excludes. Widening the exclusion to
ErrWorkerUnroutable reddens four of its entries plus the real-adapter
scheduling spec.
unroutable keeps no budget-first guard and the reason is now written at
it: unlike controlFailure it reads one already-recorded error rather than
racing a live deadline, and an expiry is not in streamRefusals, so it
falls to the umbrella without one. The two implement the same split at
two layers and each now names the other.
Fourteen comments still described the bus. Among them the reconciler
saying a drain would "churn NATS every tick", a spec comment naming a
subject builder this branch deleted, and the agent-skip comment
explaining the skip by a subscription that no longer exists.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The ten backend and model lifecycle verbs stop being NATS requests and
become HTTP calls on the worker's own control routes, reached through
that worker's tunnel on the `http` stream tag that already carries file
staging. Nine subject builders and the per-op install-progress subject
are deleted with their entries in the worker's NATS permissions; the
request and reply DTOs are untouched, so a body on the wire is byte for
byte what the subject carried.
This closes the merge gate Task 3 left open, which was worse than lost
commands. Once the worker stopped subscribing, PingNode was still asking
nodes.<id>.backend.list and nodes.<id>.models.running, so EVERY healthy
worker answered no-responders, nodeAnswersOnBus read it as absence and
pickReachableNode demoted it on the scheduling path. PingNode is a
control RPC now, and no control RPC can produce ErrNoResponders, which
is the only error that exclusion acts on. Two specs drive
pickReachableNode against a real adapter and a worker answering over its
control plane, which is the only arrangement that can see the difference:
the router's own double never touches a transport and stayed green for
the whole window the defect was live.
How a control RPC FAILS is the whole of this change, so it is decided in
ONE function reading ONE table. A worker's answer passes through
unwrapped, so cluster.IsWorkerAnswer still sees it and a reap guard may
act on it; everything else is wrapped in ErrWorkerUnroutable so nothing
can. There is no third branch, because a third branch is how the eight
collapses on this branch happened: each was a site that decided for
itself which errors were evidence. A 404 under the prefix is its own
sentinel, because it is the worker stating a deployment fact about
ITSELF rather than a verdict about a backend, and only the legacy
upgrade fallback may act on it.
The caller's budget is checked FIRST. A timeout is not a verdict: a
refusal arriving in the instant a deadline expires would otherwise be
reported as the worker's non-transient answer, which reaps a row, and
nothing orders the two timers.
A 5xx and an undecodable body are transport failures, not answers. An
empty ModelsRunningReply means "this worker is running nothing", which
the reconciler acts on, so it must never be manufactured from a body
that would not parse. A stream that ends before its reply line is the
same rule one layer up: a tunnel dying mid-install is not the worker
saying the install failed.
backend.stop is split by node type rather than moved. Agent workers hold
no tunnel, so they have no control plane to serve, and they still
subscribe to nodes.<id>.backend.stop to drop cached MCP sessions; that
subject and its agent permission both survive. It is the honest
intermediate state until agent workers hold tunnels too.
A failed control RPC no longer demotes a node anywhere. ErrNoResponders
meant "not on the bus"; a control failure means "this frontend could not
route to it", which is equally what a healthy worker re-homing its tunnel
between replicas produces. Absence is a fact read from the database, and
the scheduler starts reading it in a later task.
The rolling-update fallback re-fires a DESTRUCTIVE force-reinstall, so
it runs only on the worker's own 404. Its negative direction was pinned
at the admin call site and unpinned at the reconciler's, where widening
the condition to any error left all 676 specs green: a background drain
nobody is watching would then force-reinstall every queued backend the
moment a replica lost its tunnels. Three specs cover it, arranged so the
force install IS reachable in the negative case and a fallback that
fired would show as a call and a drained row.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review fix round 1. Seven non-blocking findings; the blocking one is a
merge gate for Task 4 rather than anything in this diff, and the report's
concern about it is corrected: until Task 4 lands, PingNode probes two
subjects no serve-backend worker subscribes to any more, so every healthy
worker reads as absent and is marked unhealthy on the scheduling path.
The rune-boundary cut in truncate was true behaviour with nothing
holding it: a byte-wise mutation survived all 201 specs. isRuneStart is
replaced by utf8.RuneStart, the same predicate the cluster package uses
for this rule, and two specs pin it, one with a rune straddling the
bound and one with a rune ending exactly on it so the fix cannot be
"always walk back".
unloadModel answered Success:true whatever Free did. That is the worker
saying "done" about work it did not do, and the frontend's only caller
is EvictLRU, so a false yes told the scheduler VRAM had been released
and let it place the next model on a node still holding the old one. It
now reports the failure, following stopModelExact, which is the honest
pattern already in this package. Still a 200: the worker answered, only
its verdict is negative. An address with nothing loaded still answers
success, which is a true answer rather than a claim about work done.
NewDebouncedInstallProgressPublisher had no production caller after the
last commit, only its own spec. Deleted rather than wired: wiring it
would publish every event on two carriers, which is what the carrier
decision exists to avoid. Its specs now run against the sink, plus one
that pins the identity stamped on each event, since the subject used to
carry the op and node id and now nothing but the body does.
The install progress wiring was exercised by no spec, because with no
gallery nothing ever invokes the download callback. The guard moves into
startProgress, shared by install and upgrade, which also emits one
resolving event before any gallery work. That is worth having on its
own: a cold install spends minutes on a manifest and a progress stream
with nothing on it is indistinguishable from a broken one. It also makes
the wiring observable end to end, and four specs now drive the real
installBackend and upgradeBackend over HTTP with no override.
model/stop and backend/stop keep taking Background rather than the
caller's context, and the sites now say why. model/stop is the
acknowledged stop path: it reserves the process, frees it, kills it,
waits for exit and releases the port, and abandoning that because the
caller hung up would leave a process marked stopping, a port not
returned to the allocator and a row nothing reconciles. In
stopBackendExact the Free is a courtesy before a kill that happens
anyway. model/unload differs because Free IS the operation there.
A route set with no prefix or no registrar is now a startup error rather
than a silent no-op: a server that comes up healthy while every route
the caller registered answers 404 is, through a tunnel, indistinguishable
from a version skew. And the AllPaths spec no longer claims to catch a
constant that was never added to the set, which it cannot; it asserts
the whole set instead, which catches a verb dropped from it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Ten NATS subscriptions on the worker become ten HTTP routes under
/v1/control/, served on the loopback HTTP server the worker already runs
and reached only through the tunnel's existing `http` stream tag.
The carrier is the tag that already exists rather than a new one. A new
tag would have had to invent correlation, per-request deadlines,
unbounded payloads and a progress stream, and each of those is a place
this branch has already put a defect. It would also have added a fifth
entry to the worker's stream-refusal vocabulary, which decides what a
frontend reaps on and took eight fixes to settle. Riding `http` means a
control RPC to a worker another replica holds takes the same relay the
inference path takes, which is the path that has been measured.
The request and reply DTOs are untouched, so a body on a control route
is byte-for-byte what the corresponding subject carried. No subject was
deleted: agent workers still subscribe to nodes.<id>.backend.stop.
Install and upgrade stream. They answer application/x-ndjson: zero or
more {"progress":...} lines carrying the same event the per-op NATS
subject carried, then exactly one {"reply":...} line, always last. That
deletes the 8000-byte notification cap structurally instead of
reproducing it on a new carrier: a progress line is written into the
response the caller is already reading, so there is nothing to size and
no subscribe-before-request window. The debouncer is shared with the
NATS publisher rather than forked, so the ~4/s tick bound is one fact.
A verb's own failure is a 200 with Error set, never a 5xx. The frontend
maps a transport failure onto "no route to that worker", which nothing
may act on, and the worker's answer onto evidence a reap guard may act
on; answering 500 for a failed install would put the worker's verdict
in the bucket reserved for a broken link. Only a request that could not
be read or routed is non-2xx.
Control RPCs carry the caller's budget. r.Context() replaces four
context.Background() calls at the gallery-install sites, and the one
pre-existing fixed timeout on model.unload is now derived from the
caller's context so a shorter budget is honoured. No timeout is invented.
The inner `go func()` in the install and upgrade handlers is deleted
rather than nested: it existed because one subscription served every
install, and over HTTP each request already has its own goroutine.
Per-backend serialization stays lockBackend, which is what actually
prevented two requests racing the gallery directory.
Bounds against a boundary the worker now serves: every body is capped at
8 MiB before any decode; the 404 echoes at most 128 bytes of the request
path, cut on a rune boundary so a half rune cannot travel downstream as
a replacement character; non-POST is refused before the body is read so
a probe cannot fire a command; the streaming responses set nosniff.
The routes mount through nodes.AuthenticatedRoutes, which hands the
registrar a private mux and puts the whole prefix behind the same
constant-time bearer check as the file routes. The worker's HTTP server
now takes the supervisor as a required parameter, so there is no way to
start it without the control plane mounted.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The retention a worker's departure is kept for is now derived from the
reconnect grace, so a purge can never outrun the window Presence measures
against. Nothing pinned that. The sweep could be reverted to pass the constant,
or the setter emptied out, and the suite stayed green either way: the specs
covered the arithmetic helper, and the fix is the wiring.
The loop now has a spec of its own. It departs two workers either side of the
difference between the floor and the derived retention, and the row that must
go is what witnesses the sweep running at all, so the row that must stay cannot
survive by nothing happening.
The default grace goes from 60s to 90s. Two of the worker's ceiling backoffs is
60s, but the failed dial between them costs its handshake timeout too, which
puts the worst case at 70s, and the backoff resets only after a session long
enough that a replica accepting a dial and then dying denies it. So the ceiling
is reachable exactly during the rolling restart this window exists for, and 60s
sat on the edge of it. Too short reports a live worker as gone and costs a
model reload; too long reaps a dead one later. The cheaper mistake is the long
one.
A held row whose owner is dead and whose stamp is stale is the state a rolling
upgrade actually produces, and it was the one state no spec built. It has an
answer now, and the two ways to get this wrong land either side of it: reading
the stamp first says gone, reading held-ness without the liveness join says
connected.
Two comments claimed more than the code did. There IS a grace at which a live
worker is reported as gone, which is the point of it being a duration; and the
switch that reads held-ness first is only a partial second gate, since with the
SQL gate gone and a dead owner it answers gone rather than reconnecting.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A worker whose tunnel is gone is not, by that fact, a worker that has left.
Absence is what makes the scheduler stop placing work, reap the worker's rows
and evict its models, and one of those paths runs during inference, so the
deployment needs to tell a worker re-homing between frontend replicas from one
that is really gone before anything acts.
Registry.Presence answers that in one joined statement, with four values and
not a boolean: unknown when there is no row at all (this package cannot tell a
worker that has never dialled from one whose departure aged out, and must not
guess), connected while a live replica holds the tunnel, reconnecting while the
departure is inside the grace, and gone once it is older. Only the last is a
verdict a caller may act on.
Held-ness is asked FIRST and the departure only refines it, in the SQL and
again in the switch that reads it. Every writer here clears disconnected_at in
the statement that writes the owner, but that is a property of these writers
rather than of the table: a replica running a binary from before the column
existed re-claims without clearing the stamp, so during a rolling upgrade a
held row carries an old departure, and a read that consults the stamp first
reports a connected worker as gone for the whole upgrade.
Both windows are computed by the database, for the reason every other window in
this package is: they are compared across replicas, and replicas disagreeing
about whether a worker is gone is the flapping this branch exists to remove. No
behavioural spec can see the difference, since the test container shares the
host clock, so the statement shape is pinned instead.
The grace is an operator's knob, defaulting to twice the worker tunnel's
maximum reconnect backoff. That made the fixed departure retention wrong: an
operator raising the grace past it gets a purge that deletes departures before
the grace elapses, so a worker that is gone reads as unknown forever and
nothing ever reaps it. The retention is now derived from the grace, with the
old constant as its floor.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The membership tick's call to PurgeDepartedBefore was the only production
wiring this change introduced, and removing it left the suite green. A
retention nothing applies is a departure that never ages out, which is
the state the sweep's held-ness filter exists to make reachable at all.
A spec now ages a released row past DepartedRetention on the database
clock, starts a real Membership, and waits for the row to go.
Release and Deregister leaned on "no owner id is ever empty" to avoid
touching an already-departed row, which is the accident Owner refuses to
lean on. A departed row keeps its epoch and carries an empty owner, so a
release or a deregistration naming an empty id matched it and stamped a
fresh departure over the old one, making a worker that left long ago look
like one that has only just gone. Both now filter on connectionIsHeld.
The comment on DisconnectedAt claimed a held row never carries a
departure. A binary from before this column existed claims without
clearing the stamp, so a rolling upgrade produces exactly that row. The
comment now says what holds, and says to ask held-ness first and read the
stamp second.
The sweep's vocabulary follows the code: it records departures where the
comments still said it deleted rows, and its log line separates the
instance rows it deleted from the connection rows it left behind.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Releasing a worker tunnel deleted its node_connections row, so "this
worker's link dropped a moment ago" and "this worker has never connected
here" were one observation: no row. Nothing above could tell a worker
re-homing between replicas from a worker that is gone, and any grace
period built on top would have had nothing to measure from.
The row now survives a departure. Release clears owner_instance_id and
stamps disconnected_at on the database clock; the membership sweep and
Deregister do the same for every connection a dead or departing replica
held; Claim clears the stamp in the same upsert that writes the owner, so
a reconnect is never observed half-applied. PurgeDepartedBefore deletes a
departure once it is older than DepartedRetention, and the membership
tick owns that schedule.
Owner and OwnerRow report a departed row as ErrNoConnection, through the
one predicate connectionIsHeld, the way instanceIsLive is the one
predicate for replica liveness. This change records the departure and
does not interpret it: how long ago it happened is nobody's answer yet.
The sweep only clears rows that are still held. An empty owner is in no
instance's id, so without that filter every heartbeat would restamp every
departed row and no departure could ever age out.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Web Chat Settings left the System Prompt field empty but still treated a
blank/whitespace value as an explicit system turn. That satisfied
tokenizer chat templates' messages[0].role == system check and suppressed
the model YAML system_prompt on fresh chats.
Omit empty/whitespace system messages in the React and Alpine UIs, strip
them server-side, and inject config.SystemPrompt for tokenizer-template
models when the request has no real system turn.
Fixes#11834
Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
* fix(distributed): evict only when a node is known to be full
scheduleNewModel asked the registry for a free replica slot and treated
every error as "this node is full", so a control-plane database slow
enough to time out the lookup evicted a healthy loaded model. The
evicted process died, a peer frontend still holding its address dialled
the dead port and retried, and the model thrashed between nodes. The
comment on the branch already said it meant a full node; the code never
tested for it.
Evict only on ErrNoFreeSlot. Any other error now returns and names the
lookup that failed, so a slow database degrades into a diagnosable
load failure instead of into lost work.
An audit of the rest of the router found one branch of the same shape:
node selection discarded the error from its last-resort finder, so a
database timeout there also produced a nil node and evicted for it.
That path now returns unless the finder said gorm.ErrRecordNotFound,
which is the only answer that means the cluster had no node to give.
No other destructive branch in router.go fires on a generic error.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): checkpoint heartbeat writes instead of writing every beat
Every heartbeat UPDATEd backend_nodes. Six nodes at a ten second beat is
roughly 52,000 writes a day against a six-row table, and that churn is
what turned a blocked autovacuum into a 460 MB table whose six-row scan
cost 867 ms and timed out the queries that place models.
A beat carrying only a fresher timestamp now waits for the checkpoint
interval. Each reported field is compared against the value last
persisted rather than tested for presence, because a worker sends its
disk figures on every beat and presence alone would suppress nothing.
A node's first beat, a changed total VRAM, total disk or GPU vendor,
and a free VRAM, RAM or disk reading that has moved more than 256 MiB
from the persisted value all still write at once. A node that is not
active is never suppressed, because it recovers only when the health
monitor sees a fresh timestamp.
The persisted column is up to one interval stale by design, so the
stale-node threshold moves from 60s to 5m to cover it.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): fail worker readiness when a held backend is unreachable
The readiness gate tracked only the NATS link, so a worker whose backend
processes had died still answered /readyz with 200 and kept receiving
loads. One node did exactly that during an incident: it reported healthy
while its backend port refused connections, and every load routed to it
failed.
Readiness is now the NATS link and, for each backend process the worker
believes it is running, a short dial of its recorded address. A worker
holding no backends stays ready, because idle is a healthy state.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): keep a starting backend out of the readiness dial set
A backend process is inserted into the supervisor map with its gRPC
address already recorded, but the address refuses connections until the
gRPC server binds, which the startup poll allows up to 30 seconds for and
which takes 10 to 15 seconds on a slow node. The new data-path readiness
probe dialled that address straight away, so a worker answered /readyz
with 503 for the whole of every cold backend start. The container
HEALTHCHECK absorbs that, but a Kubernetes readinessProbe at 10s does
not, and the worker would leave rotation each time it loaded a model.
The skip for a stopping process had no counterpart at the other end of
the lifecycle. Backend processes now carry a serving flag, set where the
startup health-check gate succeeds, and the probe dials only processes
that are serving and not yet stopping. backendStartStillValid becomes
markBackendServing: the check and the mark must share one lock hold, so
the flag can only ever land on the entry the key currently owns.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(distributed): export control-plane database health gauges
Four transactions wedged on a corrupt index held the vacuum horizon open
for 42 days. Nothing measured it, so the first symptom anyone saw was
models failing to load six weeks later, by which time a six-row table
had grown to 460 MB.
Export the oldest xmin age, the longest open transaction, and the dead
tuple ratio on the registry tables. The first is the number that would
have caught it: it sits near zero in health and was 21,002,291.
Sampling is scrape-driven behind a cache, and a failed sample reports
the last good values rather than failing the scrape, because these
gauges matter most when the database is already struggling.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): rate-limit failed control-plane database samples
The cache advanced its clock only on a successful sample, so once the
database started failing every scrape retried the query immediately.
That turned the cache off in the one regime it exists for: a retry
storm at scrape cadence aimed at a database already in trouble. A
catalog read that consistently exceeds the 5 second timeout also paid
that cost on every scrape, with all scrapes serialised behind the
sampler mutex.
Time every attempt rather than every success, so failures and timeouts
cost the same interval as good samples. Whether a good sample exists
moves to its own field, keeping the gauges absent until the first
success and holding the last good values through later failures.
Also note in the runbook that pg_stat_activity cannot see prepared
transactions or replication slot xmins, so a healthy-looking xmin age
does not by itself rule out a blocked horizon.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(distributed): pin that a failing database evicts nothing
Exercises the real distributed stack against a control-plane database that
refuses the router's slot lookup, and asserts the scheduler reports the
lookup it could not answer instead of falling through to eviction.
The failure is injected with privileges rather than a statement timeout. A
timeout set with ALTER DATABASE also breaks AutoMigrate, and it leaks into
every later spec in the suite unless it is reset, so the spec would end up
testing the migration rather than the scheduler. Instead the spec creates a
dedicated login role, points a second gorm handle at it, and revokes that
role's SELECT on node_models.replica_index. This has to be a separate role:
the test container's owner is a PostgreSQL superuser, and superusers bypass
every privilege check, so revoking from CURRENT_USER is recorded and then
ignored.
The revoke is scoped to one column on purpose. Revoking the whole table
would also blind node selection, which runs first and has a guard of its
own, so the scheduler would never reach the slot lookup this spec is about.
Leaving every other column readable lets selection succeed and lands the
refusal exactly on NextFreeReplicaIndex, which plucks replica_index. The
grant is restored from BeforeEach via DeferCleanup, so a failing assertion
or a panic cannot hand the next spec a role that cannot read.
Reverting the eviction guard fails this spec, which is the point of it: the
router then reports "no replica slot on keeper and eviction failed" for an
error that was never evidence the node was full. The surviving-row
assertions are secondary under this injection, because the eviction path
reads whole node_models rows and the same revoke blinds it too; a comment
in the spec says so, so nobody mistakes them for the load-bearing ones.
Also documents why the vector store and the control plane must not share a
database: the removable-tuple cutoff is per database, not per table, so one
transaction left open anywhere stops autovacuum reclaiming the node
registry, and a six-row table bloats into hundreds of megabytes. The note
names LOCALAI_AUTH_DATABASE_URL and LOCALAI_AGENT_POOL_DATABASE_URL as the
two knobs that must differ, and the localai_control_plane_oldest_xmin_age
gauge as the way to see it coming.
grep for StaleNodeThreshold and HealthCheckInterval in
core/config/runtime_settings_registry.go returns no matches: the
distributed duration knobs are not exposed as runtime settings, so the new
heartbeat checkpoint interval follows them and needs no registry entry.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): close the review gaps in the heartbeat and health path
The stale-node threshold moved from 60 seconds to 5 minutes in this branch
because checkpointing makes last_heartbeat up to one checkpoint interval
behind by design. Two things were left inconsistent with that. NewHealthMonitor
still fell back to a hardcoded 60 seconds when handed a zero threshold, so any
future caller that stopped passing the configured value would mark every
healthy, beating node offline on every cycle. And the threshold itself had a
flag-name constant but no AppOption, no CLI field and no env binding, so an
operator who widened --node-heartbeat-checkpoint had no way to widen the
threshold to match. The fallback now tracks config.DefaultStaleNodeThreshold,
and --stale-node-threshold / LOCALAI_STALE_NODE_THRESHOLD is wired the same
way its sibling is.
Heartbeat suppression compared the RAW reported free VRAM against the
snapshot, but the column persists capAvailable(raw, ceiling). On any node with
a VRAM budget set, whose actual free VRAM oscillates above that ceiling, every
beat looked material while the persisted value never moved: suppression was
defeated on exactly the nodes an operator had configured, and the write
amplification this branch exists to remove came straight back there. The
comparison and the snapshot now both hold the capped figure, so they measure
the same quantity as the column.
Fixing that needs the ceiling, and reading it cost a SELECT on every beat,
including suppressed ones. The skip decision therefore moved ahead of the
updates map and now reuses the ceiling cached on the last durable write, while
the write path still re-reads it before capping anything. A ceiling that
changed inside the checkpoint window can cost one extra or one late write; it
cannot persist a wrong figure. A suppressed beat now costs no query at all.
Also: the operations section now says to grant pg_read_all_stats to the
LocalAI role, because PostgreSQL blanks backend_xmin and xact_start for
sessions owned by other roles, and the transaction that wedged the horizon in
the incident was a co-located vector store connecting as a different role, so
without the grant the new gauge sees only our own sessions. The compose
healthcheck comment now describes readiness covering the backend data path,
and the control-plane gauge registration records the otel.SetMeterProvider
ordering it depends on.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): resolve the gauge's table names through gorm
The dead-tuple gauge queried pg_stat_user_tables against a hardcoded list
of three table names. Those three do not agree on where their name comes
from: BackendNode and NodeModel take gorm's default pluralisation, while
GalleryOperationRecord overrides TableName, and gallery_operations
already had a constant of its own that the list duplicated.
A literal list keeps compiling after any of that moves, and the query
then matches nothing. The failure is silent and it points the wrong way:
a dead-tuple ratio that matched no rows reports the same numbers as a
cluster with no bloat, so the gauge would look healthiest exactly when it
had stopped working.
Ask gorm what each model is stored as instead, which follows a TableName
override and the default pluralisation alike. A spec pins that the
override really is consulted: naive pluralisation of the type would give
gallery_operation_records, so the resolution cannot quietly stop asking
the model.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Clamp requested generation to the usable context after prompt sync while preserving the legacy 256-token fallback for omitted limits.
Constrain each speculative MTP cycle to the remaining request budget so accepted tokens cannot advance beyond the visible output limit.
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
The worker re-classified a failure a local service had already classified.
classifyServiceFailure preserved exactly one of the four refusal codes,
which was faithful to its own comment for as long as there was one worth
keeping; once ErrStreamNotServed existed, a service returning the code
whose whole job is to say "I learned nothing" had it promoted to
ErrStreamTargetUnavailable, which every reap guard acts on.
ErrStreamTagUnknown was promoted too, and cost nothing only because both
sides of that one reap. No in-tree service produces either, which is the
same "unreachable, therefore safe" argument that let the request-frame
merge survive a whole phase, and LocalService is exported.
The cause was a fifth site enumerating the vocabulary by hand, so the fix
is one table. streamRefusals pairs each sentinel with its wire code and
with whether a frontend may act on it as evidence about a backend, and the
writer, the reader, IsWorkerAnswer and the new IsStreamRefusal all read it.
A fifth code is now taught to every one of them at once.
The codes are also pinned against literals written out in a spec, the way
this branch already pinned the NATS vocabulary. The round-trip table
cannot see a rename, because a rename moves the writer and the reader
together; an unrecognised code is deliberately not the worker's answer, so
renaming "unavailable" would turn every crashed backend on a tunnelled
worker into a row nothing can ever reap, silently and with the suite green.
Three comments the previous fix falsified, corrected:
- tunnelHeaderTimeout still said the window bounds only framing the
frontend writes immediately after opening the stream. That is true on the
direct path and false on the relay path, and it was the argument for
treating an expiry as the frontend's fault.
- classifyServiceFailure's deny-list is three causes, not two: on a dial
error net.Error.Timeout also covers ETIMEDOUT and EAGAIN. Both are kept
deliberately, because reaping a wedged or resource-starved backend is the
eviction this phase exists to prevent, and ECONNREFUSED still reaps.
isReadTimeout is renamed reportsTimeout, which is what it asks.
- The operator table named three refusals and said a refusal is acted on.
It now lists four, with when each is sent and whether the row is reaped.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Making a worker's refusal reaping evidence created a defect one layer
along, at the producer. The worker refused a ReadStreamRequest failure with
ErrStreamRequestInvalid and its own comment said "Includes the deadline
above expiring", which was harmless while every refusal reached the
frontend as "no route" and became a reap the moment one of them did not. So
a request frame that had merely not ARRIVED yet was reported as a
non-transient verdict about a backend.
It is reachable on the relay path, which carries most production traffic:
the worker's header timer starts when the OWNING replica opens the stream,
while the frame is written by the DIALLING replica only after the relay's
acceptance travels back to it, so a whole peer-link round trip runs inside
that window, on a link this design deliberately loads with multi-gigabyte
artifacts beside token streams. For a long-deadline caller the endpoint is
ConnectionEvictingClient, which stops the model across the fleet. It also
falsified the "neither clears on its own" argument that licensed the reap.
There is now a fourth refusal, ErrStreamNotServed, for what a worker could
not serve for a reason of its OWN. It is deliberately outside
IsWorkerAnswer, so it reaches a consumer under the no-route umbrella and
reaps nothing, which is the same treatment an unrecognised code already
gets. Four producers move onto it: a request frame that timed out (a
malformed one stays a verdict, because that is a frontend bug no retry
fixes), both SetReadDeadline failures, which are facts about the stream and
not about a target nothing has dialled yet, and WriteStreamRefusal's
default for a reason nobody classified.
classifyServiceFailure keeps ErrStreamTargetUnavailable as its default on
purpose: inverting it would make errno enumeration the single point of
failure for the reap, and a miss there is a row nothing can ever delete.
What it gains is a deny-list of two causes that are provably this worker's
own clock or its own context.
Also:
- The read-site caller-deadline guard in the handshake was unpinned: the
existing seam spends the budget before the handshake starts, so only the
write could ever fail. A spec whose deadline falls between the request and
the reply pins it, and each guard now reddens on its own.
- The documented worker-first failure line omitted the JSON error envelope
the old frontend returns, so an operator grepping it found nothing.
- The peer-link disclosure names the aimable per-session receive window in
all four places, and LastDialErrorOf records why a third consumer must go
through IsWorkerAnswer rather than roll its own list.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A worker that refuses a stream has answered, and cluster.Dial keeps the three
tunnelproto sentinels out of the ErrNoRoute umbrella precisely so a consumer
can act on that. No consumer did. Since workers stopped listening, a backend
process that crashed on a healthy worker is no longer a dead listener's
codes.Unavailable: the worker refuses the stream with
ErrStreamTargetUnavailable, gRPC flattens it into Unavailable anyway, and
nodes.unroutable reported the whole thing as "this frontend has no route".
Every reap path then answered ProbeUnknown and left the row, so the replica
slot never freed and at the default MaxReplicasPerModel=1 the only cleanup
left was LRU eviction of models that were working.
isWorkerAnswer is exported as cluster.IsWorkerAnswer, so the errors the dialer
keeps out of the umbrella are by construction the errors the consumers treat
as the worker answering. nodes.unroutable and pkg/model's transportFailure
both use it; ConnectionEvictingClient, the site reached during inference, goes
through transportFailure rather than asking the transport directly. A reply
code this frontend does not recognise is still not an answer, so a newer
worker's vocabulary costs a retry and not a replica.
The reap guards keep the allow-list rather than requiring ErrNoRoute: an
unrecognised dial error must mean "no route", never "the backend is gone".
Also in this final pass over the branch:
- Docs: recommend upgrading FRONTENDS first, with the symptom of each order.
Workers-first fails now that a 4xx registration is a verdict rather than an
outage, so an old frontend's "address is required for backend workers" makes
each restarted worker exit and drains the fleet a node per restart.
- Docs: LOCALAI_WORKER_TUNNEL=false is a fatal startup error, not a degraded
mode, in both places that described it; and a frontend rollback needs every
worker restarted, because re-registration force-clears the address columns.
- A replica with no advertised address now says so every five minutes and
names the workers only it can reach, instead of one startup warning for a
cost paid for the life of the process.
- callerRanOut's rule now holds at all three siblings, so an expired caller
deadline stops reading as a broken tunnel; probeHealth's withdrawn reason
for using the raw client is corrected; the dead DoOrCached is deleted and
its coverage kept on DoOrCachedResult; sweepLeakedInFlight enumerates the
outcomes that reach it.
- The peer route's self-declared id is recorded as a phase-3 deferral, in the
handler, in the isolation claim it narrows, and in the operator docs.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review round 1 on the end-to-end proof. Zero blocking items, eleven
non-blocking, and three of them turned out to be production defects rather
than notes on the report.
The one that matters is a misclassification the phase is built to prevent. A
dial carries the caller's deadline down to the socket, so when the budget runs
out the socket's timer fires and the error travels back up through the
WebSocket handshake and the multiplexer. The context's cancellation is a
separate timer whose func the scheduler has to run before ctx.Err() stops
returning nil, and nothing orders the two. Under contention the socket's error
is back in PeerPool.Open first, ctx.Err() reads nil, and a peer that is
listening and healthy is reported as ErrPeerUnreachable to a caller that simply
ran out of time. An unreachable peer is a fact a caller may act on and an
expired deadline is not, and core/services/nodes routes around a replica it is
told is unreachable.
callerRanOut answers that question in one place: ctx.Err() when it is set, and
otherwise the wall clock against the caller's own deadline. That is sound
because it is the same instant the socket compared itself against, so if the
socket's timer fired this comparison is past it too. The ambiguous instant
resolves towards the caller, which is the direction that never blames a peer.
The spec that caught it, peerlink_test.go's "blames the caller's deadline",
was red in three of seven -race runs and had been since Task 5, which is often
enough to read as noise and is why single-run verification never saw it. Rather
than leave the proof to a coin flip, a second spec makes the window
deterministic: Open is handed a context whose deadline has passed and whose
cancellation has not been delivered, against an address nothing is listening
on, so the dial fails for real. It reddens without the fix.
The peer link's yamux windows were applied to one end only. A receive window is
advertised by the side that RECEIVES, so configuring the dialler alone tunes
exactly one direction, and the direction left on the 256 KiB default is the one
that carries a relayed model artifact INTO the replica that owns the worker's
tunnel. That is the largest thing the link ever moves and it is the direction
the load measurement exercises: the review read it as flowing toward the
dialler and it does not. PeerLinkConfig is now exported and used on both ends.
Measured, same box, 128 MiB staged through the relay against the same transfer
without one: the relayed path cost 1.6x to 2.0x the direct path's transfer
window before, and 1.06x to 1.25x after.
The SSRF reachability spec could be fooled into reporting an SSRF that did not
happen. It bound the victim on 127.0.0.2 at an ephemeral port and required
127.0.0.1 at the same port to refuse, so any other spec in the run holding that
number made the dial succeed; red one run in seven, green five of five in
isolation. It now picks from below the kernel's ephemeral range, the same fix
the harness got for the adjacent-port collision.
The rest are the specs and the report saying what they mean.
Scenario 1's advertisement assertion could not tell "the worker advertises
nothing" from "the JSON key moved", which matters because removing the
advertisement is the change it covers. It was green against a renamed key. The
roster now keeps the raw key set beside the decoded fields and the spec
requires both keys present before reading them as empty.
Scenario 4's refusal-body check was a four-way disjunction admitting bare
"tunnel", "not connected" and "unroutable". Those alternatives were inert and
each would be satisfied by refusals that say nothing about routing, in the one
assertion the whole negative control rests on. It is "no route" alone.
The head-of-line gate bounded the worst probe by the whole transfer window,
which admits about eightfold degradation and loosens as the box slows. It is
now half the window, plus a scale-free ratio against the worst probe under the
SAME cold load with nothing to transfer, which is the control that isolates the
transfer from the load. Not tighter than that, and the reason is measured
rather than cautious: under a concurrent -race suite the worst relayed probe
reached a fifth of its window, so a quarter-window gate would have had 1.2x of
margin, and a spec that fails one run in three is worse than no spec.
The report entry printed p90 and p99 off samples of twenty, where both land on
the same element and p99 often lands on the max, so one number appeared three
times under three names. A quantile is now printed only when the sample can
separate it.
Two claims in the report were wrong and are withdrawn rather than softened.
Scenario 2's race is closed by the trailing re-read of the owner, not by the
pre-assertion the report credited: a move to the non-owner mid-request would
serve directly and still return 200, and only the trailing read reddens on it.
And "the median request is unchanged" holds on this box and not on the
reviewer's, where the relayed median rises up to 82% and p99 up to 3.5x. What
survives on both is structural: the worst probe is a small fraction of the
window in which bytes are moving, so the session interleaves rather than
serialising. Sharing a session with a bulk transfer costs latency; it does not
cost service.
The disk footprint note undercounted, and the reviewer lost a run to a full
disk on this box, so it is worth having right: two bulk models seeded into two
frontends and staged to the worker is about 768 MiB, not 512 MiB.
Left alone deliberately: the worker's backend port allocator still hands out
ports without checking they are free, and its default range still overlaps the
kernel's ephemeral range. It is confirmed, it is out of scope here, and it is
being tracked as a named follow-up rather than fixed under an e2e task.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): link librdma from the static ggml-rpc build
ggml-rpc gained an Apple RDMA transport in this llama.cpp range and
declares its librdma dependency with target_link_options(ggml-rpc
PRIVATE "LINKER:-weak_library,..."). Link options are not a usage
requirement of a static library, so the llama-cpp-grpc variant, which
builds with BUILD_SHARED_LIBS=OFF, dropped the flag and left every
ibv_* symbol of transport-apple.cpp undefined when grpc-server linked
on darwin.
prepare.sh now re-declares the same weak link as INTERFACE on the
ggml-rpc target, so the flag reaches whoever links the static library.
The append is guarded on a marker for repeat runs, and on
GGML_RPC_RDMA_APPLE, which the turboquant and bonsai forks lack.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Everything this phase built was proven by unit and integration specs. This is
the first run of it against the real binaries: a frontend replica per process,
a worker that binds nothing routable, real inference over the result.
Four scenarios, each with the question "what would make this pass if the tunnel
were doing nothing" answered rather than left open.
A worker with no advertised address is reached through its tunnel. The roster
is asserted to report it advertising nothing, so there is no address a frontend
could have dialled instead, and node_connections is asserted to name the
replica that serves the request.
A request landing on the replica that does NOT own the worker is relayed to the
one that does. With N replicas behind round robin that is (N-1)/N of production
traffic, so it gets the FIRST request for its model: the backend install, the
file staging on the http tag, and the gRPC load and predict all cross the
relay. Which replica owns the tunnel is read from the ownership table through
the production Owner query and mapped to a frontend index through the address
the harness pins per replica; the non-owner is derived from that reading and
asserted to be a non-owner immediately before the request, rather than assumed
from the harness default. Sending the same request to the owner reddens it.
Killing the owning replica re-homes the worker onto the survivor. The worker
dials a balancer rather than a replica, because LOCALAI_REGISTER_TO is resolved
once at boot and is the tunnel endpoint as well as the registration one: aimed
at a single replica, a worker has nowhere to reconnect to when that replica
dies, and the re-home cannot happen at all. Removing the kill reddens it.
And the negative control for the whole suite, which is why the other three mean
anything. Frontend and worker share a host here, so every backend port the
frontend names in a stream target is one it could have dialled directly; if it
did, the first three would pass with the tunnel inert. LOCALAI_WORKER_TUNNEL is
no longer usable for this, because it is a fatal startup error and a worker that
never started says nothing about a worker reachable some other way. The balancer
answers the tunnel connect path itself instead, leaving a worker that registers,
heartbeats, reports healthy and holds no tunnel. It is asserted to have dialled
and been refused, asserted to be held by nobody, and then asserted unreachable
with the refusal naming the missing route. Then the block is lifted, nothing
else changes, and the same request succeeds: that is what attributes the refusal
to the tunnel rather than to any of the ordinary reasons an e2e inference fails.
The fifth spec measures the head-of-line blocking this phase deferred three
times. 128 MiB crosses the session while a warm model is probed back to back,
direct and relayed. Median latency is unchanged, the worst probe is about 3x the
baseline median and about a seventeenth of the transfer window, and the transfer
runs at 415-490 MB/s direct and 222-268 MB/s relayed. A session that
head-of-line blocked would park a probe for the length of the window. Leave the
yamux windows untuned; and note this is loopback, so it says the multiplexing
does not serialise and says nothing about a link with a bandwidth-delay product.
The load spec is measured against a control that the first version did not have.
It passed with the bulk artifact cut to 4 KiB, because the window it read probes
against was mostly cold-load overhead: it would have reported a clean bill on a
session carrying no large message. The same cold load now runs twice, once
empty and once bulk, and the difference between the windows is asserted to be
real before any latency is read from it.
Two defects on the base commit came out of this.
cluster_peerlink_test.go has been red since the relay landed, deterministically,
in isolation and in the suite. It asserted that an accepted peer stream is
refused at once, on the premise that phase 1 installs no relay. The relay
correctly waits fifteen seconds for a frame naming the worker, and the spec's
budget was five. It now writes a relay request for a node no replica holds and
asserts the refusal is ErrNotOwner and specifically not ErrNoConnection, which
is a stronger spec than the one it replaces and the only thing in the e2e suite
that exercises the relay's refusal path.
The harness handed a worker's own HTTP port to a backend process. It took two
ports from freeport and used one as the gRPC base and the other for the file
transfer server; freeport returns adjacent ports often, and the backend
allocator hands out base, base+1, base+2, so the second backend started on a
worker was regularly given the HTTP server's port and died with EADDRINUSE. No
spec had started two backends on one worker before, so it had never fired; the
load spec starts five and it failed about one run in three. Each worker now
reserves a contiguous bind-probed block laid out the way production lays it out,
below the kernel's ephemeral range, with LOCALAI_GRPC_MAX_PORT bounding the
allocator to it. The underlying production defect is not fixed here and is
recorded in the report: allocatePort never checks that a port is free, and its
default range overlaps the ephemeral range on every Linux box.
Constraint 6, whether distributed mode should now refuse to start without an
advertised address, is DEFERRED, and the comment and the docs that described the
cost were understating it. A replica with no advertised address writes no
instances row, and Owner joins a connection against a live instance, so a worker
whose tunnel lands there is unroutable from every OTHER replica while being
registered and healthy. Refusing to start would still be wrong, because the
deployments it would break are single-host ones with no peers to be unreachable
by, and telling those apart at startup is a design with its own specs. Both
places now say what actually happens.
Suite wall clock 592s for 15 specs, up from 502s for 10 of which 2 were red. The
CI budget of 20 minutes does not move.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review round 1 on the change that stopped workers listening. One blocking item
and seven notes.
LOCALAI_WORKER_TUNNEL=false was the blocking one, and the ruling was to make it
fatal rather than to correct the comment that still promised it fell back to the
advertised address. There is no fallback left: a worker on this branch
advertises nothing and binds only loopback, so turning the tunnel off leaves it
reachable by nothing while it registers, heartbeats and reports healthy, and the
scheduler keeps placing models on it. That is the worst available failure shape,
so a new Config.validateStartup refuses it before prefetch, registration and
NATS, while the worker is still invisible to the cluster. It absorbs the
pre-existing empty-registration-token check, which had the same shape and no
spec. The flag is kept rather than deleted so an operator who set it is told the
promise is gone instead of having the setting ignored, and the guard around
StartTunnel is removed, because a branch nothing can take reads as a supported
no-tunnel mode that does not exist.
The justification for erroring on an install that names no address was wrong,
and the review is right that this is the dangerous form of overclaiming, because
the conclusion holds and the mechanism does not. It said the resulting empty
target would be refused as an invalid stream and that the refusal would read as
the worker answering about its backend. Nothing in this repo branches on
cluster.ErrNoRoute, and nodes.unroutable treats any recorded dial error as
unroutable, so that refusal reaches every reap guard as ProbeUnknown and deletes
nothing. The site now stands on what holds, that an install naming no port
produced nothing routable and the failure belongs to the install rather than to
a later probe, and records the retracted claim so nobody re-derives it. This
retracts the same paragraph in the body of 1cf847f29.
The reviewer deleted the whole tryWarmPath unnamed-replica guard and the suite
stayed green, including the reservation release. It is specced now, and the
asymmetry the review asked about is decided at the site: the row stays, unlike
the sibling !alive branch which removes it. That branch has observed a backend
dead; this one has observed only that the row is unreadable, which says nothing
about whether a process is running, and the row is the last record that one
might be, since the acknowledged stop path refuses a stop whose ExpectedAddress
does not match and an empty one cannot be cleaned up through it either.
The cross-version wire claim rested on two struct tags nobody asserted:
renaming only the json keys survived mutation while the gorm column rename went
red through raw SQL. Both keys are pinned now, marshal and unmarshal, per
struct.
A worker-first upgrade showed the operator a status code and not the reason. The
registration client discarded the body, so "address is required for backend
workers" was read off the socket and thrown away, and the ladder then spent four
minutes on a verdict the frontend reached instantly. Refusals now quote the body
and carry ErrRegistrationRejected, and both the ladder and the credential
manager's Acquire stop on the first one. Acquire matters more than the ladder:
it is the default path and its bound is 100 attempts, not 10. 408 and 429 are
deliberately not refusals, since both are the frontend asking for the same
request again.
Also: the stale "not blocked by firewalls" troubleshooting line, which now names
the real cause and the knobs that move the port range; and the inert address
fields on the MCP Node DTO, which the Assistant was still being handed. The
review named http_address there and I removed address too, because it is inert
by the same argument and leaving one of a pair is arbitrary.
Five mutations, all red. Deleting the warm-path guard reddens four specs and
falsifying only its reservation release reddens one, so the two halves are
pinned separately. Renaming only the json keys reddens both wire suites.
Discarding the refusal body reddens two. Dropping the rejection classification
does not fail the suite, it hangs it, which is the operator-visible symptom, so
it is recorded red under a ginkgo timeout.
The verify list is now derived from the diff rather than from the brief, which
is what let the previous round ship a spec asserting 200 where the endpoint
returns 201: nine ginkgo suites, the e2e vet, route auth coverage, the leaf
check, build, the healthcheck shell suite and lint. The two jsx files have no
harness in this worktree and are recorded as the one unverified surface.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A worker now opens no listener on a routable interface and states no endpoint at
registration. Backend processes and the file-transfer server bind loopback, and
the frontend reaches both through the tunnel the worker dials. The bind address
is built from loopbackHost, the same constant the tunnel's grpc tag dials, so
"the worker binds where its tunnel dials" is one fact in one place rather than
two literals that can drift.
All three advertisement sites are closed, not one: the registration body,
RegisterNodeRequest, and the per-backend address in the install reply.
That third one was hiding a live bug. stopModelExact refuses a stop whose
ExpectedAddress does not match what the worker recorded for the process. The
worker recorded 127.0.0.1:port; handleBackendInstall reported advertiseHost:port;
the router stored the reported one and sent it straight back. On any worker whose
advertise host was not 127.0.0.1, every acknowledged model stop failed with an
address mismatch. Nothing caught it because the e2e harness set
LOCALAI_ADVERTISE_ADDR=127.0.0.1, which made the rewrite a no-op. Removing the
rewrite makes the two strings the same by construction.
The brief was wrong about two of the four functions it called dead.
effectiveBasePort is the base of the backend port allocator and resolveHTTPAddr
is the file server's bind address; deleting them would have deleted the port
allocator and the file server. Only the two advertise* helpers were dead, and
addr_test.go is rewritten rather than deleted, because the port arithmetic it
pinned still needs pinning.
NodeModel.Address survives with a narrowed meaning and is renamed
WorkerLocalAddress, along with the install reply field that feeds it. The
frontend still has to say WHICH backend process on a worker it means, and the
port in this string is how it says it: it travels as a stream target and the
worker dials its own loopback. The gorm column and the json key stay "address",
so neither a migration nor an API break rides along. Every fall-back to the
node's address is gone. installBackendOnNode now errors when a worker reports
success without naming one, because substituting the now-always-empty node
address would name an empty target, and the worker refuses that as an invalid
stream, which is classified as the worker answering about its backend. That is
the "a present worker reads as something it is not" class this phase forbids.
DistributedModelStore.Range had the same shape and was already wrong: it built
each remote model's client from the node's base gRPC port, never the port a
backend process listens on, so Free and Status went to the wrong place. It uses
the replica's address now.
BackendNode.Address and HTTPAddress are kept but made provably inert: no writer,
no reader that acts on them, and Register force-clears both on re-registration so
an upgraded worker's stale advertisement does not outlive its own upgrade in the
API and the Nodes page. Dropping the columns is a ~90-site edit across the specs,
the e2e suite, the MCP dto and the UI; it is recorded as a follow-up rather than
folded in here.
A persistent tunnel 401 still does not trigger re-registration, and now for a
reason rather than a deferral. Register CLEARS the node's replica rows, so
re-registering on a 401 would delete a live worker's rows on every retry, and
under the name collision that causes the 401 the two workers would take turns
doing it forever: a credential failure causing model reclamation. It also cannot
fix the named cause, since a collision is indistinguishable from a restart. The
401 log now names both causes and says nothing can reach this worker, which is
true only now that it has no listener.
The container healthcheck did not break the way the brief expected, since the
listener still exists on loopback and the probe runs inside the container. It did
have a real #10987 defect that this change makes the common case: it read
LOCALAI_SERVE_ADDR only, while effectiveBasePort reads LOCALAI_ADDR first, so a
worker on a non-default base port was probed on 50050 and reported unhealthy
while working. It follows the same precedence now.
Docs, the compose file and the e2e harness are updated in step: no inbound rule
or published port is needed for a worker, the two advertise variables are gone,
the remaining address variables are read for their port only, the
firewall-the-file-transfer-port warning is narrowed to the LOCALAI_HTTP_ADDR
opt-out, and the upgrade-order note no longer claims the worker still listens.
The Nodes page showed node.address, which is now always blank, so it shows the
node id instead.
Eight mutations, all red on a named spec, including reverting the loopback bind,
re-adding the address to the registration body, restoring both node-address
fall-backs, dropping the force-clear, storing the endpoint's address again, and
un-fixing the healthcheck. One of them caught a defect in a spec I had just
written: it asserted 200 where the endpoint returns 201, which went unnoticed
because core/http/endpoints/localai is not on the task's verify list. It is run
here.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
I reported that enabling gocritic pushed make lint past 600s. That was wrong,
and it was wrong in a way worth naming: those runs happened right after I
changed pkg/grpc's and core/services/nodes' interfaces, so the Go build cache
was cold for essentially the whole repository including every backend, and test
suites were running concurrently on the same machine. I attributed a cold-cache
full-repo typecheck under load to the linter I had just enabled, and raised it
as a cost without ever timing it against a baseline. A number with no control is
not a measurement.
Measured properly, with the golangci cache cleaned before every run and isolated
GOCACHE directories for the cold ones so the shared cache was not wiped: warm,
base 15s then 7s and current 8s then 7s; cold, base 87s and current 78s running
base first, base 136s and current 79s running current first. The spread between
the two cold base runs is larger than any gap between base and current, so
gocritic with only the ruleguard checker costs nothing measurable.
So the rule stays, unscoped. Scoping it to core and pkg was the fallback for a
cost that does not exist, and adding that configuration would buy nothing.
The one override gets the protection it needs instead. InFlightTrackingClient's
nolint is exactly the kind of thing a later reader tidies away, so it now opens
by saying not to, and states what breaks rather than what is intended:
WrappedBackend embeds the full Backend interface, so adopting it there would
promote every inference method as untracked pass-through, the build would stay
GREEN, and in-flight accounting would silently stop covering whatever was added
next. WrappedBackend's own doc carries the counterpart warning so a reader
arriving from either side finds it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Re-review round 2. One blocking item, and it was a spec I wrote: eight
goroutines raced at the probe cache and nothing made them coalesce, so a
straggler that missed the flight re-entered the probe and double-closed a
channel. It panicked about one run in three and took the four-suite race block
down. The green verification I reported was not reproducible, which means one
green run was never evidence for a spec that coordinates goroutines. Its comment
claimed the probe blocked until every goroutine was inside flight.Do, and that
gap was exactly the panic: the comment described the design intended rather than
the one written.
It is deterministic now rather than tolerant. singleflight.DoChan registers its
channel on an in-flight call under the group's own mutex and returns without
running its function, so calling it while the leader is provably parked inside
the probe joins that exact flight with no window and no dependence on the
scheduler. The spec asserts the join really happened, that the joiner got the
reason and not only the answer, and that the probe ran once; the entered channel
is sent on rather than closed so a second probe fails an assertion instead of
panicking. Twenty runs green under race against the committed code, five out of
five red on the mutation back to a closed-over variable.
The future-decorator gap is closed in the lint gate, but not the way the review
suggested, and the reason is worth recording. HasMethod rejects inline
signatures outright, its method-reference form needs a package ruleguard's own
typechecker can import and that typechecker cannot import this module, and
Implements tests the value method set while every Unwrap is on a pointer
receiver, so it fired on all three wrappers that already had one.
So the safe shape is structural instead. grpc.WrappedBackend gives the same
pass-through method set plus Unwrap on a value receiver, and a decorator that
embeds it is transparent by construction; forgetting stops being expressible
rather than merely discouraged, which is the move loopbackService already makes
in the worker. FileStagingClient and ConnectionEvictingClient embed it and their
hand-written Unwrap methods are gone. The ruleguard rule then only has to catch
the raw embedding, needs no type filter, and cannot misfire. It was verified to
fire on a throwaway wrapper and stay silent on a correct one, and reports
nothing across core and pkg with the baseline disabled.
InFlightTrackingClient is the one exception and says why in a nolint: it embeds
ControlBackend deliberately so that leaving an inference method unwrapped breaks
the build, and WrappedBackend embeds the full interface, so adopting it would
silently restore pass-through for every inference method and delete that
guarantee.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Re-review round 2. One blocking defect, and it was the concern I filed myself
last round and mis-scoped as a future trap. It was live, and it sat on the most
destructive reaping path of the five.
RouteResult.Client is an InFlightTrackingClient, over a FileStagingClient when a
stager is configured. model_router puts that on the cached remote model and
pkg/model's checkIsLoaded asks IT whether the transport failed. Both wrappers
embed grpc.Backend, which does not declare LastDialError, so the type assertion
read nil and the guard added last round fell straight through to the old
eviction. That eviction sends backend.stop over NATS to every node holding the
model and deletes every replica row, where the other sites delete one. The spec
covering it built a bare client by hand, which is why it passed while production
did not.
This is the third time in this task a correct fix was disarmed one layer out, so
the fix is a mechanism rather than two methods. BackendUnwrapper is one line per
decorator, LastDialErrorOf walks the chain, and both consumers now call it
instead of each keeping its own assertion. One implementation, no per-caller
policy to get wrong.
Sweeping every type that embeds or holds a grpc.Backend found a third decorator
the review had not named, and it is itself a reaping consumer of the same
collapsed signal. ConnectionEvictingClient is built for remote models in
initializers.go and its evict callback runs ShutdownModel; it fires during
INFERENCE rather than on a health check, so a tunnel blip mid-request was enough
to stop a model that was loaded and serving. It consults the transport first
now. A locally spawned backend has no custom transport, so that path is
unchanged byte for byte. Everything else touching a Backend is a consumer rather
than a decorator; there is no fourth.
The probe cache joiner shape is pinned. It was the right design last round with
nothing holding it: the mutation back to a closed-over variable passed all 602
specs in the package. Eight goroutines coalesced on a probe that blocks on a
channel now assert every joiner gets the leader's REASON and not just its
answer, which is the difference between a leader declining to reap and its seven
joiners reaping on the leader's own observation.
The LastDialError scope note claimed an exactness it does not have at
checkIsLoaded, which reads a shared long-lived client after releasing opMutex.
It now says which caller is not exact, why the imprecision is accepted there,
and what making it exact would cost.
The four-outcome table in the docs still said a worker with no live owner is
treated as absent and rescheduled, contradicting the code and the paragraph nine
lines below it. None of those outcomes is absence any more, and the table says
so, names the fifth, and points at the heartbeat as the thing that does decide
presence.
Five mutations, each reddening named specs, including the two the reviewer found
surviving.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review round 1 on task 6. Five blocking findings, all with the same root: the
conditions the dialer kept apart were erased one layer out, because every one of
them arrived at core/services/nodes as a gRPC codes.Unavailable, which is also
what a backend process that died produces. Four call sites acted on that by
deleting a replica row, one of them after a single failed probe.
The fifth condition is ErrNoRoute: this replica could not get a request to a
worker's backend, and no claim at all about the worker. A worker's presence is
its HEARTBEAT, which nodes owns; a route is a separate fact that cluster owns,
and the two now differ. They differ in normal operation, not exotically: a
worker that has not dialled its tunnel yet after a frontend-first upgrade is
unroutable on every request while it heartbeats and serves.
Two properties, both mutation-tested. Every failure to resolve or open a route
carries ErrNoRoute, so a consumer has one check to make. No failure carries an
absence sentinel: routeFailure is the single place that rule lives, and it keeps
ErrNoConnection and ErrInstanceNotFound in the message and out of the unwrap
chain, the guarantee unreachableError already made for peers. Everything else
stays matchable, so ErrNotOwner and ErrPeerUnreachable are unchanged for anyone
who can act on them. A worker's own refusal carries no umbrella, because a
worker that answers has demonstrated it is there and that is the only real
evidence on the path.
Crossing the boundary needed a value, not a code. NewClientWithDialer wraps the
dialer and records each outcome; LastDialError hands it back behind a narrow
interface, and nodes.unroutable turns it into ErrWorkerUnroutable with the
cluster sentinels still in the chain. A spec asserts a dial failing with
ErrNoRoute plus ErrPeerUnreachable arrives matching all three and matching
neither absence sentinel.
The sweep found a fourth site the review had not named: pkg/model checkIsLoaded
evicts a remote model on a connection error, and a tunnel dial failure is one.
Four other reap sites were cleared with reasons - inflight and the worker
authoritative pass reap only on semantic answers, scale-down is driven by
last_used, abandoned loads decide on the node's heartbeat. Every fixed site also
grew the opposite spec, so the new check cannot pass by never reaping.
probeCache carries the reason through singleflight rather than a closed-over
variable. A variable is only written by the goroutine that runs the probe, so
the leader would correctly decline to reap while every joiner reaped on the
leader's own observation; a mutation reproduces exactly that.
The docs sentence promising LOCALAI_WORKER_TUNNEL=false restores direct dialling
is gone. There is no such path, so it said the operator could take a worker dark
and call it a rollback. Replaced with the upgrade order that is actually safe.
The deadline spec the reviewer found vacuous now waits on the dial context's own
Done channel before touching the stream, so the armed deadline has really
expired; the mutation that survived for the reviewer reddens it.
Nine mutations, each reddening a named spec, including both halves of
isAbsenceClaim independently.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The tunnel, the fence, the registry and the relay were all built and none of
them carried a byte: every dial from the frontend still went to the address a
worker registered. This is where that stops. One WorkerDialer resolves where a
worker's tunnel is held, opens a stream on it locally or relays through the
owning replica, and hands back a conn past both handshakes; gRPC, the file
stager's HTTP client and the log-streaming WebSocket are all pointed at it.
A worker's address stops being somewhere to connect to and becomes the name of
which backend process a stream is for. It still appears in URLs, logs and
errors, because that is what identifies the process; what it no longer decides
is where the bytes go.
Nothing falls back to dialling it. BackendClientFactory now has exactly one
method, NewClientForNode, and returns an error where there is no way to reach
the worker. The direct-dial constructor was removed rather than kept beside it,
because leaving one on the interface keeps the bypass one word away from every
call site that holds an address, which is all of them.
The second construction path is closed too. DistributedModelStore built remote
models with a nil client, and pkg/model.Model.GRPC then dialled the raw address
lazily on first use - reached in production by ShutdownModel's Free and by the
backend monitor's Status. Those models now carry the tunnel-backed client, and
a model that cannot be given one is logged and not listed.
Four conditions stay unmixable, and one path produces absence: the dialer
answers ErrNoConnection only where Owner's liveness join did. A peer that will
not answer, a stale ownership row, a worker's own refusal and a missing relay
path are each reported as themselves. This matters because nodes ACTS on
absence, and the collapse would have it reclaim the models of a worker that is
connected and busy.
That is not hypothetical. Writing the mutation for it exposed the bug in this
change's own first draft: probeHealth returned bare false when it could not
build a client, and tryWarmPath deletes the replica row on a false probe. A
frontend whose dialer broke would have emptied node_models for the whole
deployment while every model kept running. probeHealth now returns alive and
probed separately, the reconciler gets a ProbeUnknown outcome that neither
advances nor clears a failure streak, and the health monitor skips rather than
counting a miss.
Task 5 left the relay's open timeout at a fixed 15s and said so: no operator
has the information to set it, because the number that matters is the original
client's remaining budget, which is invisible on the relay side. The dialer has
that budget, so it now states it in the relay request frame and the owner takes
the smaller of the two. It can only shorten - a patient client must not be able
to park a relay goroutine and a stream slot on a worker that stopped accepting.
Zero is written as no budget at all, since on the far side the number zero is a
caller with nothing left and would refuse healthy traffic.
Seven mutations, each reddening a named spec: peer-unreachable as absence; the
local-failure guard dropped; max instead of min on the budget; the nil-client
model restored; ProbeUnknown falling through to the reaper; OwnerRow instead of
Owner; probed collapsed into alive. The first budget spec passed for the wrong
reason - a handshake deadline, not the relay - and was replaced by three that
each assert one link, including one where the spec plays the owning replica and
reads the budget out of the frame instead of inferring it from a clock.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review round 1 on task 5. Eight non-blocking items, all addressed.
The classifier read Remote in two predicates with a report-by-default
fallthrough behind them, so reverting either read left the whole suite green:
the error reached the same answer down the other path. A correctness argument
that rests on mutation evidence cannot afford a shape that cannot be mutated in
pieces, so the two predicates collapse into one muxVerdict deciding each error
type once. Falsifying either Remote read now reddens exactly one spec.
Three claims the comments made loudly and nothing tested:
- clearing the header read deadline before the splice. Deleting the clear
left all 49 focused specs green, while in production it is the difference
between a relayed response that streams for an hour and one that dies after
fifteen seconds of quiet;
- the open budget bounding the open and nothing after it;
- closing the worker-side stream when the acceptance reply cannot be
written, which leaks one stream on the worker per failure.
All three are pinned now. The first two share a spec that sets both budgets to
50ms and then watches the conversation outlive them by ten times, which is an
assertion about an event that must not happen and so is the one wait a channel
cannot replace. The third drives the relay with a peer stream that delivers a
request and then fails every write, because no pair of live yamux sessions can
be made to fail that write on cue.
The disjoint-vocabulary argument was specced for the accepted frame only. Both
refusal directions are covered now, and asserted as "not one of the other hop's
sentinels" rather than merely "an error", since reading a relay refusal with the
tunnel's reader always errors and the question is whether it errors as the wrong
thing.
The open budget stays non-configurable, and says so: the number that matters is
how long the original client will wait, which is not known on this side and is
not something a deployment-wide constant can stand in for. The honest fix is the
caller's remaining budget travelling in the request frame, which belongs to the
dialler that has the budget.
Two comments corrected: nothing deadlines the peer stream after the clear, so
the tunnelled protocol's own deadlines cannot be what justifies clearing it; and
the membership sweep deletes departed replicas but reports only how many, so
identifying them is work that would have to be done, not knowledge waiting to be
plumbed. Recorded at muxVerdict: a remote RST that does not ride a
typeWindowUpdate frame yields the bare sentinel and is still silenced, which is
unreachable between two go-yamux peers but keeps the new rule from reading as
unconditional.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
A worker holds ONE tunnel and it lands on ONE frontend replica, so with N
replicas behind a load balancer roughly (N-1)/N of requests arrive somewhere
that cannot reach the worker directly. This is the piece that carries them:
the SessionStore stream handler reads which worker a peer's stream is for,
opens a stream on the tunnel this replica holds, and splices the two.
Splice has had no production caller since phase 1. It has one now, and being
the first caller it settles the two endings phase 1 deliberately left open,
both of which read as normal termination until now:
- a peer-initiated *StreamError{Remote: true}, which yamux builds only from
an RST frame the far side sent (stream.go:432-449); a reset this side asks
for carries Remote: false, and Splice never resets anything, its own Close
sending a FIN;
- a graceful ErrRemoteGoAway, which handleGoAway returns for code
goAwayNormal (session.go:829-833) and close hands unwrapped to every live
stream (session.go:328-337).
Both truncate whatever was in flight. Reporting them as normal termination is
how a half-finished inference comes to look like a short one that completed,
so both are now reported; the local forms stay silent, because those are the
teardown Splice provokes itself. The decision cannot live in a caller reading
Splice's result, since a result already mapped to nil carries nothing left to
reclassify, so it lives at the classifier with the reasoning beside it. The
relay logs it at debug: a client cancelling a relayed request produces one per
cancellation, and the truncation is separately visible to the frontend's own
gRPC or HTTP client.
The relay hop gets its own request and reply frames. They have to be distinct
from the worker tunnel's, because a relayed stream carries both hops' frames
back to back, and a vocabulary shared between them would let a reader applied
to the wrong hop hand back a plausible sentinel belonging to the other. Its
three refusals stay apart for the reason the worker's three do: ErrNotOwner is
a routing fact and the caller should resolve the owner again; unavailable is
infrastructure at this replica and a retry is worth something; bad-request is
the caller's bug. None of them is, or may be built over, an absence error.
One hop, always. A stream naming a worker this replica does not hold is
refused, never resolved and relayed onward, so a stale ownership row cannot
become a loop between two replicas each certain the other holds the worker.
PeerPool is constructed and closed alongside SessionStore, so both halves of
the peer mesh now have an owner and a shutdown.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Re-review follow-up, three items. Two are the overclaiming-comment class again,
and the first is that class with a real defect underneath it.
attachTunnelToken said "enforcement is therefore structural": an ineligible node
never gets a credential, so its hash stays empty and the tunnel route's
empty-hash branch does the refusing. That was true for a node that had always
been an agent and false for one that had not. Register upserts by NAME, so a
backend node re-registering as an agent keeps its ID, and Register's struct
Updates zero-skips the credential column while writing the new node_type. The
early return left the credential the node earned as a backend sitting on a row
that is now an agent, and ConnectHandler never looks at node_type.
Fixed by making the claim true rather than by softening it, because the mint-site
gate was chosen precisely on the grounds that it was structural: an ineligible
node now has its column CLEARED, unconditionally, so the invariant does not
depend on what the row happened to contain. A spec pins it and was red before the
change. Same shape as the Register-upserts-by-name hazard already carried
forward: a name is not an identity.
Second, loopbackHost claimed to be the only host any tunnel stream is ever
dialled on. It is not: fixedService dials whatever Run built it from, which is
this worker's own LOCALAI_HTTP_ADDR, and loopbackAddr rewrites only a wildcard
bind, so an operator who binds the file-transfer server to a routable address
gets a routable dial. The property that matters is narrower and is what the
comment says now: the frontend cannot STEER the dial. The grpc tag builds its
address from a constant and a validated port with nothing from the wire reaching
the dialler, and the http tag ignores its target entirely. Worth stating exactly
rather than summarising, because the argument about what a stream can reach rests
on knowing which hosts are reachable, and an overstatement at that site is what
would let someone conclude the constant alone is doing the work.
Third, a spec named "without allocating it" measured no allocation. It now
asserts the mechanism the defence actually rests on, that the reader consumes the
two length bytes and not one byte of the body, through a counting reader. The
input carries a body on purpose: against input that ends after the header the
assertion would pass with the limit check deleted.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
Review follow-up. One blocking finding and seven others.
The blocking one first, and it is this project's recurring shape: the untested
path. loopbackService is the function whose comment calls the discarded host
"the security property this function exists for", and nothing tested it. The
reviewer replaced its body with a dial of whatever the frontend named, no port
range, and all 131 specs passed. Every spec installed the permissive test
dialler, so the real routing table was exercised nowhere.
It now has specs, and the property is stated as reachability rather than as a
property of the code: a listener on 127.0.0.2 that only the frontend's target
names must NOT be reached. Plus the port-range table, fixedService, loopbackAddr,
tunnelEndpoint, and the table itself, which moved out of Run into tunnelServices
so it can be built without starting a worker. One spec drives a real stream
through that table over the wire, so the routing rules are exercised end to end
at least once rather than only in isolation. The reviewer's mutation now reddens
ten specs, and six narrower ones redden between two and four each, so no spec is
riding on another.
The shape changed too, not only the coverage. The dial address is built from a
loopbackHost constant and strconv.Itoa of a validated int, so nothing derived
from the wire reaches DialContext at all: restoring the hole takes ADDING a data
flow, not deleting a check.
And a taxonomy fix found while specifying it. A port outside this worker's
allocator range was reported as unavailable, which tells a frontend to retry
something that can never work. It is a bad request now, and a backend that is
merely not listening yet stays unavailable, which is the retryable one.
Agent nodes no longer get a tunnel credential. Nothing dials into an agent
worker, so a tunnel replaces nothing for it and no client would open one, and
the gate is at the mint site rather than in the handler: with no credential
minted the hash stays empty and the existing empty-hash refusal covers it, so
enforcement is structural.
Two comments and one doc paragraph said an anonymous registrant gets a "working"
credential. With auto-approve off the node is pending and the credential is
inert, which is the distinction this same change argues three files away to
justify minting for pending nodes at all.
A refusal reason over the frame limit was cut on a byte boundary and could split
a rune. It cuts on a rune boundary now, and the code survives truncation, which
is what keeps a refusal classifiable.
Also: the pending-node spec asserted only that a credential was non-empty, so a
credential derived from the shared token passed it; it now pins per-node-ness the
way the headline spec does. The tunnel handler's citations into nodes.go were
stale before this branch landed, having been written against a file the same
commit was editing, and are by function name now. The static-NATS path says
plainly that an externally forced rotation locks it out until restart, and where
that gets fixed. tunnelproto gained direct specs, including that a read failure
is never reported as a refusal.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
The worker end of the tunnel. It dials wss://<register-to>/api/cluster/connect,
holds one yamux session as the CLIENT, and serves every stream the frontend
opens on it. Nothing dials into the worker, which is the point: no inbound port,
no reachable address.
Each stream opens with a length-prefixed frame naming a tag and a target, and
the worker answers before either side speaks the tunnelled protocol. The reply
is sent on every stream, not only on refusal, because the protocols carried here
are client-speaks-first and a reply sent only sometimes would arrive interleaved
with a response body. Two tags today: grpc reaches a backend process, and only
on 127.0.0.1 within this worker's own backend port range, because a tunnel
terminates inside the worker and letting the frontend name a host would make
every worker a proxy into its own LAN; http reaches the worker's file-transfer
server, whose address the frontend is not asked about.
An unknown tag, an unreachable local service and an unparseable request are
three refusals and stay three on the wire. A frontend gives up on the first and
retries the second. Each is answered AND the stream is ended: a worker that says
why and leaves the stream open has parked the caller on a request nobody will
answer, and a deadline on the far side cannot tell that from a slow worker. The
specs assert the stream ends rather than that an error occurred, which is what
phase 1 shipped in three places and held in none.
Reconnects double from 500ms to a 30s ceiling, each wait drawn between half the
interval and all of it, and the interval returns to its floor only after a
session that LASTED. Resetting on connect is how a rolling restart, where every
dial succeeds and dies moments later, becomes a retry storm against the first
replica back up. Nothing is assumed to survive a reconnect: the credential is
read at dial time, never captured.
And the credential is now real. The tunnel endpoint advertised authenticating a
worker against its own secret, but registration stored the hash of the shared
registration token, so a leak plus a known node ID still opened a tunnel.
Registration now mints a per-node secret, returns the plaintext once as
tunnel_token, and stores only its SHA-256 in a new column; the endpoint compares
against that and does not fall back to the old one. Rotating on every
registration follows from storing only the hash, since a re-registering worker
cannot be told the secret it already holds; its live tunnel is unaffected,
because the credential is checked when a tunnel is dialled and never again.
Unlike the agent API key and the NATS JWT next to it, the credential IS issued
to a node awaiting approval: the tunnel route re-reads the node's status on
every dial and refuses a pending one, so it is inert until an admin acts, and
withholding it would strand every worker that registers exactly once.
A node that has not registered since this change cannot tunnel, and the column
cannot be back-filled because the plaintext only ever existed in the response
that minted it. The boot warning that said tunnels need LOCALAI_REGISTRATION_TOKEN
is replaced: it was true while the tunnel authenticated against that token's
hash, and says the wrong thing now. What is still true, and is what it warns
about instead, is that without one, registration itself is unauthenticated.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The guard added last commit was circular. It cleared the maintenance database's
timeouts by executing SET statement_timeout = 0 on a connection that had already
inherited that database's bound, so the statement clearing the bound ran under
the bound it was clearing. Under the white-box spec's deliberate 1ms that gave
it 1ms, and it failed roughly once in fifty at 8-way concurrency with SQLSTATE
57014. The guard against invisible load-dependent flakes had become one.
The clearing is now delivered as a connection startup option, options=-c
statement_timeout=0 -c lock_timeout=0 on the maintenance DSN, so there is no
statement left to abort. Raising the imposed bound would only have bought
headroom and left the circularity in place. pgx puts every URL query parameter
into settings, options is absent from notRuntimeParams so it becomes a runtime
parameter, and runtime parameters are copied into the startup message
(pgconn/config.go:340-378, 606-617; pgconn/pgconn.go:382-388).
The spec now discriminates on pg_settings.reset_val, the value in force when the
connection started: 0 for a startup option, 1ms for a session SET. A first
attempt using a deliberately slow first statement did NOT discriminate, because
under the circular design the SET is itself the first statement, so by the time
a spec runs anything the session is already unbounded. Reinstating the circular
clearing now reddens the spec deterministically rather than intermittently.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Two advisory-lock specs named their database by literal, ALTER DATABASE testdb.
Once the test helper started handing every spec its own database on a shared
server, that statement landed on the maintenance database and did nothing to the
one the spec was holding, so both specs went green having never reproduced the
condition they exist for. They regress a model-load advisory-lock wedge that has
already shipped to production once, so the previous commit's de-flaking silently
disarmed a regression test for a real deployed bug.
Both sites now read the name back with current_database() and, more importantly,
assert the override actually landed before relying on it. A literal name can go
stale again; an assertion that the setting is in force cannot pass while it is
not. Removing either production override now fails the matching spec with the
real 55P03 and 57014 again.
That literal also meant every CREATE DATABASE and every DROP ... WITH (FORCE)
ran under the 300ms bound it set on the maintenance database, which is a new
load-dependent single-spec flake inside the change that was meant to remove one.
The helper's maintenance connections now pin one connection and clear both
timeouts on it, so no setting a spec makes can bound them, and a white-box spec
imposes the leak deliberately and proves it does not reach them.
Also pins the reclaimOne gate deferral the previous commit added without a test,
by panicking inside the re-claim's own claim statement, and drops the per-dial
empty-token log line to debug now that the boot warning says it once.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review follow-up. Twelve findings, none blocking, grouped here by what they
protect.
Panics. The handler now recovers between the WebSocket upgrade and the
hand-off, the way the peer link next door already did: net/http recovers the
panic but leaves the hijacked socket open, so without this a worker keeps a
session this replica has no entry for and will never detach. The claim gate in
Attach and reclaimOne is now released with defer, so a panic under Claim cannot
wedge one node's gate for the life of the process. SetTunnels gained the
nil-receiver guard its sibling Stop has.
Operability. A deployment with no registration token stores an empty token_hash
on every worker, so every tunnel dial 401s forever on a frontend that looks
correctly configured. That now warns at startup, logs its own line rather than
sharing the "wrong token" one, and is stated in the docs together with the fact
that setting the token later needs the workers to register again.
Authorization. A node still awaiting admin approval is refused with 403. The
rest of /api/node/ gates on nothing, but the two places that hand a node
something durable, its API key and its NATS credential, both refuse a pending
one, and a tunnel is that kind of grant. Draining and unhealthy nodes keep
their tunnels on purpose.
Comments that claimed more than the code. The global auth middleware does run
on this path and then declines to reject; the future per-node secret only lands
without a change here if it lands in TokenHash; the empty-hash guard is
defensive rather than deciding; ClusterPathPrefix is no longer only
replica-to-replica; the docs no longer say a reaped replica re-claims
unconditionally.
And the test harness. SetupTestDB started a PostgreSQL container per BeforeEach
with a readiness deadline it asserted on, which is one chance per spec to fail
one spec inside its setup, anywhere, never twice in the same place: the shape of
the flake seen twice here and never reproduced. It now starts one container per
process and creates a database per call, which is the pattern tests/e2e already
proved. Isolation is unchanged and is now asserted for the first time. All 69
call sites are untouched; the eleven consumer packages run 1404 specs green, and
jobs went from 34.3s to 3.3s, agents from 13.8s to 1.9s, cluster from 97.4s to
37.5s.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
A worker needs no inbound port: it dials GET /api/cluster/connect, the
connection becomes one multiplexed yamux session, and the frontend opens a
stream on it per request. This adds the endpoint that accepts that dial and
attaches it to the tunnel registry.
The dial is authenticated against the NODE's own stored token hash rather than
the deployment's registration token. That is the mechanism, not yet the
isolation, since a worker still registers by presenting the shared token; what
it rules out is the shortcut of comparing against the configured value, which
would have to be unpicked the day workers get their own secrets.
Every refusal happens BEFORE the WebSocket upgrade, so a dialer reads an HTTP
status rather than a handshake error. The route is registered in every
deployment, single-binary ones included, which is what puts it in front of the
route-coverage test that holds that rule in place; with no node registry it
refuses every dial, and tells a credentialed one the frontend has no cluster
rather than that its token is wrong.
A lookup that FAILED is answered as a failure. Reporting a database that could
not be read as "unauthorized" would send a worker re-registering, throwing away
the identity its tunnel and loaded models are keyed by.
Wires the tunnel registry in core/application/distributed.go and hands it to
the membership loop. Without that call the re-claim after a replica is reaped
had no production caller and could never run.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
The gate is justified by being held for one claim round trip, and Attach
held it across the close of the session it superseded. Closing a yamux
session closes the underlying conn and then waits for both its send and
recv loops to exit, and the send loop can be inside a write bounded only
by ConnectionWriteTimeout, so that is a wait on other goroutines. It must
not stand between a worker re-dialling this node and its claim.
The gate is now released after the store and before the close, which also
makes Attach match reclaimOne, where it has always been released
explicitly on every path. This is safe because a superseded session is no
longer reachable from the map by the time it is closed: the next re-dial
replaces an entry that already names the new session.
Pin the re-claim half of the gate too. A worker that re-dials between a
re-claim's commit and its record leaves the row carrying the re-dial's
epoch while the entry carries the re-claim's, so the attachment holding
the socket releases an epoch the row does not have and the row outlives
it, with nothing to sweep it while this replica is alive. Only Attach's
half of the serialisation was asserted; keying the two apart left every
spec green.
Also take the test hook's action under the lock that guards whether it
has fired. It was written from the spec's goroutine and read from
whichever goroutine issued the statement, which is a race in the harness
that pins the serialisation specs.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Two Attach calls for one node both claim, and PostgreSQL serialises the
two upserts, but nothing ordered the two map writes against the two
commits. The entry left installed could be the one whose claim lost the
row, and its Detach then released an epoch the row does not carry, so the
release matched nothing and the row survived the socket.
Nothing swept that row. This replica is alive and heartbeating, so
ReapStale leaves its rows alone, and no reconnect is coming for a worker
that has gone. Owner kept naming this replica as the live owner of a
tunnel it no longer held, and every dialer sent here was answered
ErrNotOwner, which is the relay into a replica that cannot serve the
request that this phase exists to prevent.
Claims for one node now pass through a gate, so claim and record are
indivisible. It is per node rather than one lock over the registry, the
way PeerPool locks per peer: the claim is a database round trip, and a
slow one for a single worker must not hold up Open for every other.
Detach is not gated, because it takes no context and must never park
behind an in-flight database call, and it changes no epoch.
Reclaim takes the same gate, which makes its claim the newest one for
that node, so it records the epoch on whatever attachment is installed
rather than only on the one it listed. Refusing to record onto an
attachment that replaced the listed one would leave that row with nothing
able to release it. The interleave the gate does not cover is Detach, and
a claim whose attachment detached while it was in flight is now released
again rather than left behind.
Also: restore Start's doc comment, which SetTunnels had swallowed; keep
reaping other replicas when this one fails to rebuild its own row, rather
than skipping the sweep along with the re-claim; scope the comment about
an unnoticed dead socket to the keepalive of the session whoever accepted
the tunnel built, since the worker session config does not exist yet; and
pin the sortedness of Held, the nil-session refusal, and both re-claim
interleaves with specs.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Phase 1 left the connection fence with a table and no sockets behind it.
This adds the registry that holds them: Attach claims the node and then
stores the session, Open hands out a stream over the tunnel this replica
holds, Detach releases the claim it was handed, and Held names what this
process is carrying.
The claim is written before the session is stored. A claimant that
installs itself and only then finds it cannot claim has, for that window,
published a tunnel no row records, so Held names it while a peer asking
Owner is told the worker is connected nowhere.
ErrNotOwner is produced at one place, the map miss. It is a routing fact:
some other replica may hold that worker perfectly well. A broken socket
under a held entry is returned as itself, because answering "not held
here" would send a dialer looking elsewhere for a worker this replica is
holding.
Epochs are compared for equality and never ordered. Claim guarantees an
epoch is unique and never reissued; it does not guarantee the later claim
draws the larger number, because the sequence value on the insert path is
drawn before the row lock.
The membership loop now re-claims on re-register, which closes the hole
phase 1 named in ReapStale. A replica that stalls long enough is swept by
a peer, losing its instance row and, in the same transaction, every
connection it owned; Register rebuilds the instance row and nothing else,
so without this it serves workers that every other replica reports as
connected nowhere. Re-claiming draws a fresh epoch, so an attachment
carries two: the token Attach handed back, which is what Detach matches
and which never moves, and the epoch of the row currently held, which is
what Release is given. Collapsing them would leave the re-claimed row
outliving the socket with no caller able to remove it.
A tunnel whose session is already closed is skipped rather than claimed
back, because claiming is an upsert and would take the row from whoever
holds the worker now.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review round 1 on the joined Owner read. The behaviour was accepted; three
comments claimed more than the code delivered, one spec pinned less than its doc
promised, and one pre-existing spec ranked epochs.
instanceIsLive said every reader of instance liveness uses it, which was false:
ReapStale spelled the complement by hand. The complement is now written as
NOT (instanceIsLive), so "stale" is exactly "not live", including how each side
treats a NULL last_seen, and the sentence is true. Inverting that predicate
fails 3 reaper specs, so the routing is held.
The Select("node_connections.*") in Owner was justified by a SELECT * hazard
that cannot occur: with a join present and nothing selected, gorm expands the
model's own columns table-qualified (callbacks.BuildQuerySQL), and the suite is
green with the Select removed. It stays, because the projection should be a
property of this query, and the comment now says that instead.
Owner gained the dialect guard Claim has. now() and make_interval are
PostgreSQL, so on the SQLite single-binary path it failed with "no such
function: now", which reads as a missing migration; that regression already
shipped once in phase 1. The refusal is deliberately not ErrNoConnection: a
deployment with no cluster has no answer about ownership, and reporting absence
would let a caller conclude the worker is not connected. A spec in the
non-PostgreSQL block holds all three properties.
The new specs aged rows by ten minutes, which any window between zero and ten
minutes satisfies, so nothing tied Owner's window to the one the sweeper uses.
They now age to just past InstanceLiveness, and a sibling ages to half of it and
must still resolve. Widening the window tenfold fails 2 specs, narrowing it
tenfold fails 1; before this both were silent.
The concurrent-claim spec asserted the stored epoch was the highest handed out,
and justified it with claims drawing their epoch after the row lock, which
contradicts Claim's own doc: the insert path draws nextval while the tuple is
built. It now asserts the stored epoch is one of the epochs handed out, and
ranks nothing.
OwnerRow's doc justified the function with a sweeper that does not call it.
ReapStale deletes orphans with a set difference; the callers are this package's
specs and one e2e assertion. It says that, and states plainly that a caller
needing to know who owns a node in order to dial it wants Owner.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ci): remove the e2e container before removing its image
`docker stop` returns as soon as the container exits, but the daemon
reaps a `--rm` container asynchronously after that. The `docker rmi
localai-tests` that follows teardown-e2e then loses the race against the
reaper and fails with "conflict: ... is using its referenced image", so
make exits 1 and the job goes red after every spec has passed.
This is why the E2E Backend Tests job fails at random across pull
requests. Runs 33435319093, 33435332991, 33412165884 and 33444669207 all
report "SUCCESS! -- 235 Passed | 0 Failed" and then die in teardown.
`docker rm -f` is synchronous, so the image reference is gone before
teardown-e2e returns. It also covers the case where no container is
running, which `docker stop` could not because it rejects an empty
argument list.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(ci): open a tmate session only when a PR asks for one
The tmate step runs on every failure and then holds the runner until
GitHub cancels the job at the 6 hour limit. A one second cleanup race in
the e2e teardown therefore costs a whole ubuntu-latest slot. The recent
run list is full of 6h, 7h and 12h cancelled runs for that reason.
The step now needs the `ci-debug` label on the pull request, so a
session opens when somebody wants to debug and never otherwise. The
30 minute step timeout caps the cost when the label is left behind.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
OwnerRow is a bare row read of node_connections. A connection row outlives the
replica that wrote it: a replica that dies stops heartbeating, but its rows
survive until a peer's sweep removes them, which is up to InstanceLiveness plus
one InstanceHeartbeat later. For that whole window the table names a process
that is gone. The next component phase 2 builds is the relaying dialer, and a
dialer reading OwnerRow would relay into a corpse for roughly 35 seconds after
every replica death, then report the worker as unreachable when it is in fact
absent, which is the distinction the phase 1 end-to-end specs pinned.
Owner is the resolving read: one statement joining instances, returning
ErrNoConnection when the row is missing OR its owner is not live. Both cases are
one answer on purpose, since both mean no replica here holds this tunnel; they
differ only in which sweep has run. It is one statement, not a row read followed
by an instance lookup, because between two statements the owner can die and the
caller would act on an owner the second read would have rejected.
OwnerRow stays, unjoined, for readers that need the row itself, and a spec holds
the two apart: with an aged-out owner, OwnerRow still names it and Owner
refuses, so neither can quietly become the other.
The liveness predicate is now one string, instanceIsLive, shared by Live and by
Owner's join. Two spellings of one fact drift, and this drift would show as a
relay to a replica one query calls dead and another calls alive. It is
table-qualified so it is unambiguous inside the join, and the cutoff stays on
the database clock, so replica clock skew cannot widen or narrow the window.
Both mutations were run. Dropping the liveness predicate from the join fails 3
specs, the aged-owner one among them. Replacing the database clock with a
Go-side time.Now() fails 1: the aged-owner specs still pass, because the two
clocks agree on one host, and only the recorded-SQL spec sees the literal
timestamp. That is why that spec exists.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
ReapStale deleted from instances then node_connections while Deregister took
them the other way round, both inside one transaction and both running
concurrently by design: a replica shuts down while a peer sweeps it. Opposite
orders let each hold the row the other waits for. PostgreSQL breaks the cycle by
aborting one side, so the cost today is a warning rather than lost data, but the
inversion costs nothing to remove.
Deregister now deletes the instance row first. That is the order ReapStale is
forced into anyway, since its connection delete asks which instance rows
survived, so the sweeper is the fixed side. Both functions say the order is
deliberate and shared, and name the other. A spec records the statements each
path issues and asserts they delete from the same two tables in the same order;
racing two transactions until they really deadlock would be flaky and could pass
for the wrong reason.
The rest is comment and spec accuracy, deferred from the phase 1 task reviews:
- co-location does not imply loopback. Compose's usual host=postgres resolves to
a bridge address and discovery works there; it is a DSN that NAMES localhost
that yields a loopback source address. Corrected in the DiscoverAdvertisedAddr
doc and in the spec comment that repeated it.
- unroutableReason labelled every scoped address "link-local", including the
class the check exists for, and formatted the IP with %s, which drops the
%iface, so the reported address was not the one being rejected. Split into two
cases, both rendered with their zone. CheckAdvertisedAddr passed zone "" and
net.ParseIP rejects fe80::1%eth0, so a scoped literal looked like a name and
collected no warning at all; the zone is now split off before parsing.
- Splice's "Both callers satisfy it" claimed callers that still do not exist.
It now names the two stream types the wake-on-Close property was verified
against and says a phase 2 caller over anything else has to check it.
- restored, short, why a socket-level ECONNRESET stays reported while a yamux
reset does not: the yamux endings are the teardown Splice's own Close
provokes, and whether an aborted request is routine is the relay's policy.
- the real-yamux spec's far.Read had no deadline, so a stall parked the suite
rather than failing it.
- gorilla's SetWriteDeadline is conn.go:796, not 787.
- ClusterPathPrefix is no longer derived from: the peer route spells its path
out, because core/services/cluster must not import core/http/auth. The comment
now points at the spec that holds them together instead of claiming a
derivation the move removed.
- the epoch spec asserted e2 > e1, an ordering Claim's doc tells callers not to
rely on. It asserts uniqueness, which is what the fence guarantees, and is
named for that. A sibling spec still described the epoch as incrementing in
SQL when it is drawn from a sequence.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review found the recurring class: assertions that a wrong implementation
also satisfies.
The "refuse promptly, never park the peer" guarantee was stated in three
places and tested in none. Removing the Close from the no-relay branch left
the whole cluster suite green, because the specs asserted only that some
error arrived and yamux reports a read deadline as ErrTimeout: a parked
stream satisfied that as well as a refused one. Both specs now require an
ENDING, EOF or a reset, inside a deadline short enough that parking is
unmistakable, and both go red when the Close is removed.
Deregistration existed only in a comment. Membership.Stop ended the loop and
left the row behind, so every clean rolling restart had peers dialling a
corpse for the full liveness window; the shutdown comment described the
opposite. Registry.Deregister deletes the row and the connections that
replica owned, in one transaction, for the reason the sweeper does both, and
an e2e spec pins departure inside a budget shorter than the liveness window
so it cannot pass on the sweeper doing the work. Before: the spec times out
with both replicas still live. After: 3.6s.
The configured advertised address bypassed every check discovery makes, so
the one value most likely to be copied between hosts, 127.0.0.1, was taken
verbatim and would make every peer dial itself. Both paths now share one
rejection rule: unparseable is refused, "this host" is warned about once and
honoured, because a single-host deployment uses it correctly.
Two comments claimed more than the code does. The sweeper said a stalled
replica recovers via re-register; only its instance row does, while the
connections another replica reaped stay gone and the sockets stay held here
- phase 2 must re-claim, on re-register, every connection a replica still
holds locally. And Owner became OwnerRow, documenting that the owner it
names may be dead for up to InstanceLiveness plus a heartbeat and that any
caller acting on it must join instances itself, so the deferred constraint
lives at the call site rather than in a report; the plain name is left free
for the joining version.
Minors: warn once when the peer link mounts with no registration token, so
an operator sees the cause rather than 401s; Stop no longer blocks forever
when Start was never called; corrected the NewRegistry migration doc and an
e2e comment that described a 6s window as "throughout".
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Tasks 1 to 5 built an instances table, a splice, both halves of a peer link
and an epoch fence, and nothing in the tree called any of it: no replica
registered, no route was mounted, no sweeper ran. Proving phase 1 end to
end therefore had to start by wiring it.
A frontend in distributed mode now publishes the address its peers dial,
heartbeats it, and sweeps replicas that stopped answering along with the
connection rows they owned, in one pass so the two can never disagree about
who is alive. It serves the peer link and owns the sessions peers dial in,
refusing streams on them until phase 2 installs a relay: a session nobody
accepts on does not fail a peer's Open, it hangs it.
The address is the one peers use, not the one the process binds, and it is
derived from the route to PostgreSQL. That derivation only holds while the
database is remote, so LOCALAI_DISTRIBUTED_ADVERTISE_ADDR sets it
explicitly and a replica that can determine neither warns and keeps
serving rather than failing to start.
Three e2e scenarios run against real local-ai processes, real PostgreSQL
and real dials: replicas publish addresses that can actually be connected
to; a sibling opens a stream over the peer link and is refused without the
cluster token; and a killed replica is reported unreachable, never absent,
loses the claim it held, and takes no worker with it. Each was verified by
mutation: eight injected defects, each failing the scenario that claims to
catch it.
Also moves RegisterClusterRoutes to core/http/routes beside every other
registrar, folds AutoMigrate and the epoch sequence into one
cluster.Migrate, and turns the peer route's auth-coverage spec into a real
assertion: it drives the request through the actual auth middleware
instead of comparing two string constants, which the old spec would have
passed even with the exemption deleted.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The NodeConnection model carried `default:now()`, which is PostgreSQL syntax
reaching the DDL, so AutoMigrate failed on the single-binary SQLite path and
took every SQLite caller of nodes.NewNodeRegistry down with it. Stamp the
database clock as an expression inside Claim instead, the way Register
already does, and leave the column plain.
CREATE SEQUENCE is Postgres-only for the same reason, so it is skipped on
another dialect, and Claim refuses that dialect outright: a fence that cannot
draw a token must say so rather than fail later as a missing function.
Also correct a claim the previous commit made in both the doc comment and its
message. An epoch is unique and never reissued, but it is not ordered: the
insert path draws its sequence value before taking the row lock, so a claim
that inserts after a release can be handed a lower number than one already
issued. Uniqueness is what Release needs, since it matches by equality;
callers must never compare epochs for order.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Release deletes the row, so a per-row `epoch + 1` restarted the numbering at 1
for the next claim. A replica could then be handed an epoch it already held:
claim w1 at epoch 1, lose the link silently, watch another replica claim and
release, reclaim and be handed 1 again, and its delayed cleanup for the first
dead link would match the live claim and delete it. The fence has to be
unique per node over time, not per row lifetime.
Every claim now draws nextval from a dedicated sequence on both the insert and
the conflict paths, so an epoch is never issued twice. The draw still happens
after the row lock on the conflict path, so the winning claim still holds the
highest epoch handed out.
Also drop last_seen. Nothing maintained it and it was always equal to
connected_at, but an indexed column named that way invites a second liveness
clock; whether the owner is alive is Instance.LastSeen, and whether a claim is
current is the epoch.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A worker whose link is silently broken reconnects to another replica while
the old owner's socket has not yet noticed. Without a fence both believe
they own it. Claim is a single atomic upsert returning the new epoch, and a
release must match both owner and epoch so a stale owner cannot delete a
live claim.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The peer link's WebSocket adapter and route constant lived in
core/http/endpoints/cluster, so the dialler in core/services/cluster had to
import an HTTP endpoints package to reach them. That pulled echo, core/http/auth
and core/config into a package whose doc says it is deliberately free of such
dependencies, and it made core/services/nodes reach an endpoints package
transitively. It also has no way forward: the worker-connect handler needs the
tunnel registry and the node token store, both of which are cycles from there.
Move WebsocketConn and PeerPath into core/services/cluster and let the endpoints
package import it, which is the direction the rest of core/http flows. The route
and the auth exemption still cannot drift apart, now asserted where both are
visible rather than by a const reference across the boundary, and the assertion
is stronger than the one it replaces: it pins the route under the prefix instead
of pinning the prefix's spelling.
Also guard the fresh-dial path with ctx.Err(), mirroring the cached path. A
caller with a 300ms deadline dialling a live, listening peer was told the peer
was unreachable, which would be enough for one impatient client to get a healthy
replica routed around once the relay consults these errors.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Distinguishes a peer missing from the registry from a peer that will not
answer: the second must never be readable as node absence, or a network
hiccup between replicas evicts healthy workers.
The distinction is a property of the error type rather than of the call
sites. The unreachable error formats its cause into its message and keeps
it out of its unwrap chain, so an ErrInstanceNotFound picked up on the
dial path cannot reach a caller's absence check.
One yamux session is cached per peer and re-dialled when OpenStream on it
fails, which is how both a dead transport and a graceful remote go-away
arrive. A reset of one stream never reaches the pool, so an abandoned
request cannot cost every other worker its link.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The prefix constant moves to core/http/auth beside the check that uses it, and
the endpoints package derives its route from there. Seven sibling endpoint
packages already import auth, so the previous direction would have deadlocked
the build as soon as this one registered in RouteFeatureRegistry, and it was
dragging echo, gorilla/websocket and yamux into unrelated service packages.
Four properties were argued in comments and held by nothing. Flipping the
empty-token check to fail open, making SetWriteDeadline a no-op, returning a
zero-length read for a zero-length message, and dropping the recover around the
callback all left the suite green. Each now fails a spec that asserts the
behaviour rather than the setter's return value.
SetWriteDeadline takes the write mutex because gorilla keeps that deadline in a
plain struct field applied at the next flush; SetReadDeadline must not take the
read mutex, since it goes straight to the net.Conn and would otherwise block
behind the read it exists to unblock.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A dead yamux session does not always arrive wrapped. Session.close publishes
shutdownErr and closes shutdownCh before it force-closes the streams, so a
Write or Close landing in that window gets the raw cause back instead
(session.go:507-510, 528-533), and for a peer that vanished the raw cause is a
bare io.EOF. The generic io.EOF clause then reported the dead session as a
clean completion.
Remove the clause. A clean read-side EOF never reached it anyway: io.Copy
consumes that and reports nil, and neither *yamux.Stream nor *net.TCPConn
takes a WriteTo/ReadFrom path that would hand one back. Every existing spec
still passes, the io.EOF entry in the normal-termination table included, which
is what showed the branch was dead for legitimate endings and live only for
the bug.
Add a spec driving a real yamux session end to end. Every mux shape until now
was a synthesized error, which is exactly why a race inside the real library
stayed invisible.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Upgrades to a WebSocket, wraps it as a yamux server session and hands it to
the caller. Rejects before upgrading so an unauthenticated dial sees a 401
rather than a WebSocket error, which is what the route-coverage test asserts.
The adapter keeps the reader of a partially consumed message across Read
calls. yamux reads through a 4 KiB bufio.Reader, so a small-payload test
cannot see a dropped message tail; the framing specs drive the adapter
directly with buffers smaller than the message.
An empty configured token authorizes nobody here, unlike the worker file
transfer server's check: this route is registered in every deployment, so
failing open would publish an unauthenticated mux.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Matching yamux errors with errors.Is was too broad. Session.close hands every
live stream ErrStreamReset wrapped around whatever killed the connection, so a
keepalive timeout, a broken TCP connection or a peer that simply vanished all
matched, and a relayed request that died reported a clean ending. Nothing
upstream would have retried or logged it.
Match the plain sentinels by identity, since only identity separates a stream
that was reset from the wrapped form that means the session died. Treat a
StreamError as a per-stream reset, and a GoAwayError as normal only when it
carries the no-error code, read off ErrRemoteGoAway because the constant is
unexported. ErrSessionShutdown needs no entry of its own; it is a GoAwayError
with that code.
Order matters as much as the matching: session death wraps its cause, which is
routinely io.EOF or a closed socket, so the mux checks run before the generic
endings. Reversing them alone puts a vanished peer back to nil.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
go-yamux/v5 matches none of its errors against net.ErrClosed, so the
classifier reported an ordinary teardown as a failure: when the session has
gone away, the FIN that Splice's own Close writes returns ErrSessionShutdown,
and a stream torn down under a live copy surfaces as ErrStreamClosed or a
reset. Splice owns that Close, so it owns the errors it produces; the
sentinels are named here rather than injected by the caller, which would make
a forgotten classifier reintroduce the same bug silently.
Cover the error half of the contract, which no in-memory pipe could reach: a
scripted stream now feeds Splice a genuine transport failure and each
closed-stream ending in turn. Replacing the tail of Splice with "return nil"
passed every previous spec.
Also assert that Splice does not return until the second direction has
finished, rename a spec that promised a leak check it never made, and correct
two comments that claimed more than the code did.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Returns on the first direction to finish and closes both sides so the other
unblocks; a sequential copy deadlocks on any protocol where the far side
speaks first. EOF and use-of-closed are normal termination, not errors.
The fourth spec covers a peer that stops reading mid-body, the case where a
copy is parked in Write rather than in Read. The other three tear down an
idle splice and pass even against a Splice that closes only one side.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
DiscoverAdvertisedAddr promised to return an error rather than a fallback
no peer can dial, but only rejected an unspecified address. With PostgreSQL
on the same host or pod as a replica, which is compose, single-node and any
sidecar layout, the route to it is loopback, so every replica advertised
127.0.0.1 and a peer dialling that reached itself. Loopback, link-local and
zoned source addresses are now rejected with an error naming the remedy, and
a port outside 1-65535 is rejected before it becomes an undialable address.
Liveness was also measured on each replica's own clock: Register and
Heartbeat stamped last_seen from the Go process, and Live compared those
rows against the reading replica's time.Now(). Skew therefore shrank or
stretched the window by writerBehind+readerAhead, evicting healthy peers or
keeping dead ones. Both sides now use the database clock, which is the one
clock every replica demonstrably shares.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
DS4 appends the opening thinking marker to tokenizer-templated prompts, so generated text begins directly with reasoning bytes. Starting DsmlParser in TEXT therefore puts the reasoning and closing marker in visible content.
Start the parser in THINK for structured chat requests with thinking enabled in both Predict and PredictStream. Keep the default TEXT state for raw prompts and reasoning-off requests, and add incremental regression coverage.
Assisted-by: Codex:gpt-5
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
stageDirectory and countStageableFiles already skip them, but
stageOptionDir did not - and it is the path sherpa-onnx voices take for
espeak-ng-data. The receiver writes "<file>.sha256" for every file it
accepts, so staging the sidecars made it write sidecars for those in
turn, one level deeper on every load.
Observed on a live node: "<file>.sha256" repeated eleven times, 5077
junk files out of 7832 in the models dir, and still growing. Staging
never finished, so vits-piper-it_IT-paola-sherpa stayed permanently
"staging on node" and every realtime warmup needing that voice failed
with the session then going silent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0142UfUh8HWxdim5JZqf8Tr6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Replicas need to find each other to relay worker traffic, and nothing in
the tree recorded a replica's address. The advertised address is discovered
by opening a UDP socket toward PostgreSQL and reading back the local
address, which yields the interface every replica demonstrably shares
without asking an operator to configure one.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The closure note in cluster/failure.go quoted a Gomega error that Gomega
does not emit. Describe the argument-count failure and the
Eventually().WithArguments() hint instead, so nobody greps for a string
that never appears.
The advisory-lock note in cluster_failover_test.go called the wedge
window unbounded. A SIGKILLed local child closes its socket at once, the
Postgres backend reads EOF and is reaped in milliseconds, so the
mechanism bounds the window tightly. Say bounded, and keep the low
probability but real framing, which was right.
The workflow comment attributed HealthCheckInterval to
core/services/nodes/health.go. It is declared in
core/config/distributed_config.go:64; health.go only carries the ticker
on the unexported checkInterval. Point a debugger at the right file.
Comments only, no behaviour change.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review of the whole branch found five comments that would send a reader to
the wrong place, plus three smaller inaccuracies. Nothing here changes
behaviour.
The KNOWN RACE note on both backend-log WebSocket handlers said the fix
needs an atomic snapshot-plus-subscribe "under the store lock". It does
not: BackendLogStore.mu guards only the buffers map, and AppendLine
enqueues and fans out under the per-buffer buf.mu. Whoever took the store
lock would ship and the race would survive, so both notes now name buf.mu
and say what s.mu does and does not exclude.
Two comments in the cluster harness quoted Eventually(c.FrontendAlive)
.Should(BeFalse()). FrontendAlive takes an index, so Gomega rejects that
with "requested 1 arguments but received 0". Both now quote the closure
form the specs actually use, and say why the closure is needed.
proveHealthCheckingIsAlive claimed to prove the health monitor ran for the
whole preceding window. It proves the monitor was alive at the end of it,
and inferring backwards needs any wedge to be sticky. In the
peer-replica-death spec that inverts: health checks are single-flighted by
a session-scoped pg_try_advisory_lock, the spec SIGKILLs the replica that
may hold it, and until Postgres reaps the session the survivor acquires
nothing and checks nothing silently. Consistently(healthy) can then pass
because nothing was checking, with the positive control still succeeding
once the lock frees. The doc now states what is proven, names that gap,
and says the assertion is a floor rather than a proof.
The Makefile still called DISTRIBUTED_TEST_FLAKES a retry count, which is
what seeded that error into the two docs just corrected against it, and
the workflow called the 15s window a reconcile tick when the mechanism is
HealthCheckInterval in the node health monitor.
Also: the cluster suite measured 509.1s / 509.8s / 512.3s, so about
8m30s and not the 8m39s/8m40s three files claimed; the dead-worker spec
title implied two independent detectors when both probes read one
advisory-lock-serialised verdict out of the same row; and the
sanitizeDBName length assertion used <= 50, which an empty string also
satisfies, where the invariant for an over-long input is exactly 50.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
--flake-attempts is total attempts, not retries: ginkgo v2.29.0 sets
maxAttempts = FlakeAttempts and loops attempt < maxAttempts, and the flag's
usage string reads "0 - failed tests are not retried". At 1 there is no retry
at all, so "retries a failing spec once" was false in CONTRIBUTING.md and
implied in .agents/building-and-testing.md. Both now say each spec runs once,
and cite the source so the next reader need not re-derive it.
Also restores the React-UI stub rationale, which is load-bearing because a spec
asserting on a UI asset passes locally against a real dist/ and is served the
stub in CI; explains why 213 and ~240 differ; records that the workflow also
triggers on master pushes, where paths-ignore does not apply; and completes the
LOCALAI_E2E_REQUIRE_BINARIES value table, including that any unparseable value
reads as ON.
In .agents/ci-caching.md the stale "13 of those 20" figure now carries its
qualifier inline rather than in the following sentence.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Two Make targets, a flake-budget variable and two environment variables
landed with no way to discover them. CONTRIBUTING.md now tells a contributor
how to run both suites, what each costs and which variables steer the cluster
one.
.agents/building-and-testing.md records the decisions that are easy to undo by
accident: suite-scoped containers, the shared NATS bus and what that means for
a new spec, BeforeSuite over SynchronizedBeforeSuite, the label split,
--fail-on-empty, the binary gate, the flake budget of 1, the coverage
exclusion, and why the cluster suite's long waits must not be shortened.
.agents/ci-caching.md lists tests-e2e-distributed.yml in its paths-ignore
inventory; the workflow already pointed readers there, so the cross-reference
was dangling.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Ginkgo exits 0 when a label filter matches nothing, so a refactor that
renamed or dropped Label("Cluster") would have left the job reporting
"Test Suite Passed" having started no cluster. LOCALAI_E2E_REQUIRE_BINARIES
does not cover that case: it only fires inside a spec that is already
running. Add --fail-on-empty to both distributed targets.
Drop -r from test-e2e-cluster while here. All six Cluster specs live in the
top-level package, and the cluster subpackage contributes nothing under this
filter by design, so recursing only widened the blast radius. test-e2e-
distributed keeps -r: it must reach the eight argument-validation specs in
that subpackage.
Raise the cluster job to 45 minutes, matching its sibling. The 20 minute
Ginkgo timeout bounds the suite alone; the job timeout must also cover setup,
which is the larger and more variable half here: cold-cache module download,
protoc and protogen-go, a full build of ./cmd/local-ai and a separate test
compile, realistically 8-12 minutes on a 4-vCPU runner. At 25 minutes the
runner would have hard-killed the job before Ginkgo could report which spec
hung, which is the red-with-no-evidence outcome that gets suites disabled.
Also move upload-artifact to @v7 with the rest of the repo, and note on the
react-ui stub step that it must go if a spec ever asserts on a UI asset,
since a developer box has a real dist/ and would not catch that locally.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add test-e2e-cluster and a second CI job that runs it. The cluster specs
spawn local-ai as real child processes and kill them, so they need a built
binary; keeping them in their own job means the fast in-process suite is not
held behind that build.
The binary is built with a stubbed core/http/react-ui/dist. A single
index.html satisfies the go:embed in core/http/app.go, and this suite drives
the HTTP API only, so the job skips a Node and Vite install entirely.
The job runs serial and pins --flake-attempts 1. Each Ginkgo process would
otherwise get its own PostgreSQL and NATS container while every spec spawns
two or three children, and a retry would hide exactly the nondeterminism the
suite exists to catch. Measured at 8m39s over three runs, hence a 25 minute
job timeout and a 20 minute Ginkgo timeout.
LOCALAI_E2E_LOG_DIR points inside the workspace so the per-process logs
upload as an artifact on failure; they are the only way to read a cluster
failure. LOCALAI_E2E_REQUIRE_BINARIES is set explicitly even though CI
already implies it, because a skipped cluster spec is indistinguishable from
a passing one and this job's whole value is that it cannot go green without
starting a cluster.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The two specs that assert a healthy worker stays healthy were pure negatives:
they say nothing happened. A cluster whose health checking had wedged, by
leaking the advisory lock the monitor takes at health.go:110, would freeze the
roster and satisfy both while observing a corpse. Kill the worker once the
window closes and require the roster to settle it to offline, so the preceding
Consistently is a statement about behaviour rather than about a stopped clock.
Applied to the cold-restart spec as well as the peer-death one: a restart is
exactly the event that could leave a replacement unable to check anything.
Document the hazard that can make an offline assertion hang. The staleness
branch skips a node already marked unhealthy (health.go:153-155), a skip meant
for nodes an operator took down, which also swallows the flap: an unhealthy mark
landing after the heartbeat goes stale means MarkOffline is never called and the
node stays unhealthy forever. Name the file and line at the assertion, and have
the failure message say so when the roster shows a node stuck there, so a
timeout sends the reader to LocalAI rather than to the harness.
Stop calling the two-replica registration spec a race. Start spawns workers
sequentially and the registrations land about a second apart; it is a
shared-roster identity test, and saying otherwise invites someone to trust it
for something it does not check.
WorkerRegistrar now bound-checks its index like every other index-taking method
here. It answered 0 for an out-of-range worker, and 0 is a real frontend index,
so the failure mode was a spec killing the wrong replica.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Four scenarios with no prior equivalent: killing a replica must not disturb a
worker that never depended on it, a cold-restarted replica must rehydrate the
roster from shared state and keep accepting the worker's heartbeats, a dead
worker must settle to offline on every replica, and two replicas registering a
worker each must converge on one roster.
The timings are measured, not assumed. Node liveness is heartbeat freshness, so
the only eviction path is StaleNodeThreshold (60s) plus one HealthCheckInterval
tick (15s), and neither is reachable from the CLI. A worker whose registrar was
killed was observed going offline at 74.2s. Every window here is sized to
outlast that, because an assertion that expires before the system could have
reacted proves nothing.
Two assertions are deliberately unlike the obvious form. Statuses are compared
for equality against a probe that returns a sentinel on error, rather than
asserting a name is absent from the healthy list: the list probe returns nil on
any error, and "does not contain" is satisfied by nil, so a 401 at the second
replica would have passed while observing nothing. And a killed worker is
required to settle to exactly offline, because it first flaps to unhealthy at
~8s and back to healthy at ~14s, which any not-healthy matcher would accept.
SpreadWorkerRegistrations is new, off by default, and exists so the racing
spec is a race: the harness otherwise points every worker at frontend 0, which
would have left that scenario asserting on two sequential writes through one
process. The default is unchanged because the baseline specs depend on it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add Q4 and Q8 GGUF builds with their shared vision projector. The
model is a recent refusal-removed Ornith derivative for alignment and
red-team research.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The previous round made a missing binary fail instead of skip, but only when a
workflow remembered to set LOCALAI_E2E_REQUIRE_BINARIES. That leaves the silent
pass one forgotten line away: the Cluster label partition is two specs, Ginkgo
exits 0 on skips, and a job that skips both reports "0 Passed | 2 Skipped" and
goes green having never started a cluster.
So the polarity is inverted. Binaries are required whenever CI is set, which
GitHub Actions always does, and the flag now exists to force the requirement
OFF rather than to be remembered ON. A local developer sees no change, since CI
is unset in an ordinary shell and a missing binary still skips with a message
naming the path and how to build it. off, no, n and disabled are honoured as
off; ParseBool rejects them, and reading a word that unambiguous as its
opposite would be a worse trap than the one this removes.
Also correct a claim the previous commit message got wrong. Comparing the
worker's registration id across the two replicas does not pin the topology:
NodeRegistry.Register looks a node up by name and preserves the existing id,
and both replicas read one Postgres, so registering the worker with every
frontend would yield identical ids too. The assertion is still worth keeping
for what it does catch, a replica answering from its own registry or database
instead of the shared one, and the comment now says that and nothing more.
The topology fact moves to where someone would break it: a note on
LOCALAI_REGISTER_TO recording that workers register with frontend 0 only, that
the cross-replica specs depend on it, and that nothing in those specs can
detect a change to it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The Cluster label partition is these two specs and nothing else, so a missing
binary skipped the entire job. Ginkgo exits 0 on skips, so a build step that
broke or moved its output would have left the job reporting "0 Passed |
2 Skipped" and going green without ever starting a cluster: the silent pass
this suite exists to make impossible. Skipping stays the local default, which
is the right courtesy for someone who has not run `make build`, but
LOCALAI_E2E_REQUIRE_BINARIES turns it into a failure that names the missing
path and the target that builds it. A value that is set but unparseable counts
as on, since reading it as off would restore the very skip it disables.
Failures also name themselves now. The roster poll kept returning a bare nil on
error, so a 401 at the second replica, a decode failure and "the worker never
registered" all presented identically as an empty list. It now retains the last
error and the last roster and reports whichever happened, through a lazily
evaluated Gomega description that costs nothing until something fails.
Finally, the two-frontend spec no longer depends on the harness to mean what it
says. It asserts an unauthenticated GET /api/nodes at frontend 1 is refused,
which observes the admin gate instead of assuming it, and it compares the
worker's registration id across the two replicas rather than its name. A future
harness that registered every worker with every frontend would have kept a
name-only assertion green while it quietly stopped proving anything about
shared state.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Tasks 4 to 6 built a harness that runs local-ai as real child processes, but
none of it had ever started a process: every spec so far returned inside
argument validation. These two specs are the first to run it against a real
binary, a real Postgres and a real NATS.
Two frontends against one database both see a worker that registered through
only one of them. Every failover spec assumes this, so it is asserted first.
One admin session is minted at frontend 0 and reused for both replicas rather
than registering per frontend. The auth routes share a five-per-minute-per-IP
limiter and all e2e traffic is 127.0.0.1, so a session per frontend would
exhaust the budget as soon as a spec needs a third one. Reuse is sound because
sessions live in the shared Postgres and the harness pins one HMAC secret
across replicas; frontend 1 answering /api/nodes with 200 on a cookie minted at
frontend 0 is what proves it.
The binaries are resolved before SetupInfra so a missing build skips without
first provisioning a database the skip would then have to tear down.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review round 1. Comments only, plus one guard.
The note on Process.alive claimed the exited check closed the zombie window.
It does not. The reaper closes exited only after Cmd.Wait returns, and Wait
marks the os.Process done before returning, so exited being closed implies
signal 0 already errors and the branch cannot fire earlier than the one it
precedes. The window between the child exiting and waitid collecting it stays
open in both versions, and the only real mitigation is for callers to poll
with Eventually rather than sample once. Keep the check as hygiene, say what
it actually does, and say it again on the exited field, so nobody reads the
old claim and drops the Eventually.
Record what the cold wipe destroys. The harness sets no LOCALAI_STORAGE_URL,
so the object store is a directory under DataPath, and quantization and
fine-tune outputs live there too. Postgres keeps the job row; the artifact it
points at does not survive the restart. A spec that asserts otherwise will
fail for a storage reason wearing a failover costume.
Tell callers to let a graceful stop finish before restarting: RestartFrontend
terminates with SIGKILL, so pairing it straight after StopFrontendGracefully
cuts the drain short and silently converts the rolling-update case into the
crash case.
Refuse to wipe when the cluster has no work dir. frontendDataDir is relative
when baseDir is empty, so a Cluster built by some future test helper without
one would have RemoveAll walking frontend-N/data inside the source tree. The
guard sits before terminate, so a refusal leaves the cluster as it was.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The point of running LocalAI as real child processes is to be able to take
one away. Add KillFrontend (SIGKILL, the lost replica), StopFrontendGracefully
(SIGTERM, the rolling update), KillWorker, RestartFrontend and FrontendAlive.
RestartFrontend pins the dead replica's original port. Workers read
LOCALAI_REGISTER_TO once at boot and never re-resolve it, so a replica that
returns on a fresh port is unreachable by exactly the workers that registered
with it and the failover under test never happens.
It also wipes the replica's data directory, so the process comes back with
empty local state and has to rehydrate node, session and job state from the
shared Postgres and NATS. Reusing the directory would model a pod with a
persistent volume and hide the class of bug these tests exist to find. That
is only safe because the harness pins LOCALAI_AUTH_HMAC_SECRET; otherwise the
wipe would take {DataPath}/.hmac_secret with it and every session minted
before the restart would 401 afterwards.
FrontendAlive consults the reaper's exited channel before signal 0: a child
that has died but has not yet been waited on is a zombie, and signal 0 to a
zombie succeeds, which would report a dead replica as alive.
The new specs cover argument validation only. Killing, stopping and
restarting a live process needs a built binary plus Postgres and NATS, so
those paths stay unexecuted until the failover suites land.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Session rows are keyed by an HMAC of the token under a secret generated
per instance into {DataPath}/.hmac_secret. The replicas shared that
secret only because they shared a working directory, and that directory
was the source tree. Give each frontend LOCALAI_DATA_PATH under its own
baseDir and pin LOCALAI_AUTH_HMAC_SECRET, so a session minted at one
replica resolves at every other one by construction.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The register handler answers 201 both for "user created, here is your
session" and for "this email already exists", so the status code cannot
tell a fresh registration from a repeat one. Key on the session cookie
instead and fall through to login when it is absent.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Restarting a frontend replica must not move it: workers read
LOCALAI_REGISTER_TO once at boot and never re-resolve it, so a replica that
returns on a fresh port is unreachable by the workers that registered with it.
startFrontend now takes the port, with <= 0 meaning "allocate".
Process logs are opened for append rather than truncated, so a restarted
process cannot erase the log of the instance that died, which is the log a
failover post-mortem needs. The post-SIGKILL wait is bounded, so one stuck
child no longer becomes a suite-wide timeout that names nothing. Stop is
nil-safe because Start returns a nil cluster after stopping itself.
Start's doc comment no longer claims to wait for worker registration; that
needs an authenticated admin session, so it now says callers must poll
/api/nodes themselves.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Runs local-ai as real child processes, one per frontend replica and one per
worker, against containerised infrastructure. The in-process suites cannot
express frontend-replica failure: there is no process to kill and no real HTTP
boundary between a worker and the frontend it registered with.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The path allowlist covered 13 of the 99 packages the suite reaches. Commit
1dc3aeef8 touched core/config, core/services/modeladmin and core/backend and
matched no entry, so it would have merged without running the very specs that
cover it. Use the paths-ignore denylist tests-e2e.yml already uses.
Disable the testcontainers reaper: the runner is ephemeral, so the reaper buys
nothing and its unpinned image was pulled mid-suite, defeating the pre-pull.
Drop continue-on-error, which no other workflow uses and which reports a failed
run as green. The job is advisory by staying out of branch protection instead.
Pin Go to 1.26.0 to match go.mod, and add the tmate-on-failure step.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The suite has never run in CI, so 239 specs across 32 files were verified only
by hand. Path-filtered to distributed code, advisory until it earns a track
record, and with flake retries at 1 rather than 5 so nondeterminism surfaces
instead of being retried away.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Three corrections from review of the previous commit.
The lock-order comment on SubscriberCount claimed no path takes s.mu and a
buffer lock together. Subscribe does exactly that, holding s.mu.RLock across
replica registrations that take buf.mu. State the rule that is actually true —
s.mu precedes any buffer lock, so counting after releasing it preserves the
order — and say what follows from it: the total is a sample, not a snapshot.
waitForLogSubscriber read as general-purpose but unblocks on the first
registered subscription. Subscribe attaches the exact-key buffer and each
replica buffer one at a time, so for a replicated model the count goes positive
while later replicas are still unattached and the race survives. Rename it
waitForSingleLogSubscriber, document that it holds only where Subscribe
resolves to one buffer, and assert on exactly 1: misuse then fails loudly on
the count rather than going quietly back to being flaky. Taking the expected
count as a parameter was the alternative, but that makes callers predict a
store-internal number and an under-count fails the same silent way as the
original bug.
The snapshot-then-subscribe race had no artifact outside a report, and review
found a second site carrying it. Mark both handlers identically, including the
point that swapping the two calls duplicates rather than drops and so is not
the fix. The race itself is left alone; this branch stays test infrastructure.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The WebSocket log handler writes its "initial" batch before it calls
Subscribe, so a line appended the instant that batch arrives lands in the
circular buffer with no subscriber to receive it. Three backend-logs specs
append exactly there and then wait out a 5s read deadline; once a gorilla
read hits its deadline the connection is unusable, so the spec cannot retry.
`--focus='Worker WebSocket log streaming' --repeat=25` failed on attempt 17
with nothing else running, which is far too often to wire into CI.
Add BackendLogStore.SubscriberCount, resolving a model ID by the same
exact-key and replica-prefix rules Subscribe uses, and have the specs poll it
until the handler has attached. Nothing in production calls it and no
assertion is weakened; the handler's own snapshot/subscribe window is left as
it is, being a production streaming question rather than a test one.
Verified with 60 repeats of the WebSocket specs and three consecutive
--randomize-all runs of the whole distributed suite, all at
--flake-attempts 1: 239 of 240 specs pass in about 80 seconds.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A failed CREATE DATABASE panics out of the assertion before closeDB runs,
leaking a pgx pool per attempt. With --flake-attempts 5 that exhausts
postgres:16-alpine's 100 connection slots, at which point the cleanup path's
own Expect fails the spec and one hiccup cascades across the suite. Scope the
admin handle so the panic unwinds through defer closeDB, and let cleanup use a
fallible tryAdminDB that reports rather than asserts.
Register DeferCleanup immediately after CREATE so a later failure cannot leave
the database behind, and warn on TestInfra that the container handles are now
suite-wide.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Starting a Postgres and a NATS container per spec cost roughly 48 minutes of
startup across the 213 specs behind SetupInfra, which is why this suite was
never wired into CI. Containers move to BeforeSuite and isolation comes from
CREATE DATABASE, which the dbName argument already described.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Install 404s because the gallery still points at mmproj-...-f16.gguf.
HF only ships ...-F16.gguf now, with a different sha256.
Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
Fixes#11673: on macOS the DMG launcher appeared to launch nothing. After
installing, the app sat in the menu bar with no window, nothing listening
on localhost:8080, and empty log files, because nothing ever started the
server unless the unrelated 'start on system boot' option was enabled.
- Start the LocalAI server automatically when the launcher opens and right
after a fresh install. The new auto_start_server config key defaults to
enabled and gets a settings checkbox; the legacy auto_start key was never
honored nor exposed, so every existing launcher.json carries an
unintentional false and is deliberately left behind.
- Fix the welcome window suppressing itself: its 'don't show this again'
checkbox was initialized with the inverted value, and SetChecked fired
the change callback which persisted ShowWelcome=false on the very first
showing.
- Surface auto-start failures through the systray startup-error dialog,
since there is no visible window during auto-start.
- Pass --app-version to fyne package so the app stops reporting itself as
version 0.0.0 in the About box.
- Document the first-launch flow (menu bar app, auto-start, WebUI URL) in
the macOS getting-started page.
- Repair two launcher specs that never ran in CI: a *bool matched against
BeTrue and a /tmp assertion that trips on Linux where the test tempdir
itself lives under /tmp.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add three llama.cpp-compatible mixed quantizations from ISTA DASLab. These builds give Qwen3.8-27B users an 8.4 to 10.1 GB weight tier with the shared vision projector.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add Q4_K_M and Q8_0 MTP variants with the shared vision projector.
The publisher recommends these builds for faster Qwen3.8 generation.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add the 3B, 8B, and 30B safetensors checkpoints as one vLLM variant family so LocalAI can select the largest build that fits. Configure the parsers and sampling defaults recommended for Granite reasoning and tool calls.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The Makefile already had a hipblas branch, but no CI row built it and
the gallery's `amd:` mapping stayed commented out. On an AMD host the
capability lookup found no `amd` key and fell back to `default`, so
these users silently ran the CPU build.
Add the hipblas row to the backend matrix and the two gallery entries
it publishes, then point `amd:` at them.
Drop `-DGGML_HIPBLAS=ON` while here. `SD_HIPBLAS` sets `GGML_HIP`
itself, and `GGML_HIPBLAS` is the name ggml used before the rename, so
the flag only produced an unused-variable warning. Add gfx1151 to the
local target list to match the value the workflows pass in.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Tencent released three WeMM sizes with direct Sentence Transformers support. Add each safetensor repository so users can select the quality and resource tradeoff.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
OpenAI GA clients send multipart or raw SDP requests. They expect a bare
SDP answer. LocalAI only accepted its legacy JSON envelope, so signaling
failed before media setup.
Keep the JSON contract for existing clients. Accept both GA request
shapes and choose the matching response format.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add the Q4 and Q8 GGUF builds with the shared vision projector.\nThe variant pair lets LocalAI select the build that fits available memory.\n\nAssisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add complete vLLM and SGLang entries with their exact tool parsers. Preserve an explicit zero temperature in both backend adapters.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Node placement and replica rules could only name a model, so an operator
who pinned "llama3" to the GPU tier had to rewrite the rule whenever a
different model took over that job. An alias already gives a stable name
for whichever model serves it, and a rule on that name makes it a
deployment slot: repoint the alias and the placement follows.
A rule keeps the name the operator chose. Reads resolve that name through
the config loader to the model the rule governs, so the reconciler counts,
schedules and trims replicas of the target, and the router finds an
alias-keyed rule from the target it is already routing. An alias that
resolves to nothing governs nothing loadable, so the reconciler skips it
and the write paths refuse it.
A replica is shared by every name that resolves to it, so only one rule
can decide where it runs. The REST and MCP write paths reject a rule whose
target another rule already governs. A pair that arrives some other way,
such as a seed file or an alias repointed onto a model that already has a
rule, resolves in favour of the rule named after the model itself and then
the oldest, and the rest are listed as shadowed.
The eviction guard is the exception: it matches rules to replicas in raw
SQL inside a locking transaction and cannot resolve an alias. It reads a
stored target that the reconciler refreshes each tick, and falls back to
the rule's own name when that target is empty.
Assisted-by: Claude:claude-opus-5 golangci-lint eslint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ui): move node labels into the scheduling selector field
The scheduling page kept a node-label browser open above the rules
whether or not anyone was writing one, while the field that actually
needs labels, the rule's node selector, was two bare text inputs with no
hint of what the cluster reports.
The browser is gone. The selector's key input now completes against the
label keys the cluster uses, and the value input offers only the values
that key takes. The roster already loads for the page, so the
suggestions cost no request, and a roster that fails to load costs the
admin the hints and nothing else.
Suggestions stay suggestions: a key no node reports yet still commits as
typed, which is how an admin writes a rule before labelling the nodes
for it.
Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): size model fit against the cluster, not the frontend
The models page asked the frontend how much memory a model may occupy.
In distributed mode the frontend is usually a GPU-less pod while every
model runs on a worker, so a fleet of GPU nodes was told it could only
run the smallest CPU build. The variant picker's fits flag and its
auto-selection came from the same place, as did the hardware
recommendations.
The registry now reports the largest single healthy backend node. The
largest node, not the fleet total: a model loads into one node, so four
16GB workers are not a home for a 40GB model. An operator-set VRAM
budget caps a node's contribution, because the scheduler refuses a load
above that ceiling anyway, and a GPU node beats a CPU node holding more
system RAM.
GET /api/resources and GET /api/models carry this as an additional
cluster object. Their aggregate and ram fields keep reporting the
frontend's own hardware, which is what the resource monitor shows.
Variant selection judges backends against the union of the capabilities
present in the cluster, the way backend discovery already did.
Every path degrades to the local host: no cluster object in single-node
mode, and none when the registry cannot be read, so a hiccup narrows the
answer back to single-node behaviour rather than marking the whole
catalog too large.
The verdicts now name the node they belong to, since a model fits
somewhere or nowhere.
Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): follow upstream MTMD APIs
The dependency update adds MTMD initialization options to prompt and
bitmap helpers. The gRPC adapter now passes the server options through
each affected path.
The update also replaces the per-layer MoE regex helper. Preparation
probes both APIs because older forks still reuse this adapter.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
go-m1cpu v0.1.6 runs its cgo initialiser from a package init(), where
getFrequency() dereferences the CFTypeRef returned by
IORegistryEntryCreateCFProperty without a NULL check. On Apple M5 the
pmgr IORegistry node does not expose voltage-states5-sram /
voltage-states1-sram in the shape v0.1.6 expects, so the call returns
NULL and CFDataGetLength(NULL) faults before main() runs. Every command
dies, including local-ai --version.
The package is linked indirectly: cmd/local-ai reaches
gopsutil/v3/{process,disk}, which pull in gopsutil/v3/cpu on darwin,
which calls m1cpu.IsAppleSilicon() and m1cpu.PCoreHz().
v0.2.2 adds the missing NULL guard and moves the IORegistry probe out of
init() behind a lazy sync.Once. The exported Go API is unchanged and the
non-darwin stub is byte-identical, so gopsutil/v3 compiles against it
untouched and no other platform is affected.
Bumping gopsutil/v3 is not an alternative: v3.24.5 is the final v3
release, so the v3 line will never carry this fix.
Fixes#11735
Assisted-by: Claude:claude-opus-5
Signed-off-by: Szymon Podeszwa <2962046+sz-po@users.noreply.github.com>
* ⬆️ Update mudler/vllm.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(vllm-cpp): mirror ABI v23
The new engine pin reports ABI v23 and appends mmproj_path to
vllm_model_params. LocalAI still declares v21, so the build-time ABI
guard rejects every backend build.
Grow the Go mirror by the appended pointer and update its offset checks.
ABI v23 adds a video function but does not change the mirrored text
structs.
Assisted-by: Codex:gpt-5.6 [systematic-debugging]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
/api/tags appends :latest to untagged names, but chat and the other
model endpoints looked the tagged name up as-is and 404'd.
Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
* feat(gallery): add Ornith 1.5 397B variants
Add the official Q4_K_M and Q8_0 GGUF builds with their shared BF16 vision projector.
Assisted-by: Codex:gpt-5
* feat(gallery): resolve Ornith variant ordering\n\nKeep the 35B entries from master next to the 397B variants.\n\nAssisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): adapt to the common JSON API
The llama.cpp bump replaces its nlohmann JSON alias with common_json. Update the gRPC adapter for the new exception, iterator, conversion, and container APIs.
Assisted-by: Codex:gpt-5.6 [systematic-debugging]
* fix(turboquant): adapt the JSON exception type
The shared gRPC source now follows the upstream common_json API. The
TurboQuant fork still exposes nlohmann JSON and cannot compile the new
exception type.
Translate that exception in the fork-specific source patch so both
llama.cpp variants compile from the shared adapter.
Assisted-by: Codex:gpt-5.6 [systematic-debugging]
* fix(bonsai): adapt the JSON exception type
The shared gRPC source uses upstream's common_json wrapper. The Bonsai fork still exposes nlohmann JSON and cannot compile that exception type.\n\nTranslate the exception in the fork-specific preparation step and verify that repeated preparation stays idempotent.\n\nAssisted-by: Codex:gpt-5.6 [systematic-debugging]
* fix(llama-cpp): let prepare register gRPC
The score patch duplicated the gRPC CMake registration that prepare.sh already owns. Its stale context rejects the current upstream tools file on Darwin before compilation starts.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Those backends only forwarded the flag when it was "true", so "false"
never reached apply_chat_template and Qwen3 kept thinking on.
Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
The scheduler's liveness probe asks a worker a question over NATS and
reads "no responders" as proof the worker is gone. That is only sound
when every worker in the fleet subscribes to the subject asked.
It asked models.running, which arrived in 4.6. A 4.5 worker is alive and
serving, answers backend.list, and never subscribes to models.running,
so the probe condemned it on every scheduling attempt and marked it
unhealthy. A model pinned to such a node by its selector could then
never be placed at all: on this cluster an embedding model pinned to the
one Apple node was unschedulable for exactly this reason, while that
node's log showed it handling backend.list throughout.
Ask backend.list, which has been in the worker protocol far longer, and
treat a worker that answers anything as alive. Only a node that reports
no responders on every subject is absent, so adding a newer subject here
can never condemn an older worker.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
A model's revision is published by administration and checked against on
every inference request. Those were computed by separate code: the
request path resolves through the loader, while each publisher hashed
whatever ModelConfig it happened to hold. By then SetDefaults had folded
in the GGUF guess and app-level options, so the published value was one
no request would ever carry and the model became unroutable until the
row was deleted by hand.
Fixing the publishers one at a time did not hold. Three rounds each
found another: the startup resync, then a saved edit and a toggle, then
a rename and the peer-change path.
ModelConfigLoader.RevisionFor is now the only way to obtain a revision,
and the raw hash is unexported, so a caller outside this package cannot
hash a config it holds. A publisher and a request agree by construction
rather than by two implementations happening to match.
The request path no longer falls back to hashing its merged config
either: an unstamped config is routed without a revision rather than
with a wrong one.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
When no node the selector allows has a free slot, scheduling falls back
to evicting the least-recently-used idle model. That eviction searched
every healthy node, so it freed a slot on a node the selector forbids
and the model was then placed there: pinned to one class of hardware and
running on another.
An unrelated model pays for it. On this cluster an embedding model
pinned to Apple hardware could not reach its only matching node, so each
attempt evicted a large language model from an Nvidia node, failed to
start there anyway, and left the evicted model to reload. Repeated, that
reads as one replica bouncing between nodes.
Eviction is now restricted to the candidate set the selector produced.
With no selector the candidate set is nil and eviction stays global.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
The abandoned-load sweeper treated a replica row with no load job as
abandoned. Only the request path creates load jobs; the reconciler's own
scale-up loads a replica without one. So any scale-up that ran past the
five-minute grace period was deleted mid-transfer, which for a
multi-gigabyte checkpoint is every time. The replica never finished
anywhere, and the reconciler kept re-placing it, so it looked like one
replica hopping between nodes instead of a model reaching its replica
count.
A row with no job is now reclaimed only once its node stops being
healthy, which is the case the sweeper was written for: a worker that
dropped out mid-transfer. A job that failed or stopped heartbeating
still proves abandonment on its own. Every uncertain case leaves the
slot held.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
Only backend workers subscribe to backend.list. ListBackends asked every
node that was not pending, offline or draining, so an agent worker could
only answer "no responders", which the error handling reads as a node
that has gone away. Every poll of the backends view therefore marked
each agent node unhealthy, and its next heartbeat marked it healthy
again.
While unhealthy the node is not schedulable, so this also cost agent
capacity for as long as each flap lasted.
Skip non-backend workers, as the backend-op fan-out already does for the
same reason. A backend worker that does not answer is still marked
unhealthy: that one really is gone.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
The revision was computed after SetDefaults, which folds in things that
are not persisted configuration: the GGUF guess, the hardware defaults,
and app-level options such as threads.
The GGUF guess is the damaging one. It parses the model file to fill in
values like context size, and when that parse fails it falls back to a
different default. Whether a multi-gigabyte file on network storage
parses at a given moment is not a property of the configuration, so one
unchanged YAML produced two different revisions depending on when it was
read. The controller rejected every request carrying the other one, and
the model stayed unroutable until the stored value happened to match
again. This is why it never reproduced against a model directory with no
weights in it: the guess is skipped there and both values agree.
The app-level defaults are the same class of bug with a slower fuse:
changing threads in the settings UI changed every model's revision and
made every model unroutable.
The revision is now stamped when the file is parsed, before any defaults
are applied, so it is a function of the file alone.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
Two code paths computed a model's revision. Inference resolves the
config through the loader, which applies SetDefaults a second time.
Everything that publishes a revision hashed the stored config instead,
with SetDefaults applied once.
SetDefaults is not idempotent for every model: it re-runs the GGUF guess
and the hardware defaults, both of which read state the stored config
does not carry. Where the two disagree, a publisher wrote a revision no
request would ever carry, and the model became unroutable the moment it
was published. On this cluster the startup resync republished one such
value and every request for that model was then rejected against it.
The publishers now resolve the revision through the loader, exactly as a
request does, so there is one definition rather than two that agree only
when SetDefaults happens to be idempotent. This covers the startup
resync, a saved config edit, and enabling or disabling a model.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
quay.io redirects blob downloads to pre-signed S3/Akamai URLs that
expire after about 10 minutes. On a slow connection a multi-GiB
backend layer cannot finish inside that window, so the connection
drops mid-stream on every attempt. The retry added for #10577
restarted each attempt from byte zero, which replayed the same
failure until the budget ran out and the install failed with
"unexpected EOF".
A retry now keeps the bytes already on disk and re-requests the
blob with "Range: bytes=N-". Each request goes back to the
registry, so it gets a fresh redirect URL and auth token. The
retry budget only counts attempts that made no forward progress,
so a slow link that keeps advancing keeps downloading. A resumed
file is spliced from separate responses and bypasses the digest
check in layer.Compressed(), so the assembled file is re-verified
against the layer digest before it is trusted; on a mismatch the
download starts over through the verified reader.
Fixes#10577
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): add Qwen3.8 OBLITERATED variants
Add Q4_K_M and Q8_0 llama.cpp builds with the shared BF16 vision projector.
Assisted-by: Codex:gpt-5
* fix(tests): implement node liveness stub
NodeCommandSender now requires PingNode. The endpoint test stub must
implement it before the package can compile.
Assisted-by: Codex:gpt-5 [Codex]
* fix(distributed): restore node liveness tests
The router now probes models.running before it schedules work. The E2E
workers only mocked backend.install, so every test node appeared offline.
The endpoint test double also missed the new PingNode method and stopped
the Linux, Apple, and lint jobs during compilation.
Mock the existing worker reply in both distributed fixtures and keep the
endpoint test double aligned with NodeCommandSender.
Assisted-by: Codex:gpt-5 [golangci-lint]
* fix(tests): check node liveness replies
The liveness test subscriptions ignored setup and reply errors.
Errcheck rejected each branch that carried them.
Assisted-by: Codex:gpt-5 [golangci-lint]
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(distributed): restore node liveness tests
The router now probes models.running before it schedules work. The E2E
workers only mocked backend.install, so every test node appeared offline.
The endpoint test double also missed the new PingNode method and stopped
the Linux, Apple, and lint jobs during compilation.
Mock the existing worker reply in both distributed fixtures and keep the
endpoint test double aligned with NodeCommandSender.
Assisted-by: Codex:gpt-5 [golangci-lint]
* fix(tests): check node liveness replies
The liveness test subscriptions ignored setup and reply errors.
Errcheck rejected each branch that carried them.
Assisted-by: Codex:gpt-5 [golangci-lint]
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(ci): bound Discord release summaries
The release model can return more than Discord's 2,000-character
message limit. Discord then rejects the entire release notification.
Ask the model for a smaller response and truncate extracted content to
1,800 characters before the notification step. The smaller bound leaves
room below Discord's hard limit when model output varies.
Assisted-by: Codex:gpt-5
* fix(tests): implement node liveness stub
NodeCommandSender now requires PingNode. The endpoint test stub must
implement it before the package can compile.
Assisted-by: Codex:gpt-5 [Codex]
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The resync added in 3953448f6 ran before LoadModelConfigsFromPath, so it
read an empty loader, reconciled nothing and reported success. The
symptom was a stored revision that stayed stale across restarts while
the log showed no complaint, which is exactly what the resync was meant
to prevent.
Move the call after the configs are loaded, and refuse to treat an empty
loader as a clean run: reconciling zero models is indistinguishable from
reconciling correctly, and that is what hid the mis-ordered call.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
The controller pins a model's replicas to a stored revision and rejects
any request carrying a different one. Nothing ever re-derived that value
from the configuration on disk: it moved only on an edit, a gallery
install, or a peer's change broadcast. An inference request may only
establish a revision, never replace one.
So any other way for the two to diverge left the model permanently
unroutable. A configuration edited while a frontend was down lands
there, and so does a change in what the revision is computed over: an
upgrade that alters the hashed form leaves every stored revision
describing a configuration that no longer exists. The only recovery was
deleting the row by hand, which is not something a cluster should need.
Each frontend now reconciles the stored revisions against the loaded
configurations at startup and republishes the ones that disagree. Only
those: republishing quarantines every replica loaded under the old
revision, so doing it for a model that did not drift would unload a
healthy replica for nothing. A model with no stored revision has never
been served and is left for its first request to establish.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
A replica row in staging or loading holds its slot, because slot
allocation counts every state except unloading. Nothing ever reclaimed
such a row: every reconciler pass and the router's eviction query filter
state = "loaded", and the per-model probe skips rows without an address,
which is exactly what a row that never finished loading has.
So a worker that dropped out mid-transfer left a row that pinned the
only replica slot for that model on that node. Scheduling then found no
free slot and eviction found nothing it was allowed to evict, and the
request failed with "no replica slot on <node> and eviction failed: all
models busy". The state persisted until an operator intervened.
The reconciler now reclaims a row stuck before serving when no load job
is driving it. Ownership is decided by the job's LastProgress heartbeat,
not by elapsed time: staging a large checkpoint legitimately runs for a
long while without touching the replica row, so a deadline would either
be a model-size cliff or reclaim a healthy transfer. That heartbeat is
the same signal job takeover already trusts. Any error reading the job
leaves the slot held, because holding one for another pass costs a
scheduling opportunity while a wrong reclaim restarts a multi-gigabyte
transfer.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
A node's status comes from its HTTP heartbeat. Backend installs travel
over NATS. The two are independent, so a worker that dies stops
answering on the bus at once but stays healthy in the database until its
heartbeat ages out. Inside that window the scheduler picked a node it
could not reach, and the request failed with "no responders available"
rather than moving to a node that was up.
The scheduler now probes the node it selected and, when nothing answers,
marks it unhealthy and selects again. The demotion is what makes the
retry terminate: the next selection reads only healthy nodes. It also
tells the other frontends what this one learned, so the cluster does not
rediscover a dead worker one failed request at a time.
Only nats.ErrNoResponders counts as absent. A worker that answers slowly
stays eligible, because dropping it would cost capacity that is really
there. The probe reuses the models.running subject: a new subject would
go unanswered by workers that have not been upgraded, and every one of
them would then look dead.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
Picks up "surface option errors instead of discarding them". New records
the error from applying its options and Run returns it, so a process
whose state directory cannot be created reports the real cause instead
of failing later inside os.MkdirAll("") with an empty path.
LocalAI already resolves that directory itself, so this covers the other
process.New call sites rather than changing behaviour on the backend
start path.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5
process.New applies its options and discards the error they return. When
WithTemporaryStateDir could not create a directory, StateDir stayed
empty and every later option went unapplied, so the failure surfaced
from Run as "mkdir : no such file or directory" naming no path.
That message cost a full day of diagnosis on a worker whose volume was
full: the real errno was ENOSPC and nothing reported it.
The loader now creates the directory itself and returns the underlying
error with the path attached.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
A request that carries a file stages it to the worker, which writes it
under its staging directory. Nothing removed it afterwards. The frontend
expires ephemeral keys from object storage, but that sweep never covered
a worker's local disk, so every image, audio clip and video a worker
ever served stayed on it.
One worker had accumulated 175 request directories over three months.
The volume reached 100 percent, and from that point every backend start
failed because the process manager could not create a state directory.
The worker now sweeps its ephemeral staging directory on a timer and
once at startup, so files left by a crash are reclaimed too. Staged
model files live beside that directory and are not touched.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
"stale model config revision" reported only that two hashes differed.
It named neither, so an operator could not tell an edited configuration
from a revision that is not reproducible for one unchanged file, and the
failing value appears in no table.
The error now carries the revision the request brought and the one the
controller holds. It still wraps ErrStaleModelConfigRevision, so callers
that classify the error keep working.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
Add the 2B and 4B distilled models alongside the 9B model. Their compact GGUF builds make the Qwen3.8 reasoning distill practical on smaller hosts.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
LiquidAI now publishes official DSpark draft sidecars for its 2.6B target. Pair Q4 and Q8 targets with matching draft choices so LocalAI can use speculative decoding across different memory budgets.
Assisted-by: Codex:gpt-5 [systematic-debugging]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
syncKnownUsecasesFromString rebuilds KnownUsecaseStrings by ranging
GetAllModelConfigUsecases, which is a map. Go randomizes that order per
call, and the field is part of the serialized config, so one unchanged
YAML hashed to a different config revision on every load.
A model that derives a single usecase hid the problem. One that derives
several, such as a chat model with an mmproj, alternated between as many
revisions as there are orderings. The router treats a revision it did
not establish as a config change, so requests failed with "stale model
config revision" until the stored value happened to match again.
Sorting the list makes the revision a function of the file alone.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
The request middleware merges the caller's prediction parameters into
its copy of the model config. core/backend.ModelOptions then hashed
that copy, so the revision identified the request body rather than the
persisted configuration.
EstablishModelConfigRevision stores the first revision it sees and
requires an exact match afterwards. The first request after a restart
therefore pinned the model to its own temperature, top_p and stop
values, and every later request that sent different ones failed with
"stale model config revision". No config edit was involved.
The loader now stamps the revision when it materializes a config,
before any request override reaches it, and ModelOptions reads that
stamp. Model administration keeps hashing the same persisted config, so
both paths agree on one revision per configuration.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
* fix(vram): persist remote probe metadata
The startup warmer repeated remote size and GGUF metadata probes after every restart because both caches lived only in memory. Store successful HTTP probes for 24 hours so frequent restarts reuse the prior results.
Bound the cache, reject invalid records, and purge it when gallery data changes. Local model files continue to bypass persistence.
Assisted-by: Codex:gpt-5
* fix(vram): check temporary file cleanup
The lint gate rejects the unchecked cleanup call in the persistent cache writer.
Assisted-by: Codex:gpt-5.6 [golangci-lint]
* fix(vram): make persistent cache optional
Remote metadata probes can transfer enough data that operators need
control over disk reuse and startup warming. Gallery autoload now gates
both behaviors, and the runtime setting applies changes immediately.
Assisted-by: Codex:gpt-5
* fix(ui): expose gallery startup pre-warm
The existing gallery autoload setting also gates the startup metadata warmer. Name both effects in Settings so operators can find the requested boot control.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* docs: design configurable copy buffering
Document the context-aware copy buffer option and its validation plan.
Assisted-by: Codex:gpt-5
* docs: design durable distributed staging operations
Assisted-by: Codex:gpt-5
* docs: design distributed model config revisions
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
* feat(config): add stable model revisions
Hash typed model configuration and effective protobuf options deterministically for distributed revision comparisons.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
* feat(worker): acknowledge exact model stops
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
* feat(nodes): track model config revisions
Assisted-by: Codex:GPT-5 [apply_patch]
* fix(distributed): retry quarantined model cleanup
Stop quarantined replicas by exact process identity, retain failed cleanup as durable capped retries, and compare-and-delete only the claimed registry row. Process one sufficiently leased row at a time so multiple frontends cannot duplicate slow cleanup work.
Assisted-by: Codex:gpt-5
* fix(distributed): bind loads to config revisions
Assisted-by: Codex: GPT-5 [OpenAI Codex]
* fix(modeladmin): apply config revisions consistently
Route model edits, patches, state changes, deletion, and peer refreshes through the same revision lifecycle. Quarantine stale replicas before exact cleanup and report durable pending cleanup without failing successful config writes.
Assisted-by: Codex: GPT-5 [OpenAI Codex]
* feat(distributed): expose model config revision state
Document replica revision observability and durable cleanup behavior. Keep pending cleanup explicit in model mutation responses and verify endpoint contracts expose revision state without serialized load options.
Assisted-by: Codex:GPT-5 [OpenAI Codex]
* test(distributed): cover model revision convergence
Exercise cross-frontend quarantine, stale replay rejection, exact cleanup retry, worker re-registration, and current-generation replica convergence against the distributed PostgreSQL harness.
Assisted-by: Codex:gpt-5
* fix(distributed): pass config revision CI checks
Keep configured gallery sources out of authoritative runtime snapshots only after validating their real schema, and harden rollback snapshots against symlink races and non-regular files.
Assisted-by: Codex: GPT-5 [OpenAI Codex]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The Qwopus variant pair was added twice, which makes the later pair unreachable and fails the gallery invariant on master.
Keep the first pair that LocalAI already resolves and remove the dead duplicate declarations.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Keep the homepage focused on runtime capabilities and move engine details to their canonical directory. Make installation choices stable and explicit for users across supported hardware.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-bot <306113404+localai-org-bot@users.noreply.github.com>
* docs: design configurable copy buffering
Document the context-aware copy buffer option and its validation plan.
Assisted-by: Codex:gpt-5
* docs: design durable distributed staging operations
Assisted-by: Codex:gpt-5
* fix(distributed): merge durable staging operations
Use active model load jobs as the durable operations baseline and overlay replica-local staging progress without duplication. Preserve tracker-only operations when the registry cannot be read.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The gallery ranker reads serving features from tags instead of backend options. Mark two entries that enable draft MTP so ranking can identify them correctly.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix: GHSA-qwww-vcr4-c8h2 security vulnerability
Automated dependency upgrade by OrbisAI Security
Signed-off-by: anupamme <mediratta@gmail.com>
* fix: upgrade react-router-dom to 7.18.2 to fully remediate GHSA-qwww-vcr4-c8h2
The prior fix pinned react-router@7.18.2 directly but left react-router-dom
at ^7.18.1, which bun resolved to 7.18.1. That package bundles its own
react-router@7.18.1 sub-dep, leaving the vulnerable version in bun.lock
via the react-router-dom/react-router scoped resolution. Pinning
react-router-dom to 7.18.2 and regenerating the lockfile removes all
7.18.1 resolutions.
Assisted-by: Claude Code:claude-sonnet-4-6
Signed-off-by: Anupam Mediratta <mediratta@gmail.com>
---------
Signed-off-by: anupamme <mediratta@gmail.com>
Signed-off-by: Anupam Mediratta <mediratta@gmail.com>
* docs(blog): add the 4.9 release post and its demo clips
The 4.9 cycle changed how you authenticate, how chat handles a history
that no longer fits, and where models and backends live in the UI. The
release notes list every pull request; this post covers the three
changes that alter day-to-day use, and leads with the auth one because
it needs an action before upgrading.
Two clips are recorded from a real session against a local-ai built
from master with the live gallery loaded: model-lifecycle.mp4 walks
the unified models and backends pages, import-model.mp4 shows the
rebuilt import form. Both follow the clip conventions in
.agents/preparing-a-release.md: h264, no audio track, 1000x562, under
30 seconds, and named after the feature so they stay reusable.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Playwright]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(blog): anti-slop pass over the 4.9 post
Ran the post through the humanizer and no-ai-slop rules, calibrated
against what-landed-in-localai-4-8.md. That post is the one #11324
left unchanged, so it sets the rate for this series.
The draft ran denser than it on two constructions: "rather than" at
5.1 per thousand words against 3.5, and "instead of" at 3.1 against
1.6. Both are now at or below the 4.8 rate, 2.7 and 1.5, by rewriting
seven of them as plain statements.
Also cut: "keeping both cost a mode switch", a ledger metaphor for
something that is not money, which is the tell #11324 removed eight
times from the APEX post. "A follow-up fixed the thing that made that
awkward", an unearned framing plus a colon reveal. "This release adds
a different one: compress them", a second colon reveal. And
"byte-structurally identical", a second exactness idiom in a post that
already uses "byte-identical" where the precision carries weight.
Five paragraphs opened with "Two things" or "Two details", so three of
them start differently now. The summary listed three items, which is
the rule of three; it lists four, like the 4.8 summary.
Every figure, PR number, link and media reference is unchanged, checked
by diffing them out of both revisions. Hugo builds clean and the
rendered HTML has no em dashes.
Assisted-by: Claude Code:claude-opus-5 [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Expose the upstream Q8_0 build and the MTP drafter for the existing Gemma 4 12B agentic entry. The variants let LocalAI select higher fidelity or faster speculative decoding from the same model family.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update NVIDIA/NeMo-Speech.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(nemo-speech-cpp): build required sentencepiece
The updated runtime uses SentencePiece for RNNT context biasing even when Flashlight and text normalization are disabled. Build upstream's pinned static archive before configuring the engine so every backend variant has the newly required library and headers.
Assisted-by: Codex:gpt-5
* fix(nemo-speech-cpp): support BSD install
The pinned SentencePiece script uses GNU install -D for license files. macOS provides BSD install, which rejects that flag after the library compiles.
Create the license directory and use the portable install form on Darwin.
Assisted-by: Codex:gpt-5
* fix(nemo-speech-cpp): support CMake 4
Homebrew CMake 4 rejects the old policy floor in the pinned SentencePiece helper. Select the oldest supported policy floor for the Darwin dependency build.
Assisted-by: Codex:gpt-5.6 [Codex CLI]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add the official Q4_K_M and Q8_0 GGUF builds with the shared BF16 vision projector.
Assisted-by: Codex:gpt-5 [Hugging Face API]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add Tencent’s compact translation model in Q4 and Q8 GGUF formats. The linked variants let LocalAI select the best build that fits each host.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* test: make coverage failures observable
Keep per-root logs, reject concurrent coverage runs, and avoid relying on /bin/sleep in the worker timeout test.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: parallelize coverage without remote fixtures
Assisted-by: Codex:gpt-5 [apply_patch] [exec_command]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: add offline resource infrastructure
Introduce versioned resource manifests, a checksum-verified CAS preparer, offline test wrappers, and a guarded network transport. Replace live Hugging Face, GitHub, and OCI cases with deterministic fixtures and inject fixture metadata into importer discovery.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: enforce offline resource replay
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: harden offline resource refresh
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: expose slow coverage waits
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: eliminate avoidable wall-clock waits
Inject a clock into Hugging Face retry handling, reuse a process-scoped PostgreSQL container with per-spec schemas in the nodes suite, and poll local import jobs promptly.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: remove repeated fixture startup waits
Share PostgreSQL fixtures across parallel endpoint and agent suite workers, and make the worker Free deadline injectable so the wedged-backend test does not spend five seconds on wall-clock time.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: fix offline resource CI portability
Normalize Docker archive metadata before content addressing, derive archive checksums during explicit refreshes, make network lint portable to macOS, and prepare distributed images before running their offline suite.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* ci: cache Go modules before offline tests
Warm the complete module graph before the Linux and macOS test jobs enter offline replay mode, so tool dependencies such as Ginkgo are not fetched through the guarded proxy.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: drop the static network lint in favour of real isolation
The offline test suite already prevents tests from reaching the network
twice over: run-test-linux-offline.sh puts the test process in a cgroup
and REJECTs egress outside the private ranges, and HardenedTransport
installs testnetwork.LocalGuard to refuse dials that resolve to a public
address. Both fail the test with a precise error at the moment of the
dial.
test-network-lint.sh added neither. Its diff stage defaulted to a HEAD
base, so on a clean checkout it compared the tree against itself and
inspected nothing; the branch's own commits were never examined. It only
produced output when an earlier job step dirtied the tree, and then it
matched a bare https?:// against whatever changed. make react-ui runs
npm install rather than npm ci, so CI rewrote
core/http/react-ui/package-lock.json and the lint reported an npm
registry URL as forbidden test network access:
+ "resolved": "https://registry.npmjs.org/hono/-/hono-4.12.25.tgz",
Its fingerprint stage was self-defeating in a quieter way: hashing the
whole tree's network-mechanism inventory meant every rebase onto a master
that touched any _test.go needed a manual baseline bump, so the check
mostly caught its own staleness.
Remove the script, its make target and the two prerequisite edges, along
with the test-network: fixture markers that existed only to suppress it.
The isolation itself is untouched.
Assisted-by: Claude:claude-opus-5 [go vet]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* ci: keep hidden files in the offline test bundle artifact
Cherry-picked from 15a37b0ac on the remote branch. The offline bundle lives
under .cache/, which actions/upload-artifact skips by default, so the Linux
job packed an artifact missing the very file the next step restores.
The other half of 15a37b0ac moved test-network-lint out of the `test` and
`test-coverage` prerequisite lists into a recipe line, so parallel make could
not fingerprint the tree while generated fixtures were still changing. That
is dropped: the preceding commit removes the lint entirely, and the race it
worked around is one more reason a whole-tree fingerprint was the wrong
mechanism.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* refactor: share bounded exponential backoff
Use overflow-safe saturating arithmetic for retry delays across model import polling, downloads, registration, node operations, and model loading. Keep model import status checks responsive initially while capping their interval at 500ms.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* ci: mirror Jetson Python wheels
Keep the CUDA aarch64 wheel subset in GHCR and serve it as a local PEP 503 index during L4T backend builds, preserving last-known-good packages through upstream outages.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* docs(agents): index the Jetson wheels mirror
Mention the GHCR-hosted L4T wheel mirror in the CI caching guide summary so maintainers can find its outage and cache documentation.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* ci: add defensive build network proxy
Record build destinations and byte counts, retry observable idempotent HTTP downloads, and isolate explorer database tests that race under coverage.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(kokoros): implement updated backend trait
Return unimplemented for image upscaling, matching the backend's other unsupported modalities after the protobuf API update.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(ci): clear recovered proxy errors
Do not mark a request failed when a later safe retry succeeds.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* ci: require HTTPS build interception
Inject a short-lived proxy CA into BuildKit and Dockerfile RUN steps, reject plain HTTP and opaque tunnels, and retain method/status/byte telemetry for verified HTTPS traffic.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(ci): preserve system trust in unproxied builds
Mount the generated interception CA at a dedicated secret path and add it to the trust bundle only in proxy-aware dependency stages. This prevents optional secret mounts from masking the system CA bundle in ordinary backend test builds.
Install the requested Go toolchain before starting the proxy and satisfy cleanup error checks found by CI lint.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(ci): persist build proxy trust
Install the generated proxy CA through the system-managed local certificate directory so ca-certificates upgrades retain it. Avoid turning canceled matrix jobs into proxy cleanup failures.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(ci): trust proxy in nested build scripts
Install the build proxy CA before nested source fetches, route the DS4 package setup through the HTTPS mirror helper, and avoid repeated OCI setup in gallery behavior tests.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(ci): use HTTPS apt sources for Bonsai
Rewrite ARM64 package sources before installing GCC and check gallery fixture cleanup errors so the optimized tests satisfy errcheck.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(privacy-filter): trust build proxy CA
Install the mounted build proxy certificate before privacy-filter's make target fetches its HTTPS sources, for both source and prebuilt builder paths.\n\nAssisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: fail on hidden offline egress
Count cgroup-scoped firewall rejects and fail the offline test harness with bounded aggregate diagnostics. Inject the gen-audio GGUF probe so fixture-backed importer tests do not attempt real network access.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(ci): preserve system CA trust
Build a combined runner certificate bundle instead of replacing public roots with the generated proxy CA. Centralize additive container installation in the shared proxy CA helper.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add Q4_K_M and Q8_0 GGUF builds for Tencent UI-Mate-9B. Include the f16 vision projector so llama.cpp can process desktop screenshots.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The V8 coverage build inlines every dynamic import, so the stale chunk tests cannot intercept a page chunk. Run those tests against the normal code-split bundle and exclude them from the inlined coverage pass.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
A deploy replaces the whole content-hashed asset set at once. A tab
holding an older index.html, or one whose request lands on a replica
that the rollout has not swapped yet, asks for a page chunk the server
no longer has. The dynamic import rejects and React Router's default
error boundary replaces the app with "Unexpected Application Error!"
until someone reloads by hand.
The router now reloads the page itself when a chunk fails to load.
index.html is served no-cache, so the reload lands on a self-consistent
asset set. A timestamp in sessionStorage bounds this to one reload per
10 seconds, so a chunk that is genuinely gone reaches the error boundary
instead of looping forever.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m]
Nine commits in this repository, all merged: native Prometheus metrics for agent
chat runs (#10689), PII/audit events as a Prometheus counter (#10641), optional
Anthropic prompt-cache breakpoints (#11158), plus fixes to cloud-proxy, the agent
UI and the Python backend.
The author's public GitHub profile names the employer, which is the evidence this
file asks for. Counts verified with the GitHub commit search API on 2026-08-18.
Placed by commit count, between Spectro Cloud and VictoriaMetrics.
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
We run LocalAI as the inference layer of a self-hosted appliance for a German
B2B consultancy — local-only, on AMD Strix Halo (gfx1151) with ROCm.
In daily production use for agents with MCP tools, RAG over an internal
knowledge base, and a document/bookkeeping pipeline.
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
* feat(config): add context compression policy
Define the opt-in model configuration contract before the chat middleware consumes it. Document each policy field so later request handling does not invent a second schema.\n\nRefs #9534\n\nAssisted-by: Codex:gpt-5
* fix(config): register compression fields
The model editor metadata gate rejects new config fields without descriptions and suitable controls. Register the compression policy so operators can edit its six fields safely.
Assisted-by: Codex:gpt-5 [monitoring-prs]
* feat(chat): compress long contexts
Long conversations currently fail once they reach the model context window. The opt-in policy now summarizes complete older turns before primary inference and preserves the newest tool chains.
Both OpenAI and MCP chat routes share the same transformation. Usage metadata and metrics expose each compression event.
Refs #9534
Assisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update 0xShug0/audio.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(audio-cpp): map the MIDI task
audio.cpp now appends MIDI to its task enum. Keep the LocalAI mirror and conversion switches aligned so the backend builds against the new pin.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update mudler/vllm.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(vllm-cpp): track ABI v21
The updated engine exposes ABI v21 after extending its speech API. The Go
backend does not bind that API, so its existing mirrors remain valid.
The qwen3.5 warning fix is now present upstream, so the old patch no
longer applies.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* chore(llama-cpp): update upstream revision
Assisted-by: Codex:gpt-5.6
* fix(llama-cpp): refresh server patch contexts
The new llama.cpp pin changed the slot reset and prompt batch code. GNU patch accepted stale hunks with fuzz, which left the L4T build with invalid source.
Refresh both server patches against the pinned source so each hunk applies at its intended location.
Assisted-by: Codex:gpt-5
* fix(llama-cpp): adapt metrics result fields
The updated llama.cpp groups cumulative counters under server_metrics. Probe the result layout so the shared adapter also compiles against older forks.
Assisted-by: Codex:gpt-5
* fix(llama-cpp): refresh TTS patch offsets
GNU patch rejects the stale pre-decode hunk after the score patch changes the same file. Anchor the TTS hunks to the pinned llama.cpp source so the full series applies without fuzz.
Assisted-by: Codex:gpt-5.4
* fix(llama-cpp): normalize batch threads
The updated llama.cpp creates its batch threadpool during model initialization, before the context-level fallback can replace the -1 sentinel. Resolve that sentinel from the inference thread count so model loading does not overflow the threadpool allocation.\n\nAssisted-by: Codex:gpt-5.4
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(router): make KNN a first-class classifier with a persisted, curated corpus
Add `classifier: knn` — similarity-weighted voting over labelled
example prompts. Unlike score/colbert it needs no classifier model:
label knowledge lives in a corpus seeded and curated through the
admin API, so routing decisions are deterministic, auditable, and
grounded in graded experience rather than a model's opinion.
Epistemic gate: corpus entries below knn.similarity_threshold cannot
vote; when none clears it the classifier activates no labels and the
router uses the fallback — a prompt unlike all labelled experience is
treated as undecidable, not guessed. Decisions record
nearest_similarity (also on fallback rows) so admins can see how far
the nearest labelled experience was; the Routing tab explains
out-of-corpus fallbacks and shows per-label corpus counts.
Persistence: one JSONL file per router under
<data path>/router-corpus (text, labels, vector, embedder
fingerprint). The file is the source of truth; the local-store index
is rebuilt from it at classifier build time and stays a pure
in-memory index. Entries recorded under a different embedding model
re-embed on load. Also corrects the docs' false claim that
local-store collections persist — the embedding cache never survived
restarts (and still doesn't); the corpus does.
Corpus input is API-only by design (entries may contain example user
content): POST /api/router/{name}/corpus seeds (labels validated
against declared policies, embedded server-side, indexed
immediately), GET .../corpus/stats inspects — label counts only,
entry texts are never returned by any surface — DELETE .../corpus
wipes. Admin-gated like the sibling router endpoints, and exposed as
MCP tools (seed_router_corpus / get_router_corpus_stats /
clear_router_corpus) in both the httpapi and inproc clients with
coverage-test route mappings.
Plumbing: VectorStore gains SearchK (top-K was hardcoded to 1);
local-store gets InsertBatch/Delete as optional fast paths;
RouterConfig gains a knn block (embedding_model, k,
similarity_threshold, vote_threshold, store_name) with meta-registry
fields; the classifier dropdown now offers knn and the
previously-missing colbert; embedding_cache is ignored (with a
warning) for knn — it IS an embedding-KNN lookup; the stale
/api/instructions intelligent-routing entry is rewritten (it
described a classifier that no longer exists); swagger regenerated.
Tests: KNN vote/gate specs with hand-computed vote shares, corpus
manager suite (restart reload without re-embedding, fingerprint
re-embed, dedupe, hostile store names), middleware specs (corpus
routing, gate fallback, config validation, cache-wrap refusal),
corpus endpoint specs pinning the texts-never-returned contract, MCP
catalog + route-mapping gates, and a Playwright spec for corpus
stats and the out-of-corpus decision detail.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(router): name consulted corpus neighbours in knn decisions
Every knn decision (decision log rows and the /api/router/decide
response) now carries neighbors: the K retrieved corpus entries by
descending similarity - including ones below the epistemic gate, which
is what makes fallback decisions diagnosable - each as {id, similarity,
labels}. The id is the entry's content hash (first 8 bytes of the
SHA-256 of its text, hex): stable across reseeds and re-embeds, and
text-free, so an external platform that seeded the corpus can recompute
text->id on its own copy and bucket decisions by corpus region (per-
region reliability accounting) without corpus text ever leaving the
server. A corrupt index payload surfaces as an id-less neighbour at a
real similarity instead of disappearing.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* refactor(router): deduplicate knn plumbing and cut corpus hot-path waste
Post-review cleanup of the knn-first-class-router branch; no behaviour
changes on the API surface.
Reuse/altitude:
- RouterKNNConfig.ResolvedStoreName is now the single source of the
router-corpus-<name> default (was hand-derived in four files).
- corpus.ResolveKNNRouter + corpus.Seed carry the shared model
resolution and seed validation; the REST endpoints and the assistant
MCP client are thin transport adapters over them, with sentinel
errors mapped to HTTP statuses at the echo boundary.
- middleware.NewClassifierDeps assembles the classifier dependency set
once for all five entry points (OpenAI, Anthropic, realtime, decide,
corpus) instead of five hand-copied literals.
- router.AllClassifiers feeds both the status endpoint and the
unknown-classifier error, ending the classifier-list drift.
- Per-classifier requirements moved out of validateRouterPolicies into
their buildClassifier arms; the knn arm owns its embedding_cache
opt-out instead of a name-check in the shared wrap tail.
- adminOnly replaces four inline copies of the admin gate in the
middleware routes.
- localVectorStore.Search delegates to SearchK (identical traces).
Efficiency:
- Manager.Add embeds outside the manager mutex and appends to the
JSONL file (O(new) instead of O(corpus) rewrite); a torn tail from a
crash mid-append is tolerated on read and repaired on next write.
- Stats memoises per store keyed on the file's stat fingerprint and no
longer takes the manager mutex, so the 5s status poll stops parsing
vector-laden JSONL and stops blocking behind seeds.
- KNN Classify decodes each neighbour payload once (was twice) and
builds refs and votes in a single pass with one fallback return.
- Corpus file writes fsync before rename/close.
- The corpus manager is built eagerly in newApplication (sync.Once
dropped); test helper dead branch removed.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(router): bind knn corpus vectors to an embedder fingerprint and fail closed on mismatch
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* chore(mcp): align corpus tool prompts and the mutating-tool safety list
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(proto,backend): report embedding shape from the llama-cpp backend
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(embeddings): Go-side pooling — mean/last/decayed_mean with half-life
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(embeddings): accept chat messages[] and per-request pooling on /v1/embeddings
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* chore(middleware): name the failing fields when post-merge validation 400s
An intermittent post-merge validation failure surfaced as an opaque 400
during integration (pooling scheme mismatch that no client had sent).
Log the model, the request's pooling override, and the merged config's
pooling fields at the failure point so the next occurrence identifies
whether the request or the stored config carried the bad value.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(embeddings): scheme override must not inherit the config's half-life
A model config defaulting to decayed_mean pooling carries
pooling_half_life_tokens; a request overriding the scheme to mean/last
without its own half-life inherited that value, and post-merge
validation rejected the pair the server itself had assembled. Zero the
inherited half-life when the overridden scheme is not decayed_mean; a
request that explicitly pairs a half-life with a non-decayed scheme
still 400s.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix embedding pooling validation and router bounds
Declare backend embedding layouts and reject incompatible pooling modes. Reset local-store dimensions after a full clear, validate KNN thresholds, and add real backend and store integration coverage.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* ci: run local-store integration tests
Build and install the local-store backend in the Linux test job, then run the existing store integration suite so new specs are discovered automatically.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
The official GGUF repository publishes both Q4_K_M and Q8_0 builds, but the gallery only exposed Q4_K_M. Link the higher-quality Q8_0 build so capable hosts can select it automatically.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The transform WebSocket accepted any model and opened its frame-based RPC. Any-to-any models use a different stream contract, so liquid-audio failed with an unimplemented RPC after the handshake.
Reject incompatible model use cases before loading the backend. Direct realtime-audio callers to the OpenAI Realtime API.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The upstream editable install pins generic PyTorch packages. It
replaces the HIP wheels with CUDA wheels in ROCm images.
Remove those pins only for hipBLAS builds before the editable install.
Keep the existing CPU and CUDA dependency behavior unchanged.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
feat: bound backend admission and expose running traces
Add process-wide backend execution admission without blocking UI or administrative HTTP work. Represent backend operations while they are in flight, surface running traces with immediate log links, and tie streaming admission leases to the gRPC receive lifecycle.
Assisted-by: OpenAI Codex: GPT-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Add the 2B and 4B reasoning distillations in Q4_K_M and Q8_0 formats. These sizes extend the existing Qwen3.8 family to compact and edge hosts.
Assisted-by: Codex:gpt-5 [web]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Branch protection rejects the weekly workflow's direct push to master. Reuse the repository's create-pull-request automation so counter updates go through the protected-branch review and CI path.
Assisted-by: Codex:gpt-5 [actionlint]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add the smaller 3.69-bit mixed quantization to the existing Qwen3.8 27B variant group. Enable its embedded MTP head so compatible hosts can prefer speculative decoding.
Assisted-by: Codex:gpt-5 [Web]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add Q4_K_M and Q8_0 llama.cpp builds with the shared vision projector. The MIT-licensed agentic coding model is absent from the current gallery.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update mudler/vllm.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(vllm-cpp): mirror ABI v20 layouts
The dependency bump advances the engine ABI from v17 to v20. The old
binding stops every backend build at the ABI guard and undersizes
structures used at runtime.
Mirror the appended model and video fields so every platform uses the
pinned engine layout.
Assisted-by: Codex:gpt-5
* fix(vllm-cpp): satisfy Apple Clang
The new engine pin captures a namespace-scope help string in a lambda. Apple Clang rejects the redundant capture because upstream enables -Werror.
Carry the one-line source patch until the engine pin includes the fix.
Assisted-by: Codex:gpt-5
* fix(vllm-cpp): align the carry patch
The Apple Clang patch used context from another source revision.
Source preparation rejected it before every backend build.
Align the patch with the pinned engine revision.
Assisted-by: Codex:gpt-5 [monitoring-prs]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add Q4_K_M and Q8_0 llama.cpp builds for BAAI AREX-Turbo. The compact research agent is absent from the current gallery.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(ui): add installed model lifecycle
Models now owns catalog exploration and installed runtime controls under one canonical route. URL-owned state keeps lifecycle context recoverable through links and browser history.
Assisted-by: Codex:gpt-5 Playwright
* feat(ui): add installed backend lifecycle
Backends split discovery from backend-binary management. The canonical
page now keeps both lifecycle views under one URL-backed shell while it
preserves target-node placement.
Assisted-by: Codex:gpt-5 Playwright
* fix(ui): repair lifecycle state updates
Installed models lost distributed refreshes and kept a deleted selection. Backend searches also stopped tracking URL changes, while batch upgrades stopped after their first error.
Preserve background refreshes and finish each requested batch action. Drive catalog results from URL-backed state without losing full metadata.
Assisted-by: Codex:gpt-5 [Playwright]
* feat(ui): make resource pages canonical
Replace Host navigation with canonical Models and Backends lifecycle routes, preserve legacy management URLs, and surface shared host capacity on the Operate overview.
Assisted-by: Codex:gpt-5 [Playwright]
* feat(ui): complete canonical resource lifecycle
Finish the responsive list-to-detail behavior, remove the retired Host implementation, and keep Explore focused on discovery while Installed owns destructive actions. Update regression coverage, localization, documentation, and development binding for the canonical resource pages.
Assisted-by: Codex:gpt-5 [Playwright]
* docs(ui): record the UI design context
Record the approved users, brand character, and design principles so
future interface work uses the same product direction. Index the context
from the repository's agent instructions.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Adds eight curated vllm-cpp entries to the model gallery. Until now the
backend had gallery coverage only for MiniMax-H3 video, so serving text on
it meant hand-writing engine_args.
The flagship tier is what vllm.cpp gates its correctness and speed claims
on: Qwen3.6-27B and Qwen3.6-35B-A3B in NVFP4, each with a speculative
sibling (MTP on both, DFlash on the 27B). Qwen3-Coder-30B-A3B covers
agentic tool use, and Qwen3-4B / Qwen3-0.6B in bf16 are the entries that
run where NVFP4 cannot, CPU included.
Three details are load-bearing rather than incidental:
- The 27B entries pin revision 890bdef7. That repository was later
re-quantized in place from NVFP4 to FP8 W8A8 under the same name, so an
unpinned entry resolves to different weights and reports nothing.
- Qwen3-Coder names tool_parser: qwen3_coder explicitly. Its dialect is
byte-identical on the wire to step3p5's, so chat-template sniffing
cannot separate them and auto-detection picks wrong.
- enable_prefix_caching is deliberately left unset everywhere. It defaults
on for dense models and off for the GDN hybrids, and that per-model
default is the right answer.
num_blocks is sized per model from its real KV footprint rather than
copied between entries, which ranges from 20 KiB/token on the 35B to
144 KiB/token on the 4B.
Docs: adds features/vllm-cpp.md covering installation, the model table,
the pinning rationale and how to choose between the speculative variants,
and cross-links it from the existing engine_args reference. It also
records that the CUDA images are built for Blackwell only, which is
narrower than vllm.cpp's own ten-architecture release and makes an
otherwise cryptic "no kernel image is available" failure legible.
Verified: gallery suite green; all eight decode and validate as a
ModelConfig. qwen3-0.6b-vllm-cpp confirmed end to end on a real cluster,
chat plus engine-parsed tool_calls. The NVFP4 entries are not yet
runtime-verified: no available node has kernels for them.
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Bash] [Edit]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
A remote can serve stale or corrupted bytes for one request. Mark the
integrity failure as transient so the bounded download planner retries it.
Assisted-by: Codex:gpt-5.6
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(advisorylock): set statement_timeout alongside lock_timeout
WithLockCtx already overrides a deployment-wide lock_timeout on its
dedicated connection so a blocking pg_advisory_lock() waits its turn
instead of failing with 55P03. statement_timeout aborts that exact same
statement independently, with SQLSTATE 57014, and was not overridden.
Production roles commonly carry statement_timeout=60s. Any guarded
section longer than that (a cold model load stages for tens of minutes)
therefore killed every concurrent waiter:
advisorylock: acquiring lock 9003261067483446873: ERROR: canceling
statement due to statement timeout (SQLSTATE 57014)
Derive it from the same context budget as lock_timeout, with a matching
RESET so the pooled connection is returned clean.
Assisted-by: Claude Opus 5 [claude-code]
* feat(distributed): add ModelLoadJob, the durable cold-load record
A cold load in distributed mode is a long-running background job, but it
was modelled as a synchronous side effect of an inference request: the
whole of it (backend install, multi-GB staging, checkpoint load) ran
inside the per-model advisory lock. Loading a 35.7 GB GGUF held that lock
for ~20 minutes, so every concurrent request for the same model blocked
on pg_advisory_lock and died at the role's 60s statement_timeout.
Introduce the row that lets the lock shrink to a decision. Exactly one
ModelLoadJob may be active per tracking key; that uniqueness — not the
lifetime of a lock — is what de-duplicates concurrent loaders across
replicas. ClaimLoadJob does its read-then-write under the advisory lock
and nothing else: no network, file or gRPC I/O inside the guarded
section, so a claim costs milliseconds no matter how long the resulting
load takes.
LastProgress is a heartbeat rather than a byte counter. A checkpoint load
legitimately moves zero bytes for many minutes, so a reaper keyed on byte
movement would reclaim a healthy job mid-load; byte progress stays the
concern of load_deadline.go. A job whose heartbeat stops for longer than
the orphan window is reclaimable, so a replica killed mid-load cannot
wedge a model permanently.
Failed jobs keep their row for a short grace so an immediately-following
request reports the real cause instead of silently starting a fresh load
of a model that just failed.
No caller yet — the router moves onto this in the next commit.
Assisted-by: Claude Opus 5 [claude-code]
* refactor(distributed): run cold loads as jobs, outside the advisory lock
Route wrapped the entire cold load — node selection, backend install,
multi-GB staging and the remote LoadModel — in the per-model advisory
lock. The lock's job is to de-duplicate concurrent loaders, a decision
that takes milliseconds; holding it for the tens of minutes the resulting
work takes is what turned a dedup mechanism into a cluster-wide outage
for that model.
Split it into a claim and a run. The claim is the only thing left inside
the lock. The run is a background job owned by the claiming replica and
bounded by the same progress-extended deadline as before; every other
request for that model — local or on another replica — attaches as a
waiter and is served the moment the model is ready, with no duplicate
load and no lock contention.
Waiters share one broadcast rather than an ordered queue: they all want
the identical outcome, so ordering them would add fairness machinery that
changes no result. The local channel wakes same-replica waiters instantly
and a 2s DB poll is the authority, because a waiter on another replica
has no channel to close. On wake a waiter re-runs the warm path rather
than trusting the signal — the model may have been evicted in between.
A waiter whose client disconnects returns immediately and the job keeps
running; it belongs to the job record, not to the request. A failure is
recorded on the row so every waiter reports the real cause, and the row
survives briefly so the next request does not read "no job" as "not
loading" and start a duplicate load of a model that just failed.
The runner heartbeats the row on a fixed interval whether or not bytes
are moving, which is what keeps a legitimately silent checkpoint load
from being reclaimed as an orphan. Phase (installing/staging/loading) and
placement ride to the heartbeat on the context, the same seam
load_deadline.go already uses, so single-host paths are untouched.
Non-distributed mode (no DB) keeps the inline load exactly as it was.
Assisted-by: Claude Opus 5 [claude-code]
* feat(distributed): bound the wait for a loading model and answer with progress
A request whose model is cold-loading now attaches to the running job and
is served the moment the model is ready. That wait has to be bounded: a
held HTTP request cannot survive real infrastructure, and an ingress or LB
idle timeout kills a twenty-minute request regardless of what LocalAI
does.
New LOCALAI_MODEL_LOAD_WAIT (default 60s) bounds the CALLER, never the
load — the job keeps running either way. On expiry the request gets 503
with Retry-After and a structured body naming the model, the node, the
phase, byte progress and an ETA. The `error` envelope keeps OpenAI
clients working; `loading` is additive so they ignore it.
The ETA comes from the job's own observed rate and is omitted rather than
guessed until enough bytes have moved for that rate to mean anything: a
confidently wrong ETA on a twenty-minute wait is worse than none.
Retry-After is that ETA when known, clamped to [5s, 300s], and the wait
budget otherwise.
LOCALAI_MODEL_LOAD_WAIT=0 waits unbounded, for deployments with no proxy
in front. Zero in the config struct still means "unset, use the default",
so the CLI records the operator's zero as ModelLoadWaitUnbounded rather
than losing the distinction.
The distributed branch of ModelLoader.loadModel wrapped the router's
error with %s, which flattened it to a string. Use %w: the typed error is
what the HTTP layer keys the 503 off.
Assisted-by: Claude Opus 5 [claude-code]
* feat(api): add GET /api/models/{id}/load-status
A client that receives 503 while a model stages onto a worker needs
somewhere to poll. This returns the same `loading` object the 503 carries
— phase, node, byte progress and ETA — or 404 when no load is running.
Read-only and observability-shaped, so it is deliberately neither
admin-gated nor feature-gated: it explains a 503 the caller just
received, and hiding that behind a per-modality feature would make the
explanation for a failed image request depend on chat permissions. It
also gets no MCP tool, since there is nothing here an admin would manage
conversationally.
Registered on the surfaces from .agents/api-endpoints-and-auth.md: the
swagger block (existing `models` tag, so /api/instructions needs no new
area), the endpoint discovery maps in RegisterLocalAIRoutes, regenerated
swagger, and the distributed-mode docs page. No FLAG_* usecase is
involved, so capabilities.js is unchanged.
Assisted-by: Claude Opus 5 [claude-code]
* feat(ui): show cold-load progress in Chat and retry when the model is ready
A chat request for a model that is still staging onto a worker now gets a
503 carrying live progress instead of an error. Render it: the composer
shows the phase (installing / staging / loading), the node, the percent
and the ETA, then polls load-status and re-sends the request the moment
the model is ready.
Reuses the staging progress idiom the page already had rather than
inventing a second one — the two sources are folded into one
loadProgress, with the load job winning because it is authoritative
across frontend replicas and knows the phase, where the staging operation
only knows about a byte transfer this replica happens to be performing.
Waiting is bounded (three send attempts, ~30 min of polling each), so a
load that never finishes still surfaces as an error rather than as a
spinner nobody questions. An aborted generation stops the polling too.
Assisted-by: Claude Opus 5 [claude-code]
* fix(distributed): check warm-path cleanup errors
The router moved legacy cleanup calls onto newly linted lines. Report
cleanup failures while preserving the fallback to a cold load.
Assisted-by: Codex:gpt-5 [golangci-lint]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add LiquidAI’s compact edge model in Q4_K_M and Q8_0 builds. The
variant pair lets LocalAI choose the highest-quality build that fits.
Assisted-by: Codex:gpt-5.4
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Unexpected runtime exits only reported an exit code, which hid the backend diagnostic. Include the final non-empty stderr line when one exists.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The Darwin workflow passes BUILD_TYPE=metal but does not define OS=Darwin. The backend therefore omitted its Metal CMake flags and shipped the runtime source path instead of the embedded library.
Map the requested build type directly to the Metal flags and guard the build contract with a dry-run regression test.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add the official Q4_K_M and Q8_0 GGUF files with the shared vision projector. Include an MTP variant for speculative decoding.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(ui): keep agent import action visible
The header hid its full import label after the agent list became non-empty. Hide only the nested file input so users can import more agents.
Assisted-by: Codex:gpt-5
* test(ui): match the agent import label
The Agents page renders the action as Import.
The test searched for Import Agent, so it failed before checking visibility.
Mock the observables request to remove backend timing from the fixture.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The vllm-cpp CUDA images were built for Blackwell only: 120a;121a on
amd64 and 121a alone on arm64. vllm.cpp's own release archive builds ten
architectures, so LocalAI shipped one or two of them.
The failure mode is the problem. An unlisted card is not slower, it dies
at the first request with "no kernel image is available for execution on
the device", long after `backends install` reported success. That covers
A100, A10/3090, L4/4090/RTX 6000 Ada, H100/H200, B200, B300, Jetson Orin
and Jetson Thor, and it is how a Jetson Thor node was found serving
nothing at all.
amd64 now builds 80;86;89;90a;100a;103a;120a;121a and arm64 builds
87;90a;100a;110;121a, split by where the silicon exists: Jetson is
arm64-only, desktop 120a is amd64-only, and 90a/100a are on both because
of GH200/GB200.
Triton-AOT stays ON for both, which the old comment said was impossible.
It is not, at the version we pin: only maintainer REGEN needs a single
arch, while the BUILDER path embeds every vendored cubin tree and selects
by exact SM, so 87/103a/110/120a take the portable CUDA kernels and can
never load a neighbouring cubin. Upstream ships its ten-SM archive that
way.
The CUDA 13 guard now covers both branches rather than amd64 alone. arm64
needs compute_121a just as much, and CI already builds it with 13.
Cost is smaller than the arch count suggests, because gencode is
per-source: fp4-mma still resolves to 120a;121a, and the CUTLASS
scaled-mm kernels to one arch each, so the added architectures do not
multiply the expensive translation units.
Verified: flag generation checked for both branches, CUDA 12 still
refused, CPU build untouched; both arch lists expanded through vllm.cpp's
own vt_cuda_gencode_options and per-feature arch gating, and all six
vendored Triton trees confirmed intact, at the exact pinned commit. A
real compile is CI-only: there is no CUDA toolchain on the dev box.
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Bash] [Edit]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): refresh the TTS patch
The llama.cpp update moved and changed the generated-audio pipeline. Refresh the carried patch so backend builds can apply it to the new revision.
Assisted-by: Codex:gpt-5.4
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(mcp): surface configured server failures
Keep model-configured MCP servers visible when discovery or connection setup fails, propagate status through distributed discovery, and let the Chat UI show actionable errors while retrying unavailable servers.
Add model-editor metadata for remote and stdio configuration and document the expected format, deployment networking boundary, and alternate MCP scopes.
Assisted-by: Codex:gpt-5 Ordino golangci-lint
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* build(compose): match CUDA development image
Configure the API image with the cublas, CUDA 13, auth-tagged build settings used by the local development Makefile invocation, including the 24-way Docker build.
Assisted-by: Codex:gpt-5 Ordino
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* revert: keep host build settings out of compose
The CUDA development deployment is managed from ~/docker/localai, not the repository example Compose file. Restore the generic example and keep machine-specific build settings in the host deployment.
Assisted-by: Codex:gpt-5 Ordino
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(docker): exclude local agent artifacts
Keep Claude worktrees and locally installed verification tools out of the Docker build context. These host-only directories added roughly 1.9 GB to every root image build.
Assisted-by: Codex:gpt-5 Ordino
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Forward BUILD_TYPE=metal to PARAKEET_GGML_METAL so macOS backend artifacts include Metal support.
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: fierywaters <fierywaters13@gmail.com>
Add the smaller Fara1.5 computer-use model alongside the existing 9B and 27B gallery entries. Offer Q4_K_M and Q8_0 builds so LocalAI can select for memory or quality.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The DeepSeek V4 Flash base entry points at a Hugging Face repository page instead of a GGUF object. When variant probing cannot rank a concrete build, the base fallback downloads no usable model weights.
Use the validated IQ2XXS object and checksum already shipped by the q2 variant. Pin that payload in the gallery resolution test.
Assisted-by: Codex:gpt-5.6
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add the publisher-recommended Q4_K_M build and a near-lossless Q8_0 variant for the 3B formal-logic reasoning model.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add the official llama.cpp GGUF builds for NVIDIA Nemotron 3.5 Lightning 30B-A3B, linking Q4_K_M, NVFP4, and Q8_0 as installable variants.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The validated gallery checksum changed in #11459, but its dedicated regression assertion kept the previous value and now fails the master test suite on Linux and macOS.
Assisted-by: Codex:gpt-5.4
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update ikawrakow/ik_llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(gallery): align Higgs Audio checksum test
The validated gallery checksum changed in #11459, but its dedicated regression assertion kept the previous value and now fails the master test suite on Linux and macOS.
Assisted-by: Codex:gpt-5.6
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
LOCALAI_WEBRTC_ICE_INTERFACES was silently ignored whenever
LOCALAI_WEBRTC_UDP_PORT was set. Every interface was gathered regardless
of the allow-list, so a browser was handed the docker0/veth addresses the
setting exists to suppress, and the connection established on a good pair
and then dropped when consent checks failed on the unreachable ones.
Two things combine to cause it. A mux built over a wildcard socket makes
pion derive its host candidates by enumerating interfaces itself, with a
nil filter and loopback included. Independently, the muxed gathering path
in pion/ice never consults SetInterfaceFilter, so setting it has no effect
there either.
Bind one socket per admitted interface address via NewMultiUDPMuxFromPort,
which takes the filter, instead of one wildcard socket. All the sockets
share the same port, so the firewall requirement is still a single rule.
Networks are pinned to UDP4 to match the socket family this replaces.
An allow-list that matches no address on the host now reports the
misconfiguration rather than binding nothing and leaving signaling to
succeed while no candidate is ever advertised.
Two tests: one asserts an unmatched allow-list is an error, and one gathers
against a real peer connection and asserts no address outside the allowed
interface appears (skipped on single-interface hosts).
Assisted-by: Claude:claude-opus-5 go vet gofmt
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
The gallery checksum did not match the Q8_0 object currently published by audio-cpp. Use the SHA256 reported by both Hugging Face LFS metadata and the live x-linked-etag so model installation can verify the download.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Eight header controls across seven pages had two or three elements' classes
collapsed into one string. The wrapper or the icon ended up wearing the
button classes, and the buttons themselves were left with no class at all,
so they rendered in the browser's own chrome. Reported on Agent Jobs; the
grep found the rest.
`fas` does not draw anything by itself: it sets
`font-family: "Font Awesome 6 Free"` and weight 900 on whatever carries it,
and the `fa-*` class supplies the glyph via ::before. So
<button className="btn btn-primary fas fa-plus">
renders its own label "New Task" in the icon font, and
<div className="hstack btn btn-primary btn-sm fas fa-edit btn-secondary fa-arrow-left">
<button>Edit</button>
<button>Back</button>
</div>
styles the flex wrapper as a button that is both primary and secondary,
points two glyphs at one ::before, and leaves both real buttons bare.
Fixed, all of them keeping the correct `<i>` child they already had:
- AgentJobs, AgentTaskDetails (x2), AgentCreate - icon classes off the
button.
- AgentTaskDetails, AgentJobDetails - wrapper back to plain `hstack`, and
the two buttons inside each get the variants the wrapper had been
holding. Back is secondary and leads, Edit/Cancel is the emphatic one
and trails, matching every other detail header.
- VoiceLibrary, VoiceProfileCreate - the title `<i>` had swallowed the
action link's classes, so "Create voice" and "Back to library" were
unstyled anchors. Back was also drawing a "+" because it had inherited
fa-plus while its own fa-arrow-left sat up in the title.
- P2P - a stray fa-circle-info on the title icon.
The ninth instance was ImportModel, where this class of bug was first
found. #11461 rewrote that file and landed first, so nothing is left to fix
there.
Guarded by e2e/class-hygiene.spec.js, which reads the source rather than
walking routes: several of these pages need agent or voice data before they
render a header, so a route walk would skip exactly the pages that had the
bug. It fails on an icon-font class outside an `<i>`/`<span>`, on two glyphs
or two button variants on one element, and on a layout wrapper that is also
a button. Font Awesome modifiers (fa-spin, fa-fw, sizes) are excluded, so
the `fa-spinner fa-spin` idiom stays legal.
e2e: 428 passed.
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] [Playwright]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The import page took the new palette in #11305 but kept its old layout, so
it stayed a 760px column with the primary action detached from the form it
submits. Two of the problems were outright bugs.
The Import button carried no className at all, so the page's single most
important control fell through to the user-agent button: system chrome,
wrong radius, no design-system focus ring. The YAML button carried
`fas fa-save fa-upload`, which sets Font Awesome as the button's own font
family (its label text inherits it) and points two glyph classes at one
::before.
On the layout: `page--narrow` is documented for "forms / single-record edit
views", and in Advanced mode this page held a URI field, a six-section
format guide, ten modality chips, nine preference fields, a key-value
repeater and a YAML editor at `calc(100vh - 400px)`. The width was the
symptom; one column was the disease.
- `page--medium` with a work column and a format reference beside it.
The reference answers the only question a first-time admin has and used
to sit behind a chevron, closed by default. Below 1024px it becomes a
disclosure rather than disappearing.
- The source field is the hero: monospace, because it holds something you
paste, and it carries its own Import button. That removes the hidden
aria-hidden submit button that existed only because the real action sat
outside the form.
- Simple and Advanced are gone. They were ~80% the same surface, and the
overlap cost a mode switch, a localStorage key and a three-button
Keep/Discard/Cancel dialog whose only job was protecting state that
switching modes would hide. One form with a collapsible options panel
hides nothing, so none of it is needed. What genuinely differs is the
kind of input, which is now the two tabs: a source, or YAML.
- The size/VRAM estimate reports under the field that produced it instead
of as a banner above the page header, and an import in flight gets the
progress, phase and byte counts the poller already returned and the old
status card threw away.
- ModalityChips resolves its labels through the same `modality.*` keys as
the dropdown it filters. It hardcoded English shorthand, so one modality
carried two names on one screen ("Speech" on the chip, "Speech
recognition" on the group it scrolled to) and seven locales had neither.
Its inline styles and its pill radius move onto the design system.
- Three inline styles go, including both conditional-padding hacks; the
only one left is the progress bar's runtime width. Baseline 538 -> 535.
Docs updated in the same change: the WebUI section described a Simple and an
Advanced mode and told the reader to "Toggle to Advanced Mode".
e2e: 426 passed. The mode-switch suite is replaced by one covering the tabs
and the disclosure, and a new layout suite pins the width, the styled
primary action, the absence of an icon-font button, the reference column at
both widths, and the estimate's position.
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] [Playwright]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(model): surface backend startup exits
Preserve the local backend process exit code and bounded stderr diagnostic when the process dies before its gRPC service becomes ready.
Fixes#9050
Assisted-by: Codex:gpt-5
* fix(model): satisfy startup diagnostic checks
Assisted-by: Codex:gpt-5.6 [Codex]
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(ci): parse current vllm-metal version pins
vllm-metal renamed its installer pin from vllm_v to VLLM_VERSION, breaking both the nightly bumper and the Darwin backend installer after a bump. Share a strict parser that accepts both formats and cover the transition with shell regressions.
Assisted-by: Codex:gpt-5 [Codex]
* fix(ci): parse scoped vllm-metal pins
The pinned vllm-metal installer declares its version as a local shell variable. Accept that optional declaration while retaining strict validation of the assignment and semantic version.
Assisted-by: Codex:gpt-5.4
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add Q4_K_M and Q6_K llama.cpp builds with the matching vision projector and verified Hugging Face artifact hashes.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add the text-only IQ2_XXS build with the model card's required Jinja, reasoning parser, KV cache, and sampling defaults.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Expose the existing audio.cpp Higgs support as an installable Q8 gallery model and document voice cloning and licensing constraints.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add the official dynamic and 17 GB K-quant GGUF builds with the shared multimodal projector and optional DFlash acceleration.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(realtime): add shared WebRTC UDP port
Allow realtime WebRTC peer connections to reuse one configurable UDP mux, and surface listener bind failures through signaling.
Assisted-by: Codex:gpt-5
* test(realtime): keep UDP mux alive during bind check
The returned SettingEngine owns the UDP listener. Retain it through the duplicate-bind assertion so macOS cannot finalize the listener early and make the exclusivity check spuriously pass.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* test(realtime): use IPv4 for UDP mux checks
Match the socket family used by the WebRTC UDP mux so macOS does not allocate an IPv6 probe that can coexist with the IPv4 listener.\n\nAssisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add official multimodal GGUF builds for the new BigBang-v1 scientific and agentic reasoning model, including Q4_K_M and Q6_K variants with MTP enabled.\n\nAssisted-by: Codex:gpt-5 [Hugging Face API]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
MiniMax-H3 ships two independently-served DiT partitions and each refuses the
other's tasks. The gallery had only the FL2VA one (t2va + fl2va); this adds the
Ref2VA checkpoint, which is what serves REFERENCE conditioning - a reference
image, a reference clip, or reference audio prepended as their own blocks.
It is a genuinely separate 19.9 GB download, not a mode flag, which is why it
needs its own entry. Everything except the DiT is shared with the FL2VA entry.
Q4_K_M specifically, and the description says why: the NVFP4 Ref2VA weights
render a multicolour patch grid. Three upstream investigations attributed that to
the ref2va reference-row path before the checkpoint was varied; on Q4_K_M the
identical code renders coherently (period-16 seam 1.13 against 2.28 broken,
VAE-input latent adjacent-cell cosine 0.8526 against 0.06 white). Pointing users
at NVFP4 here would hand them that dead end.
known_input_modalities carries image, video AND audio, unlike the FL2VA entry:
the ABI's ref2va slots are ref_image (PPM), ref_video (a directory of
frame_%06d.ppm) and ref_audio (16-bit PCM WAV), and the backend's
checkPartitionConditioning already routes all three.
Checksum verified two ways: sha256 computed over the downloaded 19,864,208,064
byte file, and HuggingFace's x-linked-etag for the same URI - both
17925612821ea3037ffaf5f7f9789f5460e87025385bd45e9ec6c7d536684d56.
Not verified here: an end-to-end render through LocalAI itself. The render was
proven with vllm.cpp's own driver on this exact checkpoint; the backend path is
the same library seam but has not been exercised with these weights.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(config): do not read a TTS speaker-encoder mmproj as vision support
Qwen3-TTS on llama-cpp ships an mmproj holding the speaker encoder and
code predictor. VisionSupported() treated any non-empty MMProj as proof
of image input, so every such model would be advertised as vision-capable.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(llama-cpp): add TTS request option parsing helper
Validates text and speaker reference presence and strictly parses the
top_k / top_p per-request params, in a header with no llama.cpp or gRPC
dependencies so the standalone C++ unit test gate picks it up.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(llama-cpp): range-check the TTS top_k and top_p request params
Format validation alone let NaN, infinity and out-of-range values through.
The consumer copies both values into the audio generation input
unconditionally and only guards its separate sampler assignment with
"> 0", a test NaN also fails, so a NaN reached llama.cpp with the guard
never firing. top_k must now be >= 0 and top_p must fall within 0.0 to 1.0
inclusive, with the bound written as a negated in-range test so NaN is
rejected rather than silently accepted.
Also cover the two checks the suite could not previously kill: the
whole-string check in the float parser and the int32 range check.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(llama-cpp): bump pin to f9e832c10 and carry the TTS server task
Picks up ggml-org/llama.cpp#26254 (Qwen3-TTS via mtmd) and #26536 (the
short-input audio chunk fix). Adds 0002-add-server-task-type-tts.patch,
the server-side half of the still-draft #26603, so TTS runs through the
slot scheduler instead of racing it. Remove that patch when #26603 merges.
The patch is rebased on top of the score patch: its tokenize-switch hunk
collided with the SERVER_TASK_TYPE_SCORE case, and its lone SRV_WRN call
passes no variadic argument, which the macro cannot expand. The score
patch itself needed no refresh.
Also fixes fallout from the bump in grpc-server.cpp: upstream dropped the
per-slot n_ctx argument from server_schema::eval_llama_cmpl_schema. Only
the schema branch loses it, since forks predating the server-schema split
still expect the old argument list.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(llama-cpp): implement the TTS and TTSStream RPCs
Both were declared in backend.proto but unimplemented. They now submit a
SERVER_TASK_TYPE_TTS task and drain the response reader, the same shape
PredictStream uses.
The streaming path emits a leading sample_rate message and then raw PCM,
because ModelTTSStream builds the WAV header itself; the non-streaming
path emits a complete WAV to the requested dst.
The streamed samples are converted from the pipeline's float32 to signed
16-bit first. MTMD_HELPER_GEN_AUDIO_OUTTYPE_PCM hands back floats, while
the header ModelTTSStream writes announces 16-bit samples, so shipping
the floats verbatim would decode as noise.
prepare.sh and CMakeLists.txt now stage tts_request_options.h alongside
the other grpc-server helpers, and register its standalone test with
ctest the way passthrough_options_test is registered.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(llama-cpp): mask non-codec tokens for Qwen3-TTS generation
The Qwen3-TTS gen-audio pipeline maps a sampled backbone token to a
codebook row with an unchecked subtraction, in mtmd-helper-gen.cpp:
inp.code0 = sampled - codec_0;
For ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF the vocab is 155008 tokens,
<|codec_0|> is 151936 and the codec codes end at 153983. The model's own
tokenizer.ggml.suppress_tokens holds 1023 ids covering 153984..155007,
every special above the codec range except <|codec_eos_token|> (154086)
which stays reachable as the stop token. Nothing masks the text range
0..151935, so the backbone can sample a text token at any step, the
subtraction goes negative, and ggml_compute_forward_get_rows aborts the
whole backend process on GGML_ASSERT(i01 >= 0 && i01 < ne01).
Complete the mask upstream started: bias every token below <|codec_0|>
to -INFINITY for TTS tasks so only codec codes and the codec EOS remain
reachable. The biases are appended to task.params.sampling.logit_bias,
which common_sampler_init already merges with the model's suppress
tokens into one llama_sampler_init_logit_bias, so no sampler is added to
the chain. Measured cost is 0.082 ms per sampled token and 1.16 MB, set
against a forward pass in the multi-millisecond range.
It lands in launch_slot_with_task rather than in a route handler so that
llama.cpp's own POST /tts and LocalAI's TTS/TTSStream RPCs are both
covered, and <|codec_0|> is resolved from the vocab rather than
hardcoded so a model without it is left alone.
This is reproducible with upstream's own llama-tts and no LocalAI code
loaded, aborting at frame 55 on Q4_K_M and frame 71 on Q8_0, so it is
neither a quantization artifact nor an artifact of the gRPC adapter.
Two further defects in the same draft pipeline still prevent end-to-end
audio; they are independent of this one and are recorded in the task
report for an upstream bug report.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(llama-cpp): bump pin to 9de0fcf2b and drop the TTS codec mask
Upstream fixed the Qwen3-TTS abort in ggml-org/llama.cpp c8e03ce81
("mtmd/ggml: add ggml_build_forward_order", #26649), landed one hour
after the previous pin. ggml_build_forward_expand marks a tensor and all
its ancestors for compute, so using it as a pure ordering hint defeated
ggml_build_forward_select and made GEN_WAV calls execute the GEN_CODE
branch against a stale inp_code0, hitting the get_rows bound assert in
ggml_compute_forward_get_rows.
That single defect accounts for every abort seen on this model, so
0003-mask-non-codec-tokens-for-tts.patch is removed rather than rebased.
The mask changed the observed behavior, but it was perturbing a graph
ordering bug rather than fixing a sampling one: at the new pin the whole
path works without it. Keeping it would have meant carrying a 152k-entry
logit bias, and rebasing it on every pin bump, for no benefit.
Verified at 9de0fcf2b with only 0001 and 0002 applied, which both apply
clean with no fuzz and needed no rebase:
non-streaming HTTP 200, 410924 bytes, 8.56 s
RIFF (little-endian) data, WAVE audio, Microsoft PCM,
16 bit, mono 24000 Hz
streaming HTTP 200, 560684 bytes, 11.68 s, exactly one RIFF at
byte 0, same format, which also exercises the
float32-to-s16 conversion at runtime for the first time
Pristine unpatched llama-tts at the same pin now also completes, 130
frames to a valid WAV, where it aborted at frame 55 before.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(llama-cpp): clear the TTS slot sequence between requests
Only the first TTS request in a backend process succeeded. Every later
one failed instantly, in about 0.13 s, with "TTS prompt processing
failed" from step_prompt, regardless of streaming or non-streaming and
regardless of the text. With LOCALAI_SINGLE_ACTIVE_BACKEND=true the
process is kept alive between requests, so a deployment would have
served exactly one utterance per backend start.
The cause is missing KV hygiene, not anything in the gRPC adapter. TTS
slots never enter the shared batch: pre_decode() returns early for them
and process_tts_slots() drives them instead, so they skip the
prompt-cache bookkeeping that clears a slot's sequence between requests.
Nothing in the gen-audio path makes up for it: mtmd_helper_gen_audio_reset
only clears host-side buffers, and the pipeline always decodes from
position 0 into the sequence identified by slot.id. So the second task
on a slot writes positions 0..N over the first task's tokens and
llama_decode fails.
Fix is one call to slot.prompt_clear(), the same helper the normal path
uses, in the SERVER_TASK_TYPE_TTS branch of launch_slot_with_task before
set_input. It goes into 0002 rather than a new patch file because it is
a defect in the code that patch introduces, and the header now records
it as ours so we know whether it still needs carrying if #26603 merges
without it.
Verified in one backend process, different text on every request:
three consecutive non-streaming requests, three consecutive streaming
requests, and an interleaved non-streaming, streaming, non-streaming,
streaming run. All ten returned HTTP 200 with
RIFF ... WAVE audio, Microsoft PCM, 16 bit, mono 24000 Hz, the streamed
ones carrying exactly one RIFF header at byte 0, and every output
measured as real speech rather than silence or a truncated fragment.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(llama-cpp): expose max_frames for TTS requests
The Qwen3-TTS backbone does not always emit <|codec_eos_token|>, and
when it does not, generation runs to upstream's 512-frame n_predict
default. At the model's 12.5 Hz frame rate that is 40.96 s of audio,
which a short input can trigger: one request in this session produced
40.96 s for a ten-word sentence. prepareTTSTask hardcoded n_predict to
-1, so callers had no way to bound it.
Add a max_frames key alongside top_k and top_p, parsed with the same
strict whole-string parsing so a typo is an error rather than a silently
truncated value, and rejected with a field-naming message when negative.
0 keeps the existing sentinel convention and means unset, so a request
that omits it behaves exactly as before.
Named max_frames rather than n_predict because frames are what the
parameter means at a TTS endpoint: one frame is 0.08 s of audio.
The 512-frame default is deliberately unchanged. Lowering it would
truncate legitimately long inputs, which is a worse failure than an
occasionally overlong one.
Verified end to end on one text of thirty words:
max_frames=25 HTTP 200, 96044 bytes, 2.00 s, exactly 25 frames
max_frames=50 HTTP 200, 192044 bytes, 4.00 s, exactly 50 frames
no max_frames HTTP 200, 572204 bytes, 11.92 s, stopped at its own
codec EOS after 149 frames, unchanged behavior
max_frames=-1 InvalidArgument "max_frames must be >= 0, got \"-1\""
max_frames=many InvalidArgument "max_frames must be an integer, got \"many\""
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(llama-cpp): send the TTS sample rate up front, and tidy three review items
Four items from the Task 4 review.
Streaming first-byte latency. TTSStream sent the sample-rate reply only
once the first audio result arrived, and a chunk needs a whole 72-frame
window, roughly 5.8 s of audio and far longer in wall time on CPU. The
Go side blocks on that reply before it can emit the WAV header, so a
streaming client sat at zero bytes for the whole stretch. The rate is a
property of the loaded model and is available synchronously from
mtmd_gen_audio_get_info, so it now goes out immediately after post_task
and the rate_sent bookkeeping is gone. Measured on a warm model, first
byte drops from 30.48 s to 0.014 s, and the output is still a valid WAV
with exactly one RIFF header at byte 0.
Unchecked close. The non-streaming path ignored ofstream::close(), so a
failure that only surfaces on flush was reported as success while
leaving a truncated file at dst. It now returns INTERNAL like the other
write failures.
Wrong comment on set_lang. gen_audio::inp::get() already maps a stored
blank to nullptr, so our guard is behavior-preserving, not
behavior-fixing. The comment claimed otherwise; the code was right.
Repetition penalty. penalty_last_n = -1 is inert at this pin, because
llama_sampler_init_penalties clamps it with std::max(penalty_last_n, 0)
and then builds a disabled sampler, so the 1.05 penalty never applies.
Upstream's README attributes looping to a missing repeat_penalty, so it
was worth testing as a root-cause fix for the model running to the frame
cap. Dropping the line lets the sampling default of 64 apply, which was
confirmed in the sampler chain trace as penalty_last_n = 64 with
repeat_penalty = 1.050. Over 15 uncapped short requests each way it did
not help: 0 of 15 ran to the cap with the penalty inert, 1 of 15 with it
active. Both lines are therefore kept for parity with upstream's draft,
and a comment now records that the pair is inert and why, so the next
reader does not believe a penalty is applied. max_frames remains the way
to bound output.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* build(llama-cpp): let unpatched forks opt out of the TTS task
turboquant and bonsai copy grpc-server.cpp into llama.cpp forks that do
not carry our patches. disable-tts-task.sh injects the same kind of
preprocessor switch disable-score-task.sh already uses, so those builds
answer UNIMPLEMENTED rather than failing to compile.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(config): keep a TTS speaker-encoder projector out of vision detection
Task 1 exempted a declared-TTS model's mmproj from VisionSupported, but the
first real gallery entry with an mmproj still came back vision-capable through
two paths the earlier fix did not close.
GuessUsecases has no FLAG_VISION branch, so it falls through to true for any
chat-ish model. That is not just a wrong answer at the call site:
syncKnownUsecasesFromString rewrites KnownUsecaseStrings from HasUsecases, and
the loader calls it more than once per config file, so the guessed FLAG_VISION
is written out and parsed back into KnownUsecases as if the operator had
declared it. Give GuessUsecases a FLAG_VISION branch that defers to the same
explicit signals VisionSupported uses.
Second, llama.cpp builds an mtmd context for the speaker-encoder projector and
reports its media marker on the first chat probe, which resurrected vision
after the model had been used once. Apply the same declared-TTS exemption to
MediaMarker that the mmproj check already had.
Verified against the qwen3-tts-llamacpp-q4 gallery entry: no vision capability
and no image input modality, before load, after a TTS request, and after a chat
probe.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): add Qwen3-TTS entries for the llama-cpp backend
Two entries over upstream's own GGUF conversion, Q8_0 and Q4_K_M, each
pairing a backbone with the Q8_0 projector. Named to sit alongside the
existing qwen3-tts-cpp entries rather than replace them.
Also tags the llama-cpp backend text-to-speech / TTS so the backend browser
surfaces the capability.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: cover Qwen3-TTS on the llama-cpp backend
Adds the gallery variants, the two-file mmproj configuration, the
required voice reference, and the language and sampling knobs. Also
corrects the streaming-support list, which named only voxcpm.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(config): register llama-cpp as a TTS and voice-cloning backend
The branch taught the llama-cpp backend to serve Qwen3-TTS and shipped two
gallery entries for it, but never told the capability table. llama-cpp still
declared only the text RPCs and usecases, so:
- VoiceCloningForModel returned nil at the capability check, before it ever
reached the model's own tts.voice_cloning override, and /tts answered 400
"selected model does not support reference-audio voice cloning" for any
localai://voice-profiles/... voice. No model YAML could opt back in.
- GET /api/backends/usecases did not list tts for llama-cpp, so the gallery
greyed out the TTS filter for the entries this branch adds.
- The React TTS page saw voice_cloning: null and kept both models out of the
Voice Library.
Add the TTS RPCs and usecase, and the reference-audio contract.
The contract needs narrowing, because the per-backend switch in
VoiceCloningForModel ends in a permissive default: an unnarrowed entry would
have advertised reference-audio cloning on every GGUF chat model in the
gallery. Narrow on the declared TTS usecase rather than the model name. The
TTS checkpoints are the only llama-cpp models carrying known_usecases: [tts];
name matching would have to guess at third-party repacks, and "base", the
substring the neighbouring Qwen and vLLM cases key on, is a routine word in
text-model names. The check reads the declared bit directly instead of going
through HasUsecases, which falls through to GuessUsecases and would hand the
decision to a heuristic that never had a llama.cpp TTS model in mind.
DefaultUsecases stays [chat]: a bare GGUF served by llama.cpp is a chat model,
and both the gallery filter and the importer read that field.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): declare what nemotron-3-nano-omni actually accepts
The entry is backend: vllm-omni with known_usecases: [chat, completion], no
mmproj and no media marker, so it used to report vision only through the
blanket GuessUsecases fallthrough that the vision branch in this branch
removed. Nemotron 3 Nano Omni is a multimodal understanding model: image,
video and audio in, text out. Declaring that is what the sibling
vllm-omni-qwen3-omni-30b already does.
known_usecases gains vision only. FLAG_VIDEO is video GENERATION, an output
modality, and this model generates none; video and audio input belong in
known_input_modalities, which is where AudioInputSupported and
VideoInputSupported read them from.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(importers): import a Qwen3-TTS GGUF repo as TTS, not chat
The llama-cpp importer hardcodes known_usecases: [chat] and assigns any
mmproj-matching file as a vision projector, so ggml-org/Qwen3-TTS-12Hz-1.7B-
Base-GGUF imported as a chat model with vision. Both fields were wrong, and
the model was unreachable from /tts and from the Voice Library.
Filenames cannot fix this. A Qwen3-TTS repo has the exact shape of a vision
repo, one backbone GGUF plus one mmproj-*.gguf, so the projector's own header
is the only honest signal: mtmd writes clip.has_gen_audio_encoder for the
projectors it can drive as a speech pipeline and refuses to build one without
it. Probe the selected mmproj for that flag, reusing the range-fetch the MTP
detection already does, and declare tts when it is set. The mmproj assignment
then stops reading as vision on its own, since a declared-TTS model already
exempts its projector from vision detection.
The probe is best-effort like the MTP one: a network blip leaves the chat
default in place rather than failing the import.
Verified against the real artifacts on disk: the Qwen3-TTS projector reports
gen-audio, its backbone does not.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(llama-cpp): stop non-TTS models crashing on the new pin
Two regressions, both hit every ordinary llama-cpp model and neither was
caught locally because every test on this branch loaded a TTS model.
The first is a null dereference. server_slot::tts_ctx::reset() called
mtmd_helper_gen_audio_reset() unconditionally, but the gen-audio pipeline
is only allocated for models carrying a gen-audio mmproj, and upstream's
implementation reads ctx->pipeline before null-checking anything. Since
server_slot::reset() runs during slot initialization for every model, any
non-TTS model segfaulted the backend the moment it loaded. Guard the call
on the is_supported() predicate already defined beside it, and keep the
plain field resets unconditional.
The second is unrelated to TTS and came in with the pin bump.
PredictOptions.Penalty is a bare proto float, so a caller that names no
repetition penalty sends 0 rather than omitting the field. Since
9de0fcf2b, common_sampler_init() rejects a non-positive penalty_repeat
outright because it would divide logits by zero, turning every such
request into "Failed to initialize samplers". Treat 0 as unset and leave
llama.cpp's own neutral default in place.
Verified with the same suite CI runs, which is what caught both:
tests/e2e-backends passes 6 of 6 including the load and predict specs
that were red. Qwen3-TTS still synthesises on both paths, 24 kHz mono
16-bit WAV with exactly one RIFF header on the streamed output.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm): align Intel basekit runtime
The latest vLLM XPU requirements install oneAPI 2026 runtime packages. The 2025.3.0 base image ships an older libsycl/UR loader pair and fails while importing torch with an undefined urDeviceWaitExp symbol. Use the current repository-wide 2025.3.2 Intel basekit patch level, which carries the compatible loader.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm): pin Intel source build to release
Build the Intel XPU backend from vLLM 0.26.0 instead of the moving main branch, and use the Triton XPU version required by that release's torch 2.12 dependency.
Assisted-by: Codex:gpt-5 [systematic-debugging]
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add Q4_K_M and Q8_0 llama.cpp builds of the compact-reasoning Gemma 4 fine-tune, including its multimodal projector.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Wrap PRELOAD_MODELS decoding failures with the setting name and expected top-level shape so startup errors point directly to the invalid configuration. Document the required array format and cover scalar and empty-array inputs.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(pii): restore request-scoped pseudonyms
Replace masked request values with unique per-request tokens when response restoration is enabled, then restore them across JSON and SSE write boundaries. Document the opt-in model setting and expose it in config metadata.\n\nAssisted-by: Codex:gpt-5
* fix(pii): wrap reversible redaction tokens
Use configurable token delimiters to avoid restoring ordinary model text that happens to match an internal identifier. Rename the option and document the confidentiality tradeoff.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(vllm-cpp): serve MiniMax-H3 video+audio generation
vllm.cpp's C ABI grew a video slice (ABI v12): a second engine handle
loaded from the MiniMax-H3 checkpoint SET, one blocking generate, and a
composed ffmpeg argv the caller execs. This wires that into LocalAI's
existing /video endpoint, so `vllm-cpp` now serves both text and video
and a clip comes back as an MP4 with a real audio track rather than a
silent render.
The video engine is a separate handle rather than a mode of the text
one because H3 is not a model directory: the DiT, the text encoder and
two VAEs are separate artifacts, and vllm.cpp has the two loaders refuse
each other's checkpoints. `Load` takes the video branch when the config
declares any of the video options; `parameters.model` is the DiT and the
rest of the set is named in `options:`.
Three details are worth calling out because getting them wrong is
expensive:
- The partition is DECLARED, not detected. The community quantisations
strip the release metadata and the FL2VA and Ref2VA DiTs are
byte-structurally identical, so the engine refuses to generate until
it is told which it has. Worse, a mismatch does not fail cleanly: a
reference passed to an FL2VA DiT renders for hours and returns a
coloured lattice over the frame. The backend refuses that combination
up front instead.
- ffmpeg comes from the host. libvllm writes frames plus a WAV and
composes the mux argv, then spawns nothing - that process boundary is
upstream's decision. The backend execs it, the same arrangement
vibevoice-cpp uses for transcoding, and ffmpeg also converts a
start_image upload into the binary PPM at the exact output canvas the
engine requires.
- It is slow. Roughly 176 s per denoise step at the default 1344x768
canvas on a 20-SM device, so the 50-step default is a multi-hour job.
Nothing on this path imposes a deadline.
The /video endpoint no longer forces 512x512 when the request omits the
geometry. Every video backend already supplies its own default for a
zero (512x512 for stablediffusion-ggml, 1280x720 for diffusers, 832x480
for longcat-video, 1344x768 for H3), so the hardcoded value only ever
overrode the model's trained canvas with one three of the four were
never trained at.
Moving the engine pin from ABI v10 to v16 also grows the text
vllm_model_params mirror by the v14 device field and the v16 KV-sizing
knobs. LocalAI sets none of them - 0 is the pre-v14 engine byte for byte
- but the struct SIZE is part of the layout contract, so leaving them
out would have vllm_engine_load read past the allocation.
Gallery: `minimax-h3-fl2va-q4` installs the Q4_K_M FL2VA set (~40 GB
across five weight files plus the two VAE configs that carry the latent
statistics).
Assisted-by: Claude:claude-opus-5 golangci-lint yamllint go-vet
* fix(vllm-cpp): unbreak the Darwin build at the new engine pin
src/capi/vllm_c.cpp opens one `extern "C" {` for the whole ABI surface,
so file-local helpers declared inside it inherit C linkage. The video
slice added one that returns std::string, which Apple Clang reports as
-Wreturn-type-c-linkage and vllm.cpp's target-local -Werror turns into a
build failure. GCC and upstream Clang do not diagnose it, so only the
metal-darwin-arm64 job saw it.
Suppress it the same way this Makefile already suppresses Apple Clang's
-Wgnu-folding-constant on the Metal build. The helper is never called
across the boundary so the warning describes no hazard here, but it is a
real upstream wart: the fix belongs in vllm.cpp, hoisting the helper
above the extern "C" block, and this flag should go when a pin carrying
that fix lands.
Assisted-by: Claude:claude-opus-5
* fix(vllm-cpp): patch the engine clone instead of the warning flag
The -Wno-return-type-c-linkage added in the previous commit does nothing.
vllm_cpp_set_warnings adds `-Wall -Wextra -Werror` as PRIVATE target
options, so they land after anything CMAKE_CXX_FLAGS contributes, and
-Wall re-enables the -Wreturn-type group that -Wreturn-type-c-linkage
belongs to. The darwin job failed again on the same line, which is the
evidence: a consumer cannot wave this off from outside the engine.
Position is the only fix, so carry it as a patch against the pinned SHA,
the way longcat-video patches its own upstream. It hoists the helper
above the `extern "C" {` that gives it C linkage; it is file-local and
never called across the boundary, so nothing else moves.
`git apply` is unguarded on purpose: a patch that stops applying must
fail the clone loudly, because the alternative is a pin that silently
ships without a fix it is documented to carry. The patch header names
what retires it - a pin carrying the fix upstream, where it belongs.
Verified by applying the patch with `git apply` to the exact blob at the
pinned SHA and diffing the result against the intended file.
Assisted-by: Claude:claude-opus-5
* chore(vllm-cpp): bump the engine pin to ABI v17 and drop the vendored OrEmpty patch
The OrEmpty linkage fix this backend carried as patches/0001-* landed upstream
(mudler/vllm.cpp#195, 7534da65), so the patch has done its job. It is deleted
rather than left in place: the Makefile applies patches/*.patch unguarded and
documents that "a patch that no longer applies must FAIL the clone", so keeping
it against fixed source would break the build the moment the pin moved. Bumping
the pin and deleting the patch therefore have to be the SAME change.
Pin f921062b -> 776c56f1 (current vllm.cpp main).
That range also carries the engine's ABI v17 (vllm_server_main: the OpenAI server
published on the public surface). registerLib compares the library's
vllm_abi_version against `abiVersion` for EXACT equality, so the constant moves
16 -> 17 in the same commit or every load fails with an ABI mismatch.
The bump is safe for the layout assertions in video_test.go: diffing include/vllm.h
across the two pins shows zero struct-field changes -- v17 adds one function
declaration, the version macro and a doc comment, nothing else -- so every
unsafe.Offsetof in the video params test still holds.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
* chore(vllm-cpp): re-pin to pick up the VLLM_CPP_SERVER=OFF link fix
The previous pin carried vllm.cpp's ABI v17 (vllm_server_main) but not the guard
that makes it link when the server is compiled out. This backend builds libvllm
with VLLM_CPP_SERVER off, so the darwin lane failed at the dylib link with
vllm::entrypoints::openai::VllmServerMain undefined.
Fixed upstream in mudler/vllm.cpp#202: the C entry point is now guarded, so the
symbol is still exported (ABI v17 stays resolvable for dlopen) while the
no-server arm reports the missing capability instead of dragging in a translation
unit that was never compiled.
Verified upstream in BOTH arms before re-pinning: SERVER=ON builds and runs, and
SERVER=OFF configures, links, produces libvllm.so, and `nm -D` shows
vllm_server_main exported next to vllm_video_generate and vllm_transcribe.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Adds a complete Brazilian Portuguese (pt-BR) translation for the
LocalAI WebUI across 14 namespaces with full key parity against the
English locale, including modelEditor.json. Registers pt-BR in
SUPPORTED_LANGUAGES with the code 'pt-BR', name 'Português (Brasil)'
and flag 'BR'. Brand/model/product names and technical identifiers are
kept untranslated, matching the existing locale conventions.
Assisted-by: opencode:deepseek-v4-flash-free python3
Signed-off-by: Matheus C. França <matheus-catarino@hotmail.com>
* feat(modelartifacts): support bounded parallel Hugging Face file downloads
Closes#11114.
Snapshot materialization fetched every file through the sequential
executor in DownloadFilesWithContext, so a repository split into many
shards spent most of its wall clock in per-file request latency rather
than moving bytes.
Add DownloadFilesWithConcurrency, an errgroup with SetLimit, and keep
DownloadFilesWithContext as a wrapper that passes a limit of 1. That
leaves the two non-artifact callers (core/gallery and the model config
loader) on exactly the path they had: tasks still run in slice order,
and the first failure still returns before any later task starts.
Only whole files run in parallel. A single file is never split, so the
.partial resume machinery and the per-file SHA check in
downloadTaskWithRetry are untouched.
Two details the parallel path forced:
- completedBytes becomes an atomic.Int64. Several AfterDownload hooks
add to it while other files' progress callbacks read it; without this
the race detector reports three races on the new specs.
- The caller's status callback is serialized. The sequential path gave
it an implicit guarantee of never being entered twice at once, and it
belongs to the caller, so the executor keeps that promise rather than
pushing locking onto every caller. AfterDownload is deliberately not
serialized -- it does the verify-and-promote work that parallelism
exists to overlap.
Manifest order needed no work: each hook already writes its own
manifest.Files slot by snapshot index, so entries stay in snapshot
order whatever the completion order. A spec now pins that.
The default is 1, unchanged behaviour. A shared models volume is often
the bottleneck rather than the link, so raising it is a deployment
decision; --artifact-download-concurrency and
LOCALAI_ARTIFACT_DOWNLOAD_CONCURRENCY expose it on both `run` and
`models install`.
Not done here, per the issue: no chunk-level parallelism within a single
file, and no throughput measurements across concurrency 1/2/4/8 -- that
needs a representative sharded repo and a real link.
Assisted-by: Claude:claude-opus-5 go-test gofmt
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
* feat(modelartifacts): expose download concurrency in settings
Follow-up to review feedback on #11162:
- The CLI flag and docs no longer describe the limit as Hugging Face
specific. It applies to any artifact source, as @mudler pointed out.
- artifact_download_concurrency is now a persisted runtime setting and
is editable from the WebUI, so it can be changed without a restart.
The manager's limit becomes an atomic.Int64 behind
SetDownloadConcurrency, because a live runtime setting can be updated
while a materialization is already in flight. Injected materializers
stay compatible through an optional setter interface, so a manager that
does not implement it is simply left alone.
Verified before taking this on: go build, go vet and go test -race all
pass for pkg/modelartifacts, pkg/downloader and core/config. The React
UI builds with vite, artifact_download_concurrency is present in the
built Settings chunk, and eslint reports the same 8 pre-existing
warnings on Settings.jsx as it does without the change.
Implementation contributed by localai-org-maint-bot on the review
thread; reviewed, verified and signed off by me.
Assisted-by: Codex:gpt-5
Assisted-by: Claude:claude-opus-5 go-test vite eslint
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
---------
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
Cosign v2.4.1 emits the current Sigstore bundle format by default and rejects --new-bundle-format. Keep recursive OCI 1.1 referrer signing, and update the producer contract test to reject the unsupported option.
Assisted-by: Codex:gpt-5
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): scaffold the backend and upstream build
Adds the backend skeleton and the NeMo-Speech.cpp build, pinned at
2e12e2def8a98ed06666f7ee3ca94e7193e04be4. The Go side is deliberately a stub:
it dlopens the runtime and starts the gRPC server, later work fills in the
symbol table and the model logic.
Three details of the upstream layout differ from what the plan assumed, and the
build reflects the real tree:
* The TTS C ABI ships as libnemo_speech_tts, not libnemo_speech_tts_c. Upstream
compiles c_api.cpp straight into the implementation library and only aliases
the nemo_speech_tts_c CMake target, so no _c object exists on disk. ASR and
NMT do build a real _c shim.
* Shared objects land in build/bin, since upstream points
CMAKE_LIBRARY_OUTPUT_DIRECTORY at ${CMAKE_BINARY_DIR}/bin.
* The ASR and NMT _c shims carry a DT_NEEDED on libnemo_speech_asr and
libnemo_speech_nmt, so those are staged and packaged alongside them.
Otherwise dlopen fails at startup.
The ggml patch step uses an order-only prerequisite. cmake writes into the
checkout and bumps its mtime past the sentinel, which would otherwise re-run
git apply over an already-patched tree and break every incremental build.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): make 'build' produce the package and bundle the ITN stack
Addresses the review of the scaffold commit.
backend/Dockerfile.golang runs 'make -C backend/go/$(BACKEND) build' and then
copies package/ into the final image, so 'build' has to end with a populated
package/. It only staged shared objects, which would have shipped an image with
no binary and no libraries at all. The old staging recipe is now stage-libs and
the chain is stage-libs, nemo-speech-cpp-grpc, package, build, matching every
sibling Go backend.
Text normalization was packaged incorrectly. nemo_speech_text_normalization is
STATIC but links sparrowhawk, fstfar and fst PUBLIC, so they land as DT_NEEDED
on libnemo_speech_asr.so, and they live in a project-local prefix that nothing
else provides. WITH_NORM stays ON by default on Linux, since normalization is a
wanted feature. Instead stage_libs now copies .deps/itn/lib when WITH_NORM=ON,
and package.sh bundles it.
Staging that prefix is still not enough on its own: Sparrowhawk drags in
protobuf, re2 and absl, which neither build_itn_deps.sh nor
package-system-libs.sh provides. Rather than hard-code another hand-maintained
list, package.sh now walks the DT_NEEDED entries of everything staged and copies
whatever is unresolved, skipping the core set and the GPU set that the shared
scripts already own. It fails at package time, not at first dlopen, when
something cannot be resolved. On a WITH_NORM=OFF build the closure is already
complete and it copies nothing.
Restore CGO_ENABLED=0 on the Go build to match whisper, parakeet-cpp and
omnivoice-cpp. Note that purego reaches dlopen through fakecgo, so the binary is
dynamically linked either way; what the flag changes is the NEEDED set, and
lib/ld.so routing in run.sh exists precisely because the binary is not static.
Replace the hand-rolled .patched sentinel with upstream's
scripts/apply-ggml-patches.sh. It applies the series in filename order, exits
non-zero when a patch does not apply, and detects "already applied" by comparing
the full-series tree hash rather than an mtime, so it is safe to run every time
and there is no sentinel left to go stale or to wedge the build when deleted. It
is wired as an order-only prerequisite so running it does not force a relink.
Also: correct the package.sh header, which claimed three shared objects when
there are five and none of the TTS ones carry a _c suffix; give 'make test' the
LD_LIBRARY_PATH the dlopen tests will need; document that a NEMO_SPEECH_VERSION
bump needs 'make purge'; and extend 'clean' to remove package/ and the ITN
libraries.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): make the closure guard fail closed and give CI its toolchain
Addresses the second review round.
The dependency-closure guard failed open. Its glob expands once per pass, so
each pass advanced the closure by exactly one level, and the fixed count of five
passes then fell out of the loop without checking whether anything remained. An
eight-deep chain packaged six libraries, exited zero and reported success. That
is the case the guard was written for: asr to sparrowhawk to protobuf to absl
already runs several levels deep, so a WITH_NORM build could ship missing its
deepest libraries and fail at first dlopen. The loop now runs until the staged
set stops growing, and exhausting the bound is a hard error rather than a silent
exit.
For the same reason, a build image with neither readelf nor objdump no longer
warns and skips. It cannot show the package is complete, so it refuses to ship
it. The guard is entered only when there is something to check, so an empty
package cannot trip the new error.
Dockerfile.golang installed ninja-build only in the Vulkan branch while this
Makefile runs cmake -G Ninja unconditionally, so the CPU, cuBLAS and L4T images
could not configure at all. ninja-build moves to the common apt list; it does
not change CMake's default generator, so it is inert for the other backends.
gcc-12 was nowhere in the tree, yet WITH_NORM defaults ON and
build_itn_deps.sh needs it, so the committed default was unbuildable in CI.
Install it, with the protobuf, absl, re2 and autotools that Sparrowhawk and
OpenFST need, gated on BACKEND so the other Go images do not carry it. The list
follows upstream's own docker/Dockerfile, trimmed of the gRPC, portaudio and
python entries a BUILD_GRPC=OFF build does not use. Text normalization stays ON:
downgrading it silently would ship a backend advertising a feature it lacks.
Also: make test depend on stage-libs, so LD_LIBRARY_PATH is not an empty
directory on a clean tree, and add an engine target so Dockerfile.golang's
cacheable prebuild layer is not skipped and a CUDA build stops recompiling all
of upstream on every Go-side change.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): pin protoc for ITN and make the norm stack its own target
Addresses the third review round.
Dockerfile.golang installs protoc 27.1 into /usr/local/bin, ahead of /usr/bin,
while libprotobuf-dev is the distro's 3.21 on noble and 3.12 on jammy.
Sparrowhawk resolves protoc from PATH at make time (configure.ac uses
AC_CHECK_PROG, so PROTOC substitutes to the bare word, and src/proto/Makefile.am
invokes it) and commits no pregenerated stubs, so the rule always runs. Code
generated by 27.1 includes google/protobuf/runtime_version.h and a
PROTOBUF_VERSION guard the older headers lack, so the WITH_NORM build could not
complete. Pin PROTOC to the apt one for that step; configure documents that a
pre-set value wins. The apt protoc and libprotobuf-dev come from one source
package at one version, which is the property that makes this correct.
The text-normalization stack is now a target keyed on a file build_itn_deps.sh
actually produces, rather than a side effect of the runtime library rule. As a
side effect make could not see whether it existed, so once the library was up to
date the script could never run again: a tree built WITH_NORM=OFF could not move
to ON, and make test hard-failed with no escape but a full 345 MB clean. It is
now built on demand and reachable on its own as 'make itn'. Staging keys on the
prefix existing rather than on WITH_NORM, so it stages what the tree actually
built, and package.sh's closure guard remains the backstop.
An already-configured build tree also now wins over the platform default, so a
tree built WITH_NORM=OFF is not silently reconfigured to ON by a bare make test,
which is what demanded gcc-12 from developers who chose not to have it. An
explicit WITH_NORM= on the command line still overrides both, and the ITN rule
preflights for gcc-12 with an error that names the alternative.
Move ninja-build out of the shared apt layer into the existing BACKEND-gated
block. Dockerfile.golang serves 225 matrix entries and only this backend
configures with -G Ninja, so the common list is byte-identical to master again
and no other image loses its cache.
Drop libabsl-dev and correct the comment that justified it. No base image here
ships protobuf 25, so nothing needs the absl split, and the cmake glob looks in
/usr/lib rather than the multiarch directory Ubuntu actually uses, so the
package could never have contributed anything.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): move the backend apt gate below the expensive layers
Addresses the fourth review round.
The nemo-speech-cpp apt block sat immediately after the shared apt layer, above
the Vulkan SDK build, the CUDA and ROCm installs, the Go toolchain and the
protoc download. Docker keys each layer on its parent, so inserting a step there
re-keys everything below it: a byte-identical shared layer is not enough, and
merging as it stood would have forced all of those to re-execute once for every
Go backend image. Move it down beside the existing opus, crispasr and
sherpa-onnx gates, which sit after those layers for the same reason.
Checked the ordering both ways before moving. Nothing between the two positions
uses these packages: the Vulkan and opus blocks install their own ninja and
pkg-config, go install protoc-gen-go needs the Go toolchain rather than protoc,
and the protoc 27.1 step is a release-binary download that needs neither
protobuf-compiler nor libprotobuf-dev. Nothing in the block needs anything those
layers provide; it uses only apt, and the mirror rewrite from the first RUN
persists in the image. It also runs no update-alternatives, so the default
compiler stays untouched for later layers. The diff against master is now a
single additive hunk with no shared layer touched.
Also preflight ITN_PROTOC. configure gates a preset PROTOC on test -n alone, so
a path that does not exist is accepted and the error surfaces much later as a
bare "No such file or directory" from inside make -C src/proto. The pin
introduced that failure on a box whose only protoc is in /usr/local/bin, which
worked before. Check it alongside the gcc-12 check and name the ITN_PROTOC=
override in the message.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): parse model options
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): guard the empty option value and warn on a bad gpu index
An empty value like "vad_model:" must stay empty, since callers read the
empty string as "unset". That branch of resolve() had no spec: dropping the
guard left every spec green while parseOptions started returning the models
directory itself. Add the spec that fails without the guard.
A known key with an unparseable value is a typo, not a config from a newer
backend, and "gpu:banna" failed expensively: the model loaded, produced
correct output, and ran on CPU with no signal anywhere. Log it. Unknown keys
stay silently ignored, which is what keeps configs forward compatible.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): detect model family and discover TTS assets
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): bind the C ABI with layout assertions
purego binds by name at runtime and the config structs are passed by pointer,
so both a renamed symbol and a mismatched struct layout would otherwise survive
a green build. registerSymbols names the failing symbol, and the layout specs
compare each Go mirror against the size the library reports for itself, against
the offsets a C compiler produces for the installed headers, and against the
default values upstream writes into the structs it returns.
Two of the bindings differ from the plan because the headers do. The plan's
nemo_speech_diar_segments signature omits the segmentation-config pointer that
diar.h declares as the second parameter, which would have shifted the output
buffer, the capacity and the count pointer one position each. And
nemo_speech_diar_stream_push_f32 was missing from the symbol table although
standalone diarization cannot work without it.
Also close the two panic and equality gaps left in family.go: ValueString panics
on a mistyped general.architecture, and the self-codec guard compared a Cleaned
candidate path against an uncleaned one, so a doubled separator let the primary
GGUF be selected as its own codec.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(nemo-speech-cpp): run the ABI specs in CI and refuse to skip them
The layout assertions were inert. TEST_PATHS does not cover this backend and
the per-backend list in test-extra had no entry for it, so nothing invoked the
package's tests. Add it next to depth-anything-cpp, supertonic and vllm-cpp,
the group whose own test target carries its build prerequisites; stage-libs
already pulls the native build chain, so no prepare-test-extra entry is needed.
The skip guard was also loader-inconsistent: librariesPresent stats bare
filenames relative to the working directory while openLibraries resolves them
through the loader search path, so any invocation other than make test skipped
every library-backed spec and still reported green. NEMO_SPEECH_REQUIRE_LIBS=1
turns that into a failure naming the directory and the remedy, and the Makefile
test target sets it. Unset, the plain skip survives so a developer without a
build can still run the pure-Go layer specs.
Trim the default-value fingerprint from roughly forty assertions to eight. It
was pinning tunables such as threads and flush_partial_chunk, so a legitimate
pin bump would have failed with a message reading like a layout error. What
survives is only header-documented contract: the lone non-zero max_alternatives,
the run of -1 sentinels and the zero that witnesses where it stops. Verified the
narrowed spec still catches a mirror and offset table corrupted in lockstep,
which is the one class only this layer sees.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): select the family at load and gate RPCs on it
Load sniffs the GGUF architecture, maps it to a family and dispatches to
the family's loader. requireFamily gates every other RPC, returning
Unimplemented naming both the loaded and the wanted family so a
misconfigured model YAML produces a message a user can act on.
The family is committed only once its loader has succeeded. A load that
fails part way through would otherwise leave the gate open on a handle
that was never created.
cstr uses runtime.Pinner rather than an ordinary Go allocation. The
address crosses the ABI as a uintptr, which the collector does not
trace, so incidental reachability through the release closure is not a
guarantee: a caller discarding that closure could have the bytes
collected before the create call reads them. Pinning is the sanctioned
mechanism, makes the release function do real work, and turns a dropped
release into a loud leaked-Pinner panic instead of silent corruption.
Free overrides the base no-op to destroy the handle and reset the
family. Every family owns C memory only its own destroy entry point can
release, so without this an unloaded model leaks an acoustic model.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): pin the load ordering, close the engineMu race
Three review items, plus a defect the race detector turned up.
The spec covering "no family selected after a failed load" wrote junk to
a .gguf, so Load returned at ggufArchitecture before a family was ever
chosen and the assertion was vacuous. Generalised the GGUF test helper
to take a string architecture, and added a spec that loads a magpietts
GGUF with no sibling codec, so familyFor succeeds and discoverTTSAssets
then fails. It self-guards on ggufArchitecture so it cannot degrade back
into the earlier path.
requireFamily read n.fam unlocked while Free wrote it under engineMu,
which the race detector confirms is a real race. pkg/grpc/server.go
calls Free without the backend lock every other RPC holds, so teardown
can land mid-request. withEngine now takes the lock, checks the family
and runs the body under one acquisition; two would leave a window for
Free to destroy the handle between check and use. The locking protocol
is stated in both directions for the RPCs still to be written.
Running -race also enables checkptr, which aborts on cstr's pointer
being read back by goString: converting a uintptr to a pointer is fatal
whenever the address lands in a Go allocation, so a pinned Go buffer can
never be dereferenced from Go. The pointer is for C alone. Both helpers
now document the one-way contract, and goString is tested against a real
C-owned string by rebinding the version symbol to return a raw char*.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): implement offline transcription
Create the ASR recognizer in loadASR and serve AudioTranscription.
Segment times are int64 nanoseconds, not seconds: the proto field is an
int64 that core/backend reads straight into a time.Duration, while the
runtime reports word offsets in milliseconds. Words are grouped into one
segment per consecutive speaker run, with the 1-based speaker tag carried
through and 0 (untagged) left unlabelled.
The whole RPC body runs inside withEngine so the family check and the C
calls happen under one acquisition of engineMu. Free runs without the
backend lock, so checking the family and then relocking would let a
teardown destroy the handle in the gap. The audio decode is inside the
closure too, which costs nothing: base.SingleThread already serialises
this backend's RPCs.
recognizeF32 guards zero-length PCM. &pcm[0] panics on an empty slice, so
Go never reaches the C side's own "empty audio" rejection, and a silent
clip or a truncated upload is ordinary input.
pkg/utils has no WAV decode helper, only the ffmpeg normalisation, so
audio.go pairs AudioToWav with go-audio the way parakeet-cpp does. It
returns the sample rate rather than a duration, since the C API resamples
off that number.
Also closes the write-side half of the race Task 5 fixed on the read
side: Load now holds engineMu across the family switch and the n.fam
commit, matching Free. The loaders still must not take it.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): implement streaming and live transcription
AudioTranscriptionStream drives a whole clip through the cache-aware
streaming API in 100 ms pushes, emitting each finalized utterance as a
delta and closing with the assembled result. AudioTranscriptionLive
serves the bidirectional RPC over the same session: config first, a ready
ack, deltas with word timings as utterances land, and a terminal result
when the caller closes its send side.
Both wrap their body in withEngine, so a stream holds engineMu for its
whole life and Free waits on it rather than destroying the recognizer
underneath a half-finished stream. That makes the way out load-bearing:
the file loop honours the request context between pushes, and the live
loop ends when the host closes the request channel, so a disconnected
client cannot pin the model against unload.
Only finals become deltas. The runtime applies punctuation and inverse
text normalization on finals only, so a final rewrites the utterance
rather than extending its interim, and delta on the wire is
newly-finalized text that consumers concatenate. Forwarding interims
would duplicate and mispunctuate every utterance.
The four streaming entry points sit behind an asrSession interface. No
NeMo GGUF is small enough to keep in the tree, so without that seam the
need-more-audio drain would have no test at all: nemo_speech_asr_stream_next
reports OK with a NULL handle when it wants more audio, which is a pause
rather than an end, and reading it either way round drops results or
spins forever.
Also folds in three items from the offline transcription review:
- empty audio is now refused before anything crosses the ABI, not
inside recognizeF32. The added integration spec caught the old
ordering panicking on an unbound entry point instead of failing;
- an undecodable sample rate is an error rather than 0, which this
runtime reads as "already at the model rate" and would have made a
wrong rate silently pitch-shift the audio;
- AudioTranscription guards its result pointer instead of relying on
an unstated invariant.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): make live deltas concatenate and fill segment words
runLive wrote the inter-utterance separator into the accumulated
transcript but emitted the delta without it, so a two-utterance turn sent
"one." and "two." while the terminal result read "one. two.". The live
consumer is the one that really concatenates: the realtime semantic-VAD
path joins the accumulated deltas with the empty string and clears them
only at a turn reset, never at an endpoint, so the running caption read
"one.two.". The separator now goes into the delta, as it already did on
the file path, and the terminal text is the verbatim concatenation rather
than a trimmed rebuild.
TranscriptSegment.Words was never populated, so a request asking for
timestamp_granularities ["word"] came back with no words at all even
though the timings were decoded. wordsToSegments now attaches them,
gated on the granularity the same way parakeet-cpp gates it, so a
transcript that did not ask for word timestamps does not pay for them.
Also: the final that comes back from the tail flush no longer claims an
end-of-utterance. It is the end of the stream, not a user yielding the
turn, and eou is what the realtime turn detector acts on.
The comment explaining why interims are suppressed led with the runtime's
postprocessing. The wire contract is the stronger reason and now comes
first: consumers concatenate deltas, so forwarding a growing hypothesis
assembles to "hehellhelloHello.". The postprocessing only explains why no
diffing trick would rescue them. It is also ITN and strip_formatting
rather than punctuation, which is off by default here.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): implement standalone diarization
loadDiarizer creates the Sortformer diarizer and Diarize serves the RPC
over a diarization stream: decode, chunked push, finish, then the
count-then-fill segments protocol.
nemo_speech_diar_segment carries start_time and end_time in SECONDS
already, not frame indices, so no conversion happens on the way to
DiarizeSegment.start/end and the model's seconds-per-frame is not
involved at all. The speaker label is the runtime's 1-based tag as a
decimal string, matching what wordsToSegments emits on the ASR path, so
the same speaker reads the same way whether a caller diarized a file or
transcribed it.
The six frame-geometry overrides are written as -1 rather than left
zero. c_api.cpp applies left_context_frames when it is >= 0 while every
other override needs > 0, so a zeroed config would silently pin the left
context to zero and change the model's streaming geometry.
nemo_speech_diar_segments writes *count before it rejects a buffer that
is too small, so a rejected fill still reports the size to retry with.
collectSegments uses that rather than truncating, bounded at four
attempts because the RPC holds engineMu for its whole body and an
unbounded retry would block an unload behind it.
Two DiarizeRequest knobs map onto the segmentation config, and the
proto and header names cross over: min_duration_on is the C
min_duration_sec and min_duration_off is the C min_gap_sec. Six fields
have no equivalent in this pipeline and are logged rather than dropped
in silence: num_speakers, min_speakers and max_speakers (Sortformer's
capacity is fixed by the checkpoint), clustering_threshold (there is no
clustering stage), include_text (no ASR here) and threads.
The empty-PCM guard fires before the stream is opened, so a silent clip
never reaches a purego entry point that would dereference &pcm[0].
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): pin the diarizer geometry sentinels and cap the segment buffer
The six frame-geometry overrides were written as -1 with nothing
asserting it. c_api.cpp applies left_context_frames at >= 0 while the
other five need > 0, so a dropped sentinel there pins the model's left
context to zero, and the struct keeps exactly the same shape, which is
all the layout assertions can see. Extracting diarModelConfig makes the
values assertable: five specs now pin all six frame fields, the device
index, the declared size and the NULL preset, each frame field on its
own line so a missing sentinel names itself.
distinctSpeakers had a spec with three segments over three distinct
labels, which len(segs) satisfies just as well as the real thing. Four
segments over three labels makes it a spec that can fail.
collectSegments sized its buffer straight from a count the C side
reported, and make() panics rather than erroring on a length it cannot
satisfy, so an uninitialised size_t coming back across the ABI killed
the backend process instead of failing one request. A ceiling of 2^22
segments, upwards of 93 hours of audio at one 80 ms frame each, turns
that into a diagnosable error.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): implement TTS and streaming TTS
The PCM callback is compiled once per process behind a sync.Once, not once per
request and not once per load. purego.NewCallback writes into a fixed table of
2000 entries (purego/syscall_sysv.go) and never releases one, so a per-request
callback panics the backend process on the 2001st synthesis, and a per-load one
reaches the same ceiling on a server that swaps models. Synthesis is routed
through that single callback plus a user_data id: engineMu is per-model, one
process holds several models, so a single current-sink pointer would be
overwritten by two TTS models synthesizing at once.
Deviations from the brief, all verified against the real headers and proto:
- TTS is TTS(*pb.TTSRequest) error and TTSStream is
TTSStream(*pb.TTSRequest, chan []byte) error, per pkg/grpc/interface.go.
The brief's context/pb.Result and server-stream forms do not implement the
interface. The channel is closed on every path, including the family
rejection, because pkg/grpc/server.go blocks on its drain goroutine and an
unclosed channel hangs the RPC with the backend lock held.
- The callback takes unsafe.Pointer, not uintptr. Converting a uintptr
parameter back to a pointer is a checkptr violation that aborts under
-race.
- resolveSpeaker refuses to turn a negative number into a speaker index. -1
is the C API's "use the default" sentinel, so the brief's rule would have
made a request naming an invalid voice synthesize in the default voice
instead of being rejected.
temperature and cfg_scale each write their override flag as well:
magpietts/runtime.cpp reads the float only when the flag is set, so a
temperature without it is silently discarded.
Also folds in Task 8's review finding on asr.go: the six bare -1 sentinels in
loadASR move to an asrDiarConfig builder reusing diarGeometryDefault, with
specs. src/asr/c_api.cpp applies left_context_frames at >= 0, so a dropped
sentinel pins the model geometry to 0 and no layout assertion can see it.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): surface NMT translation through Predict
nemo_speech_nmt_translate takes explicit source and target languages and has no
free-form generation or token-callback entry point, so there is no prompt in the
LLM sense. The pair comes from the source_language / target_language model
options, with an optional leading [src->tgt] directive as the only per-request
override, and PredictStream emits the whole translation as a single chunk
because the C API has nothing finer to give it.
Both RPCs wrap their body in withEngine so the family check and the C calls that
trust the handle share one acquisition of engineMu. PredictStream closes its
channel on every path, including the family rejection: this is the legacy
streaming contract, and pkg/grpc/server.go blocks on a drain goroutine that only
finishes when the channel closes, so leaving it open hangs the RPC rather than
failing it.
nmtTranslatorConfig is extracted so its four adjacent pointer fields can be
asserted against distinct sentinels. Transposing two of them changes neither the
struct size nor any field offset, so the layout assertions cannot see it.
Also removes goString, which had no production caller: every string-returning
symbol in abi.go is bound with a Go string return that purego converts itself.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(nemo-speech-cpp): pin the three-segment pair tag in an NMT directive
The directive regex allowed an unbounded run of two-letter segments per side, but
nothing tested it: narrowing that run back to a single optional segment left every
spec green. resolve_tag accepts a ready pair tag in one field with the other empty
(src/nmt/langpairs.cc), and those tags run to three segments (en-zh-cn, pt-br-en),
so a shorter pattern does not mis-split the tag, it fails to match the directive at
all and the whole bracket is handed to the model as text to translate.
The justification on the regex was also wrong and is corrected: pt-br and zh-cn are
two segments and parse either way. It is the single-field form that needs the run.
Renames the NMT handle to n.nmt so it stops sharing a name with the translator
interface, following n.synth, which is shortened for the same reason.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): register the backend and give its specs a CI job
Registers nemo-speech-cpp across every surface .agents/adding-backends.md
requires, and adds the CI job its unit suite never had.
backend/index.yaml gets the meta backend (capabilities map, no uri), a
development meta and 12 image entries. No amd and no intel capability keys:
upstream NeMo-Speech.cpp builds ggml with CUDA, Vulkan or Metal only, and
SystemState.Capability falls back to "default", so those hosts get the CPU
build rather than a tag that does not exist. The nvidia-cuda-* and
nvidia-l4t-cuda-* keys are present because getSystemCapabilities() refines an
NVIDIA host to them whenever the CUDA directory exists; without them every
modern CUDA host and Jetson would miss the map and quietly run on CPU.
.github/backend-matrix.yml gets 7 include rows and 1 includeDarwin row. No
hipblas and no sycl rows, for the same upstream reason. cpu and vulkan are
per-arch pairs sharing a tag-suffix so backend-merge-jobs builds a multi-arch
manifest: an ARM host with no NVIDIA GPU reports "default" and the Jetson image
does not cover it.
The CI job is the substantive part. make test-extra is dead on master, because
prepare-test-extra depends on a protogen-python target that does not exist and
no workflow invokes it anyway, so the entry added earlier in this series ran
nowhere. abi_test.go asserts the size and field offsets of every Go mirror
struct against the C ABI it is dlopened into, and those assertions are the only
defence against silent memory corruption after a purego symbol rename or an
upstream header change. tests-nemo-speech-cpp in test-extra.yml now executes
them on pull_request and on master, gated on the backend's own path filter.
The recipe sets NEMO_SPEECH_REQUIRE_LIBS=1, so a missing library fails rather
than skips. WITH_NORM=OFF skips the OpenFST leg and costs no coverage: nothing
in the four C ABI headers is conditional on it, so the layouts are identical.
Also registers the upstream pin with the bump bot, which the backend Makefile
already claimed but was never wired up, and adds the BackendCapabilities entry
so a hand-written model config gets a real usecase surface. PossibleUsecases is
the union of the four families and DefaultUsecases is transcript alone, the
audio-cpp pattern. No VoiceCloning key: MagpieTTS synthesizes from baked
speaker ids, not a reference clip.
No gallery entries: publishing converted GGUFs is a follow-up.
ModelIdentity needs no work in this backend. main.go serves through
grpc.StartServer, so every RPC lands on pkg/grpc's shared server wrapper first,
and checkModelIdentity is the first statement of all seven handlers this
backend implements. A second check inside NemoSpeech would be unreachable and
would risk diverging from the cross-language sentinel the router matches on.
AudioTranscriptionLive stays unguarded because TranscriptLiveRequest carries no
ModelIdentity field at all, which is a proto-level gap affecting every backend
and needs its own change.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): build the CUDA-13 Jetson image the l4t-cuda-13 key needs
The nvidia-l4t-cuda-13 capability pointed at nvidia-l4t-arm64-nemo-speech-cpp,
which is built on nvcr.io/nvidia/l4t-jetpack:r36.4.0 and therefore links ggml
against CUDA 12. A Jetson whose CUDA 13 runtime is present reports that
capability and would have pulled an image with no libcudart.so.12 to dlopen,
failing hard at load. That is worse than omitting the key: with no key
Capability() falls back to "default" and the host gets a working CPU build.
Fixed the way parakeet-cpp and moss-transcribe-cpp already do it, by shipping
the second L4T image rather than dropping the key. Nothing prevents building it
here: those peers use plain ubuntu:24.04 on ubuntu-24.04-arm with the same
Dockerfile.golang as this backend's other rows, and every package in the
nemo-speech-cpp apt gate exists on noble arm64.
Adds the -nvidia-l4t-cuda-13-arm64-nemo-speech-cpp matrix row and its two index
entries, repoints the key on both metas, and rewrites the capability-map comment,
which had the reasoning backwards.
Also adds the documentary inferBackendPath branch, matching all six sibling
*-cpp Go backends. Behaviour is unchanged; the generic golang fallthrough
already resolved this backend correctly.
The previous commit message said "all seven handlers" of the shared gRPC
wrapper. There are eight RPC entry points: seven are guarded by
checkModelIdentity and AudioTranscriptionLive is the unguarded eighth, which
that message already called out separately. Wording only.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(nemo-speech-cpp): document the backend and list it in the importer
Adds docs/content/features/nemo-speech-cpp.md, alongside the audio.cpp page
that is its closest sibling, and cross-links it from the speech-to-text,
diarization, text-to-speech, backend-type and compatibility-table pages so the
backend is reachable from every surface that lists its modalities.
The page covers the architecture-to-family table, every option key with a model
YAML per family, the translation prefix directive, the acceleration matrix, and
the four limitations this backend ships with: Linux-only inverse text
normalization, suppressed interim streaming results, the library's default
translation context and generation limits, and the absence of gallery entries.
knownPrefOnlyBackends gains the backend so it appears in the /import-model
dropdown. It stays preference-only and AutoDetect=false: general.architecture
lives inside the GGUF where no remote-repo probe can read it, and a translation
model carries an ordinary LLM architecture with no NeMo-specific marker.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(nemo-speech-cpp): correct the translation limits, the macOS gap and the TTS conversion
Three factual errors found in review, all of them the kind a user would act on.
The translation limits were described backwards. Input longer than the 1024-token
context is rejected, not truncated: translator.cpp throws "nmt: prompt too long
(N tokens) for context 1024", which reaches the caller as a failed request. What
is silently cut is the output, by the max_new_tokens loop at 256. The bullet now
separates the two and says which one fails quietly.
The macOS gap covers TTS text normalization as well. Both directions sit behind
the single NEMO_SPEECH_WITH_NORM flag, which the Makefile forces off on Darwin,
so tn_dir is as inert there as itn_dir. Neither fails the load: both warn and
carry on. pnc_model really is unaffected, since punctuation is compiled in
unconditionally. The tn_dir row in the option reference gained the caveat the
itn_dir row already had.
The TTS conversion procedure produced a model that could not load. It converted
MagpieTTS and stopped, leaving no NanoCodec, which the same page lists as
required; following it gave "no NanoCodec GGUF found next to ...". Both halves
are now there, each with the download that feeds it, so the block runs top to
bottom on a clean machine.
Also: any negative gpu value pins TTS to the CPU, not only -1, and FLAG_CHAT
additionally surfaces the model in the web UI chat picker.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): map every C status, not just the NMT one
INVALID_ARGUMENT was translated to codes.InvalidArgument at exactly one of
sixteen C call sites. Everywhere else a non-zero status collapsed to
codes.Internal, so the same backend answered an unsupported language pair with
HTTP 400 and an unknown TTS voice, which is the same class of caller mistake
against the same process, with HTTP 500. Status 4 is CANCELLED on the ASR and
TTS surfaces and was reported as a backend failure rather than as the consumer
having stopped listening.
asr.h, tts.h and nmt.h each declare their own status enum and diar.h reuses the
ASR one; the values they share agree, and the single divergence is that NMT
declares no CANCELLED because nemo_speech_nmt_translate has no callback for a
consumer to stop with. That is an absence, not a disagreement, so one table
serves all three. status.go carries it, with the header line numbers and a note
that a pin bump has to recheck it: purego binds by name and the status crosses
as a bare int32, so nothing in the build or the linker can see a drift.
New specs cover the whole enum, unknown values, and one real INVALID_ARGUMENT
per family driven through the shared objects rather than through the Go mapping
asserting against itself.
Also add UsecaseChat to this backend's capability entry, which the docs already
told operators to set for translation models. chat is a gallery filter key and
completion is not, so GET /api/backends/usecases would have greyed the Chat
filter out and hidden a Riva-Translate gallery entry from the one filter that
fits it. The flag gates no endpoint; it makes the model eligible as the default
chat model and puts it in the web UI chat picker, both of which Predict and
PredictStream already serve.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): audit the gosec unsafe and file-inclusion sites
gosec flags 13 alerts on this backend: one G304 and twelve G103. Each was
checked individually rather than blanket-suppressed, and each annotation
states what makes that particular site safe.
The G304 at audio.go is a false positive. The opened path is
filepath.Join of a directory the function just created with os.MkdirTemp
and a constant basename; the request-controlled path is the input to
AudioToWav and never reaches the open.
The twelve G103 sites are the package's three established shapes, and
every one was verified against them: cstr and pinPtr take the address of
something pinned on the line above and return it one-way (nothing in the
package converts either result back, which is what keeps checkptr out of
it under -race), and each *Create hands C a stack-local POD config whose
uintptr members are cstr allocations or pinPtr addresses held by a pinner
the loader unpins only after the call. The two slice-building sites are
bounded by construction: DiarSegments is handed exactly len(buf) with the
buffer sized under maxDiarSegments and a reported count larger than it
rejected rather than sliced to, and the TTS callback copies out a slice
whose length is the length the runtime declared for that buffer.
Separately, sampleRateOf gets a real fix rather than an annotation.
go-audio reads the WAV header's sample rate from an unsigned 32-bit field
into an int, so a header claiming more than 2^31-1 passed the "> 0" test
and then narrowed to a NEGATIVE rate, which the runtime would take as a
resampling ratio. AudioToWav cannot produce one today, but that is a
property of another package and this function exists precisely because
the rate is read back rather than assumed, so the bound is enforced here
and pinned by a spec.
The four remaining integer narrowings are annotated with the bound that
makes each safe: the WAV payload length is already checked against
maxWAVDataBytes, the speaker count is bounded by maxDiarSegments, and the
two segment ids are the proto's own int32 wire type.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): skip the CUDA-only ggml patch series on darwin
The macOS backend build died in patch-ggml:
scripts/apply-ggml-patches.sh: line 56: mapfile: command not found
make[1]: *** [patch-ggml] Error 127
mapfile is a bash 4 builtin (and its -d flag needs 4.4). macOS ships bash
3.2.57 as /bin/bash and GitHub's runner images add no newer one, so the
bare `bash` the recipe resolves from PATH cannot run upstream's script.
Rather than hunt for a capable bash that the runner does not have, drop
the step where it does nothing. ggml-patches/ is a CUDA series: every
kernel it adds is under src/ggml-cuda/, and its whole footprint outside
that directory is an op enum plus prototype in include/ggml.h, the
constructor and a name-table entry in src/ggml.c, and two ggml-cpu lines
that make the CUDA-only op report unsupported and abort. Nothing it
touches is compiled into a Metal kernel or changes a CPU one.
The project's own references to patch-only ggml symbols sit behind
NEMO_SPEECH_FUSED_RELPOS_ATTN and NEMO_SPEECH_FASTCONFORMER_CUDA_FUSIONS,
which cmake already forces OFF without GGML_CUDA, or behind
NEMO_SPEECH_GGML_PATCHED itself, which guards a GGML_TENSOR_FLAG_Q8_PLANAR
write that a non-CUDA buffer throws before reaching. So passing
NEMO_SPEECH_GGML_PATCHED=OFF costs the Metal build nothing, and it is
required once the series is skipped: that flag is what stops the ASR
sources referencing a tensor flag stock ggml does not define.
This is upstream's own Metal configuration. Its metal-* and vulkan-*
CMake presets inherit the cpu-* ones, which set NEMO_SPEECH_GGML_PATCHED
to OFF; docker/Dockerfile and scripts/windows/build.ps1 do the same for
their non-CUDA targets. LocalAI's Makefile never passed the flag at all
and so inherited the CUDA default everywhere.
Linux is untouched and keeps applying the series, including its
idempotency and its hard failure on a patch that does not apply. The gate
is the same uname test the WITH_NORM block above already uses, and both
branches keep the order-only clone prerequisite, which on a WITH_NORM=OFF
tree is the only thing that pulls sources/ in.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): restore std::binary_function for MeCab on libc++
NEMO_SPEECH_TTS_WITH_JA=ON compiles Open JTalk's bundled MeCab, and
mecab/src/dictionary.cpp derives a comparator from std::binary_function,
which C++17 removed. libstdc++ still ships it as deprecated-but-present
under -std=gnu++17, so Linux never notices. libc++ compiles it out and
the macOS arm64 build dies with "no template named 'binary_function' in
namespace 'std'".
This is ours, not an upstream regression: upstream defaults both
NEMO_SPEECH_TTS_WITH_JA and NEMO_SPEECH_TTS_WITH_ZH to OFF and the OSS
drop carries no CI at all, so that target is never built there. Upstream
does already carry the equivalent workaround for MSVC's STL
(_HAS_AUTO_PTR_ETC plus /FIfunctional) but has no libc++ branch.
libc++ gates the two templates on
_LIBCPP_ENABLE_CXX17_REMOVED_UNARY_BINARY_FUNCTION, and has since LLVM
16, older than any clang Xcode still ships. The name is the whole
problem: _LIBCPP_ENABLE_CXX17_REMOVED_BINDERS covers bind1st, bind2nd,
ptr_fun and mem_fun and not unary_function or binary_function, and the
umbrella _LIBCPP_ENABLE_CXX17_REMOVED_FEATURES no longer exists in
libcxx at all. A wrong name preprocesses fine and fixes nothing.
Applied through CMAKE_CXX_FLAGS rather than to the one target, because
the tokenizer CMakeLists is upstream's and sources/ is a pinned
checkout. Project-wide is also the safer scope: the macro decides
whether libc++'s internal __binary_function alias resolves to
std::binary_function or to __binary_function_keep_layout_base, a base
class of std::less and friends, so defining it for a subset of
translation units would give those class templates two spellings in one
binary. Both bases are empty and, at C++17, carry identical members, so
the define changes no layout and no ABI.
Darwin only. On Linux the branch is unreachable and the macro is not a
name libstdc++ knows, so it would be inert even if taken; a Linux
configure with the flag forced on puts it on all 23 C++ TUs of
nemo_speech_openjtalk_frontend including dictionary.cpp at -std=gnu++17,
and on none of the 16 C TUs.
Mandarin needs nothing: cppjieba v5.6.7 and limonp have no removed C++17
constructs left (limonp replaced std::not1 and std::bind2nd with
lambdas) and cppjieba's own CI builds macos-14 and macos-latest at C++11
through C++20.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): repair OpenFST's FstImpl::operator= for gcc-14
The first WITH_NORM=ON build failed compiling fst_normalizer.cpp against the
installed OpenFST 1.8.3 headers:
fst.h:690:59: error: no match for 'operator=' (operand types are
'std::unique_ptr<fst::SymbolTable, ...>' and 'fst::SymbolTable*')
FstImpl's copy-assignment operator assigns the raw pointer returned by
SymbolTable::Copy() straight to a std::unique_ptr member. No C++ standard
allows that, so the line is ill-formed everywhere; it survived because nothing
instantiates FstImpl::operator= and gcc up to 13 only checks a template
member's body when it is instantiated. gcc 14 resolves non-dependent operator
expressions at template definition time, so it rejects the line in any
translation unit that includes <fst/fst.h>. The CI diagnostic confirms the
phase: it reads "In member function", not "In instantiation of", and carries
no instantiation backtrace.
That is why this surfaces only here. build_itn_deps.sh compiles OpenFST with
gcc-12 and upstream's own images build the runtime with gcc-13, so neither
compiler reaches the check; backend/Dockerfile.golang installs gcc-14 and
promotes it with update-alternatives, and fst_normalizer.cpp is the one
translation unit in this backend that includes OpenFST.
Fix it in the installed ITN prefix, which is the only copy the cmake build
compiles against, using the same .reset() spelling FstImpl::SetInputSymbols
already uses for the identical operation. libfst.so is linked before this runs
and cannot contain the function, since no compiler could ever have emitted it,
so there is no ABI or ODR consequence. The rule is guarded on both sides so a
pin bump to a fixed OpenFST fails loudly rather than silently no-opping.
Verified with a real gcc 14.2: the CI error reproduces byte for byte from a
file whose entire content is '#include <fst/fst.h>', and gcc 14 reports
exactly two errors over the whole OpenFST include closure this backend uses,
both of them these two lines. After the patch that closure compiles clean
under gcc-14 with the target's own flags. The step is reachable only under
WITH_NORM=ON, so 'make -n stage-libs WITH_NORM=OFF' mentions neither it nor
the ITN build, and darwin, which defaults WITH_NORM to OFF, never evaluates it.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): install cmake 3.31 on bases that ship less than 3.26
The JetPack r36.4.0 row dies on the first line of NeMo-Speech.cpp's
CMakeLists.txt:
CMake Error at CMakeLists.txt:3 (cmake_minimum_required):
-- Configuring incomplete, errors occurred!
Upstream opens with cmake_minimum_required(VERSION 3.26). That base image is
Ubuntu 22.04 jammy, whose apt cmake is 3.22.1, so configure aborts before it
reads a single one of the backend's -D flags. Every other Linux row in this
block is noble, which ships 3.28 and clears the bar, so the failure is one
base image wide rather than a code problem. Everything before it on that row
had already worked, including the OpenFST and Sparrowhawk ITN build.
No other Go backend needs this. parakeet-cpp and moss-transcribe-cpp share
the same JetPack base and both declare cmake_minimum_required(VERSION 3.18),
and nothing in the repo installs a cmake newer than the distro's, so there is
no existing pattern to reuse. Nothing depends on jammy's cmake staying 3.22
either: build_itn_deps.sh never invokes cmake at all, since OpenFST and
Sparrowhawk are autotools builds.
Kitware's release tarball rather than their APT repo or pip. The tarball is a
pinned URL with a published checksum, so an upstream release cannot change
what lands here. The APT repo does carry jammy arm64, but it serves a moving
latest that today is CMake 4.4, and 4.x drops compatibility with
cmake_minimum_required below 3.5, which vendored third_party subprojects
still declare; pinning it there would mean tracking Kitware's Debian revision
string instead of an upstream version. pip would drag a Python toolchain into
a backend that has none. 3.31.12 is the last 3.x release, so it clears 3.26
while keeping the CMake 3 policy surface, and it stays close to the 3.28 the
green noble rows already use. The binaries need only glibc 2.17 and carry no
libstdc++ DT_NEEDED, well under jammy's 2.35. doc/, man/, ccmake and cmake-gui
are not extracted; the final image is FROM scratch, but there is no reason to
page 100 MB of Qt GUI and docs through the CI cache.
Gated on the installed cmake actually being older than 3.26, so the rows that
already build green keep configuring with exactly the cmake they use today,
and folded into the existing ${BACKEND} block rather than added as a new
instruction, so no other Go backend image gains a layer and nothing above the
Vulkan SDK, CUDA, Go and protoc layers moves.
The symlink lands in /usr/local/bin and shadows apt's cmake. Unlike the protoc
shadowing that broke Sparrowhawk earlier in this series that is inert: protoc
has to agree with the libprotobuf headers it generates against, whereas cmake
links nothing into the product and has no ABI relationship with anything in
the image, and it resolves the symlink back to /opt to find its own Modules/
tree, so a 3.31 binary can never read 3.22's modules.
The version test avoids $(...) deliberately. BuildKit delivers a RUN heredoc
through an outer shell with an unquoted delimiter, so a command substitution
runs there, too early, in a container where the files it reads do not exist
yet, and its empty output is pasted into the script; the first draft took the
install branch on every row because of it.
Verified by building the block against nvcr.io/nvidia/l4t-jetpack:r36.4.0
arm64 under qemu, the row's actual base image: cmake 3.22.1 detected, tarball
checksum verified, 3.31.12 installed, and a cmake_minimum_required(VERSION
3.26) project configures with -G Ninja and builds, with CMAKE_ROOT resolving
to /opt/cmake/share/cmake-3.31. Same on ubuntu:22.04 amd64 and arm64.
ubuntu:24.04 skips the install, gains no /opt/cmake and still configures on
/usr/share/cmake-3.28. The NeMo-Speech.cpp compile itself on JetPack CUDA 12
is not reproducible here and remains for CI.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The primary is a caching mirror of the same two files, so an install
resolves its gallery from infrastructure the project controls. The GitHub
URI stays as a mirror, so behaviour is unchanged whenever the primary is
unreachable.
Docs and agent guides that either state the shipped defaults or hand out a
copy-pasteable gallery list are updated to match, so following them no
longer silently demotes an install off the new primary.
Assisted-by: Claude:claude-opus-5 [go vet] [go test] [golangci-lint]
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(version): include OS and arch in the outbound User-Agent
Registries and galleries already receive LocalAI/<version>; adding the
platform follows ordinary client convention and discloses nothing a
registry cannot infer from the manifest it is asked for.
Updates the User-Agent note in docs/content/getting-started/models.md,
which documented the old format.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* feat(downloader): identify LocalAI on outbound requests
pkg/oci has always sent a User-Agent; the downloader sent none, so gallery
reads, model-file downloads, resume probes, content-length probes and the
HuggingFace safety scan all went out as a bare Go HTTP client, unattributable
to LocalAI by the hosts serving them.
HuggingFaceScan moves off the client's Get shorthand to an explicit request
for the same reason — the shorthand gives no place to hang a header.
Extends the User-Agent note in docs/content/getting-started/models.md, which
claimed the header was sent only to Ollama and OCI registries.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* feat(gallery): add a mirrors list to gallery configuration
Mirrors are an availability fallback, tried in order only after the primary
URL fails. omitempty keeps existing configurations byte-identical.
The slice makes config.Gallery non-comparable with ==, which broke the two
slices.Equal callers in the runtime settings registry. Replace them with an
explicit Gallery.Equal / GalleriesEqual so a gallery list that differs from
the baseline only by its mirrors still counts as env/CLI-set. Equal compares
the Verification block by value; == compared it by pointer identity, which
called two structurally identical policies different.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* fix(downloader): treat an HTTP error status as a failed read
ReadWithCallback handed the response body to its callback whatever the
status was, so a 404 page or a 502 from a CDN arrived as if it were a
gallery index or a model config: it parsed to nothing, got cached for an
hour, and no caller could tell the source had been down. DownloadFile has
always checked the status; this path never did.
Mirror fallback depends on it — a source that answers with an error page
has to count as unreachable, or the next candidate is never tried.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* feat(gallery): fall back to mirrors when the primary source fails
Candidates are tried primary-first with a bounded timeout each, and a
source that just failed is skipped for a cooldown so a dead host is not
re-dialled on every listing. When every candidate is in cooldown they are
all tried anyway: refusing to serve a gallery we might be able to reach is
worse than one slow request.
The one-hour index cache is untouched and stays keyed on the gallery's own
identity, so a mirror-served fetch fills the entry the primary would have.
No SSRF validation is applied to the candidates. validateGalleryConfigURL
guards GetGalleryConfigFromURL because that URL arrives in a request body;
mirrors come from the operator's gallery configuration, the same place the
primary has always come from, and the index fetch has never validated the
primary. Validating mirrors while the primary goes unchecked would buy
nothing and would break the deployment mirrors exist for — an index served
from a host on the LAN.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* fix(gallery): loosen the mirror fetch timeout and stop blaming the caller
The downloader only ever bounded response headers, never the body, so the
per-attempt deadline added with mirror fallback was the first whole-transfer
timeout this path has had. At 30s the default 2.2 MB index demanded ~75 KB/s
sustained: a rural-DSL, mobile or satellite user who used to wait 60s and
succeed would now fail, and then eat a 10-minute cooldown on a source that
was perfectly healthy. Raised to 120s (~19 KB/s), which no link that could
go on to download a model will miss, and made it a var so a test can shorten
it and prove a hanging candidate is actually abandoned.
Caller cancellation is no longer recorded as a failure of the source.
Unreachable today since getGalleryElements passes context.Background(), but
once a request context is wired through, a browser disconnect would have
blackholed every candidate for ten minutes over something the sources had
no part in.
Also document that mirrors do not cover a .ref gallery URL: the reference is
resolved before mirrors are considered, so a .ref that cannot be fetched
fails the gallery outright. Routing .ref resolution through the candidate
list needs a per-candidate resolve-and-fetch and a decision about cache
identity, which is more than this change should carry.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* feat(gallery): serve the last known good index when everything is offline
A successful fetch is cached alongside the models directory and served when
no source is reachable, so an offline or airgapped machine can still list
its gallery. Entries may be stale in that state, and the fallback is logged.
The copy is deliberately kept out of the models directory, where a <name>.yaml
file is read as an installed model's configuration, and is named after a digest
of the gallery URL so the model and backend galleries cannot collide. Writing
it is best effort: a read-only or full disk must not fail a fetch that
otherwise succeeded.
Also corrects the mirror scheme list in the docs: the HuggingFace prefixes are
huggingface://, hf:// and hf.co/, not huggingface:.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* fix(gallery): only cache a response that is really a gallery index
The last known good copy was written on any 2xx, before anything looked
at the bytes: the parse only happens later, in getGalleryElements. A
captive portal, a corporate proxy or a CDN error page all answer HTTP 200
with HTML, so any of them could overwrite a good copy. The listing fails
then and there, and the next offline start — the one case this cache
exists for — serves the interception page instead of the gallery it
already had.
Probe the body before persisting it: unmarshal into a []any and keep the
older copy unless the result is a non-empty sequence. An empty document
is rejected too. It parses fine, so a parse-only check would still let a
blank response replace a populated index with one that lists nothing,
which from the user's side is the same outage; and an empty index is
worth nothing offline, so there is no case where caching it beats keeping
what came before. The live body is still returned to the caller — the
probe gates persistence only, and getGalleryElements remains the thing
that reports a real parse failure.
Also in this pass:
- The empty-basePath guard only caught exact "". galleryCachePath(".")
and galleryCachePath("models") still resolved the cache sibling against
the process working directory, which is what the guard was written to
prevent. Reject any non-absolute base.
- The docs claimed the offline cache "applies to every gallery, with or
without mirrors". Not true for a .ref URL: the reference is resolved
before the cache is consulted, so a .ref gallery fails offline even
after a successful earlier fetch, and the cache file it writes can
never be read. Extend the .ref warning and qualify the sentence.
- pkg/oci's UserAgent comment never mentioned the platform component
added earlier on this branch.
- resetGalleryFailures and expireGalleryFailure had no non-test callers;
move them into the test file.
- The all-candidates-failed error reported len(attempt), so a three
mirror gallery with two sources in cooldown said "all 1 source(s)
failed" — which reads as a misconfiguration. Report how many were
configured and how many were skipped.
- Give the package's tests their own TMPDIR. The cache is a sibling of
the models directory, which is right in production, but specs that
build a models directory directly under /tmp made the sibling resolve
to /tmp/cache and left it behind after every run.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* fix(gallery): convert the new tests to Ginkgo and clear the lint gate
.agents/coding-style.md requires Ginkgo v2 + Gomega for every Go test and
has forbidigo enforce it; the stdlib-style tests still in the tree are tech
debt, not a pattern. Every test file this branch added was written in the
forbidden style, which is what turned CI red.
Convert all five of them. internal had no suite bootstrap, so add one;
core/config, core/gallery and pkg/downloader already have theirs and are
reused, so no package mixes styles. pkg/downloader/useragent_test.go and
read_status_test.go were not in CI's forbidigo list but used the same
forbidden calls, so they are converted too.
The one conversion with a trap in it is core/gallery. Go's t.TempDir()
yields $TMPDIR/<TestName>NNNN/001, so the gallery cache — a sibling of the
models directory — was isolated per test. GinkgoT().TempDir() yields a flat
$TMPDIR/ginkgoNNNN, which would put every spec's cache in one shared
directory and break the specs that count files in it. tempModelsDir()
restores the original isolation.
Also make the deliberate cleanup-path ignores explicit with `_ =`, drop the
gallery cache directory to 0750 (nothing outside the server's own user and
group reads it), and justify the cache read with a #nosec G304 comment in
the form already used elsewhere in the tree: the path is a hex sha256 under
a fixed directory with a non-absolute base already rejected, so no
caller-supplied text reaches it.
Re-ran the mutations these specs were verified against — dropping the
platform suffix from UserAgent, making Gallery.Equal ignore Mirrors and
ignore Name, removing persistGalleryIndex's validity probe, removing the
!filepath.IsAbs guard, not skipping a cooled-down candidate, and dropping
the per-attempt timeout. All seven still fail the converted specs.
Assisted-by: Claude:claude-opus-5 [go vet] [go test] [golangci-lint] [gosec]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
A container engine started from a socket-activated system unit leaks a bare
LISTEN_PID into every container it spawns, with no matching LISTEN_FDS. LocalAI
read that as a malformed activation attempt and refused to start:
ERROR Error running the application error=loading systemd socket
activation listeners: invalid LISTEN_FDS ""
systemd's own sd_listen_fds() treats either variable being absent as "not
activated" rather than as an error, so do the same and fall back to ordinary
--address binding. A value that is present but malformed is still rejected, so
a real activation attempt cannot silently bind the wrong socket.
Fixes#11390
Assisted-by: Claude:claude-opus-5 [golangci-lint]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(react-ui): restore 3D Studio results and history
Keep large conditioning-image payloads out of the rendered request panel so the generated viewer can mount reliably. Accept clipboard images and synchronize 3D history consumers so new results appear in Studio without a reload.
Cover clipboard input, bounded request rendering, result display, and cross-view history synchronization with Playwright.
Assisted-by: Codex:gpt-5 Playwright
* perf(react-ui): idle the 3D viewport when still
Limit auto-rotate rendering to 30 FPS and stop scheduling frames when rotation is disabled. Resize, view controls, and pointer input invalidate the still frame on demand.
Assisted-by: Codex:gpt-5 Playwright
The backend could configure four of the engine's knobs (block size, KV block
count, max sequence length, max concurrent sequences) out of a config surface
that is considerably larger. Speculative decoding, prefix caching, the
chunked-prefill token budget, the scheduling policy and the external KV
connector were reachable from vllm.cpp's own HTTP server and from nothing
LocalAI could write in a model config.
Config now goes through `engine_args:`, the same map the vLLM and SGLang
backends take, with keys spelled as vLLM's own CLI flags so a speculative_config
or kv_transfer_config block written for vLLM works verbatim. The legacy
`options:` list keeps working and reads every key too; engine_args wins where
both set one. Unknown keys are logged and ignored rather than fatal: the field
is shared with the other engines, so a config carrying their knobs must not take
the model down.
Two details worth knowing:
`enable_prefix_caching: false` maps to the ABI tri-state force-OFF (2), not 0.
0 means "let the model capability decide" and dense architectures default the
cache on, so collapsing the two would silently enable it against an explicit
false. enable_jump_forward (ABI v10) shares the encoding, deferring to
VT_ENABLE_JUMP_FORWARD instead of to the model.
The importer probes config.json on a vllm-cpp import and writes
speculative_config: {method: mtp} when the checkpoint declares an MTP head, the
safetensors analogue of the llama-cpp importer's GGUF probe. DFlash draft repos
are refused with a warning instead, since a drafter cannot serve alone and the
pairing is not derivable from either repo. The draft path is resolved against
LocalAI's model directory, because the engine only looks in a directory holding
config.json or in the HF cache and never downloads: the repo-id spelling the
vLLM docs teach used to die deep in the load with "draft checkpoint not found".
docs/content/features/text-generation.md gains a vllm.cpp section covering the
engine_args table, all three speculative methods, LMCache and the legacy list.
The backend had no documentation page before.
This replaces a branch that had gone stale behind master and carried its own
route to ABI v10, which #11386 has since landed in minimal form. Rebased onto
that as a single commit rather than replaying the intermediate steps, whose
ABI v9 mirrors no longer make sense against master's pin. The Darwin build
fixes for Apple Clang's gnu-folding-constant diagnostic on C++, Objective-C and
Objective-C++, originally authored by localai-org-maint-bot, are folded in here.
Verified: `make abi-check` agrees at v10; unit specs, core/config and
core/gallery/importers green; and the full e2e passes in 1330s against a CPU
libvllm.so reporting ABI v10 with Qwen_Qwen3.5-0.8B-Q4_K_M.gguf (load, blocking
completion, streaming, chat and tool calls).
Assisted-by: Claude:claude-fable-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Add the MTP and plain Q4_K_M GGUF builds with their shared vision projector so LocalAI users can select accelerated or fallback llama.cpp inference.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add Q4_K_M and Q8_0 builds of the popular refusal-removed Qwen3.5 9B fine-tune, including its multimodal projector.
Assisted-by: Codex:gpt-5 [web]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Switching from Backend Traces back to API Traces crashed the page
with "can't access property status, e.response is undefined" (#11376).
The API table briefly renders the previous tab's backend rows while the
refetch effect is still pending, and those rows carry no `response`
envelope. The status column dereferenced it unguarded. Render a neutral
placeholder instead of throwing, and cover the tab-switch scenario with
a regression spec.
Assisted-by: opencode:big-pickle
Signed-off-by: Nandana Dileep <110280757+nandanadileep@users.noreply.github.com>
The Go bindings mirror vllm.h by hand and refuse a library whose
vllm_abi_version differs from what they were written against. Two
automated pin bumps (#11174, #11352) moved VLLM_CPP_VERSION onto engines
declaring ABI v10 while govllmcpp.go still mirrored v5, so every
vllm-cpp image built since then panics at startup on every platform:
panic: vllm-cpp: ABI mismatch: library reports v10, backend built against v5
Grow both PODs to the v10 layout: vllm_model_params gains
speculative_config, enable_prefix_caching, max_num_batched_tokens,
scheduling_policy, kv_transfer_config and enable_jump_forward (88 bytes),
vllm_sampling_params gains the v8 logits-processor pair (136 bytes). The
offsets in the specs come from offsetof() against the pinned header. All
of the new fields are inert when zeroed, so the engine behaves exactly as
it did under v5; the backend sets none of them.
Nothing cross-checked the two files, which is why a blind pin bump could
ship a backend that cannot load. The library build now runs abi-check
first: it compares VLLM_ABI_VERSION in the fetched header against
abiVersion in govllmcpp.go and fails the build naming both, instead of
leaving the mismatch for a user's runtime.
Fixes#11379
Assisted-by: Claude:claude-fable-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Recover parser panics at metadata boundaries, skip unneeded remote arrays, and use the parser's overflow-hardened release. Keep detached gallery workers and CrispASR probes from terminating their processes on malformed GGUF input. Disable startup warming in the provided Compose files as an operational fallback.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
docs(blog): cover the terminal agent, and fix the counts in the intro
The 4.8 post never mentions that `local-ai chat` stopped being a REPL
and became an agent (#11291): the nib harness compiled into the binary,
with tool use behind an approval gate, sub-agents, MCP servers, plugins
and skills, auto-configured against the local instance. It also ships a
shell integration script for zsh, bash and fish that binds Ctrl+Space.
That is one of the larger user-facing changes in the release and it was
missing from both the post and the release-notes highlights. Added a
section after 3D generation, including the breaking changes for anyone
who had habits around the old REPL: `/clear` is gone in favour of
`/compact`, and a model switch now keeps the conversation.
While in the intro, corrected the counts. The post said 374 pull
requests in twenty-one days, which was accurate when it was drafted on
the 4th but not once v4.8.0 was tagged on the 5th. The published release
notes say 386 in twenty-two days, and the intro now matches them rather
than contradicting them.
For the record, neither figure is exactly right: `git log --format=%s
v4.7.1..v4.8.0 | grep -cE '\(#[0-9]+\)$'` counts 388 squash-merged pull
requests, and 389 from v4.7.0. The notes were cut before the last few
landed. Matching the published notes was the priority here, since that
is the artifact everyone else quotes, and 386 is the number already in
circulation.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The two landing-page clips predated the v4.8.0 interface work (#11288,
#11305, #11307): the gallery clip showed the retired light-theme Install
Models table, and the hero clip toured the Nodes pages in a full browser
window while its caption promised a chat completion on CPU.
Both are re-recorded from a real local-ai built from v4.8.0, dark theme,
app chrome only:
- hero-ui.mp4: a chat completion on lfm2.5-1.2b-instruct streaming on
CPU with the live tok/s meter, so the caption now matches the footage.
The poster frame is regenerated from the new clip.
- gallery.mp4: the Discover rail and detail pane, the hardware
recommendation lanes, the VRAM-by-context chart, and a real install
with the live progress banner.
The hand-typed model count moves from 1,585 to 1,255 in the three places
it appears, matching the distinct-model count the recorded UI shows on
screen. The 3d-generation clip is untouched: the post-capture UI changes
do not show in its footage.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
Add the official Q4_K_M and Q8_0 GGUF builds with their matching vision projectors so the compact agentic model can be installed through LocalAI.
Assisted-by: Codex:gpt-5 [web]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Cosign v2.4.1 does not select the Sigstore bundle format by default, while LocalAI's verifier only consumes OCI bundle referrers. Request the format explicitly for both registries and guard the producer contract with a shell regression test.
Document strict backend integrity configuration and release-tag identities for operators.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Register JSON API exchanges before their handlers run so the traces dashboard can surface active work. Replace the live entry with the completed persisted record under the same ID, and clean it up if a handler panics.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The runtime section described the small core and on-demand backends but
never stated the simple fact readers look for: one model can run on
llama.cpp while the next loads on vLLM, SGLang or MLX, behind the same
endpoint, and switching is one line in the model's config.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* docs(blog): new DeepSeek and Laguna numbers, visuals, humanizer pass
vllm.cpp master moved 26 commits past what the post was written against,
and two results changed enough to matter. Both came from the same lever:
staging weights device-resident at load instead of reading them from the
GGUF mmap over unified memory, which the GB10 reads about 20% slower per
GEMV than device memory.
- DeepSeek-V4-Flash against DwarfStar: 0.997x parity becomes 1.144x
ahead, 18.69 vs 16.33 tok/s decode, same generated tokens.
- Laguna-XS-2.1 against vLLM: 87% becomes 1.03x, 44.46 vs 43.10 tok/s.
New row in the scoreboard.
Adds three visuals. A chart of throughput against every reference engine,
which is worth having now that the spread is 0.976 to 1.144 rather than a
flat line at parity. The Activity page with four installs running, and the
model detail pane with all four pocket-35b variants. Both screenshots were
recaptured on 2026-08-04 because #11288, #11305, #11307 and #11222 had all
changed those pages since the earlier set.
llama.cpp is deliberately absent from the chart: its 1.18x is a prefill
ratio, and putting it on the same axis as throughput ratios would be
comparing two different measurements.
Also carries the media the release notes embed, since a GitHub release
body needs URLs that survive publishing and drag-and-drop has no CLI.
Supersedes #11364.
Humanizer pass on the prose. The post had collected five exactness idioms
in one section (token-for-token, byte-exact twice, byte-identical,
token-identical). One is precision, five is a tic, so the 27B row keeps
its "token-for-token identical" where identical output is the actual
claim and the rest say what they mean. That also fixed a hyphen in
predicate position ("is token-identical").
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
* docs(blog): redraw the benchmark chart as a branded card
The Flint bar chart was generic: default palette, no brand, and drawn
from zero, which made five ratios between 0.976 and 1.144 look like five
bars of roughly equal length.
Redrawn in the style of recorder-for-agents' render-card.sh cards, the
same shape as the vllm.cpp README GIF. Palette taken from the two logos
rather than invented (LocalAI navy #0E2632 and teal #469AAF, vllm.cpp
teal #3AB4CA), SVG generated by a small JS loop so the geometry is exact
at any scale, headless Chrome to PNG at 2x.
The substantive change is that bars now run from the 1.00 parity line
instead of from zero. Deviation is what the data is about, so DeepSeek's
+14.4% and MLX-LM's -2.4% are both legible, and the one row that is
behind is the one row in amber. Each bar carries its ratio and the raw
measurement under it.
Keeps the .html source next to the .png so the chart is editable later:
change a number, re-run render-card.sh.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This is the text users read in the backends list and the gallery, and it
was the last place still describing vllm.cpp as "a from-scratch C++20
port of vLLM created and maintained by the LocalAI team" with no
indication of maturity.
Three corrections, matching the v4.8 release notes and blog post:
- It leads with ALPHA. These are alpha development builds and llama-cpp
stays the recommendation for production, which is the single most
useful thing to know before clicking install.
- It is maintained by the LocalAI team but developed in its own
repository and usable without LocalAI. vLLM is named for what it
actually is, the reference implementation that output is checked
against and benchmarked against, rather than just the thing that was
ported.
- It records the featureset that has grown past vLLM: GGUF loading,
speculative decoding and KV offload, alongside the architecture and
hardware coverage that were already listed.
Also notes that the project is expected to be renamed, with the new name
still to be decided, so anyone who installs it now is not surprised
later.
vllm-cpp-development inherits all of this through the YAML anchor, so
both entries are covered by the one edit. Verified the file still parses
and that both entries carry the new text.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* docs(blog): final figures for the 4.8 post, and the MLX provider
The cycle closed at 374 PRs over twenty-one days, not the 321 over
eighteen the post was written against. Corrects the summary, the opening
line, the contributor count and the gallery total, and moves the date to
the day the release is cut.
Adds the MLX GEMM provider (#11137), which merged after the post was
written and is the one number an Apple Silicon reader wants: 1.54x to
2.19x on an M4 with time to first token roughly halving, both arms
toggled on one binary. The +/-10% caveat travels with the table rather
than being left in the PR.
Two lines edited against the no-ai-slop skill while I was in the file,
the same pass #11324 ran over the engines post:
- The opener balanced two clauses across a colon and closed on "without
lying to you", which is the built-to-be-quoted shape readers picked
out of the HN thread. It is a flat statement now.
- "This is a new modality rather than a new backend under an existing
one" is a binary contrast that says nothing the next clause does not.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
* docs(blog): call vllm.cpp alpha, and finish the no-ai-slop pass
vllm.cpp is not a released backend and the post read like it was. The
old wording buried the caveat in a block quote at the end of the section
and still said "first release of a young engine". It now says plainly,
before the caveat can be skipped, that these are alpha development
builds, that shipping them in 4.8 is about letting people try the thing
rather than recommending it, and that llama-cpp stays the default.
Also completes the no-ai-slop pass I had only half run. Counting the
lines built to be quoted, headings and section endings included, the post
is in reasonable shape: long flat informational stretches, tables
followed by a plain finding, headings that are labels rather than
epigram-verdicts. Three patterns survived, each one an item in eval.md:
- "and inverts that:" set the usual shape against ours across a colon.
The sentence works without the frame.
- "Two things were conflated there: a signal, which needs one line, and
the detail, which needs somewhere to put it" is a role-assignment pair.
Says what happens instead.
- "The maturity statement from the release notes is worth repeating in
full" is throat-clearing in front of a quote, and the quote is gone.
Left the rest alone. Minimum effective edit, not a rewrite.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
* docs(blog): present vllm.cpp as a community project, with its own numbers
The post described vllm.cpp as "a from-scratch port of vLLM, written and
maintained by the LocalAI team". Two things wrong with that. It is a
community project, and it has stopped being only a port: it loads GGUF,
runs on CPU, Metal and Vulkan, ships speculative decoding and KV offload,
and its benchmark page measures against llama.cpp, MLX-LM and DwarfStar
as well as vLLM, because those are the engines it competes with on that
hardware.
vLLM's role is now stated for what it is, the reference implementation.
Correctness is checked against it and the scoreboard is kept against it.
Also flags that the name will probably change, since it is drifting far
enough that vllm.cpp will eventually mislead.
Adds real numbers from the project's own docs/BENCHMARKS.md rather than
adjectives: 1.045x vLLM at concurrency 1 on Qwen3.6-27B NVFP4 with
token-for-token identical output, 1.010x and 1.013x at c16 and c32 on the
35B MoE and behind below that, prefill 1.18x over llama.cpp on CPU
aarch64, 97.6% of MLX-LM warm total on an M4. Upstream's own caution
travels with them: it treats c2 through c32 as ties because its noise
band is 0.5% and those margins are 0.7% to 1.7%.
Every figure was checked against ~/_git/vllm.cpp/docs/BENCHMARKS.md
rather than restated from memory. The heading is marked alpha to match
the section body.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
* docs(blog): say who maintains vllm.cpp, and add the DeepSeek Flash result
Two corrections to the previous commit.
"A community project" says nothing and was not quite true either. The
LocalAI team maintains vllm.cpp. Community-first is the intent, not a
description, so it now says that and says what backs it: its own
repository, its own docs, benchmark record and issue tracker, and it runs
without LocalAI anywhere in the picture.
Adds the DeepSeek-V4-Flash result, which makes the divergence point
better than any of the prose around it. That model does not run on vLLM
on a single GB10: every vLLM-loadable checkpoint is 156 GB or more
against a 119 GiB unified pool, and the only quant that fits is an
extreme-low-bit GGUF that vLLM cannot load. vllm.cpp reads GGUF and runs
it at 16.28 tok/s against ds4's 16.33, a parity result. Also notes MTP
speculative decoding, token-identical to vLLM's and about 4% faster at
concurrency 1.
Both figures checked against ~/_git/vllm.cpp/docs/BENCHMARKS.md.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
* docs(blog): lead the DeepSeek result with what we run, not with what vLLM cannot
The previous version opened on "that model does not run on vLLM on a
single GB10 at all". Wrong emphasis twice over: it makes a strong
negative claim about another project the headline, and it buries the
actual result, which is that vllm.cpp runs DeepSeek-V4-Flash at roughly
2-bit (IQ2_XXS mixed, about 80 GB) on a single DGX Spark and decodes at
16.28 tok/s against DwarfStar's 16.33.
The size constraint is still there, stated as the reason the quant is
what it is rather than as a point about vLLM: at 300B+ total parameters
even a 4-bit checkpoint is 156 GB or more, so a 2-bit GGUF is what fits
the Spark's 119 GiB unified pool.
The table row now names the quant and the box (IQ2_XXS, one DGX Spark)
instead of just "GGUF, GB10", since that is the part a reader with a
Spark wants.
Figures unchanged and still from ~/_git/vllm.cpp/docs/BENCHMARKS.md.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
* docs(blog): say the new name is undecided
"The name will probably change at some point" invited the obvious
question. It now says the rename is expected and the name is still to be
decided, which is the actual state.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Replace copied HauhauCS base-model text with metadata for the actual Genesis Hermes V6 artifact and link its upstream base model.
Assisted-by: Codex:gpt-5 [Hugging Face]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
No -gpu-rocm-hipblas-llama-cpp image has been published since 2026-08-01.
Every build since has been killed by GitHub at exactly its 6h job limit:
job 91830652349 cancelled 6h00m (2026-08-04)
job 91763226161 cancelled 6h00m (2026-08-03)
job 91466626154 cancelled 6h00m (2026-08-02 full matrix)
The registry shows the damage: master-gpu-rocm-hipblas-llama-cpp last
built 2026-08-01 05:53, latest-gpu-rocm-hipblas-llama-cpp 2026-07-15,
against master-cpu-llama-cpp which is current.
Same cause as #11321, different mechanism. Since #11255 every x86 GPU
image also builds ggml's CPU_ALL_VARIANTS matrix. SYCL died because icpx
stalls on one translation unit; ROCm dies on volume. hipcc compiles the
HIP kernels once per entry in AMDGPU_TARGETS, and that list is eleven
architectures (gfx908, gfx90a, gfx942, gfx950, gfx1030, gfx1100, gfx1101,
gfx1102, gfx1151, gfx1200, gfx1201). The CPU matrix lands on top of that.
The numbers are unambiguous. The same job took 2h27m in the 2026-07-26
full matrix, before #11255. #11255 merged 2026-08-01 07:26, an hour and a
half after the last image was published, and it has been 6h00m ever since.
The tail of the last run shows it 61% through ggml-hip at the 83 minute
mark, still building HIP template instances.
Route hipblas to the portable fallback, exactly as #11321 did for SYCL and
for the same practical reason: it is what these images shipped before
#11255, and run.sh already prefers *-cpu-all when present and falls back
otherwise. Expected to restore the 2h27m build with room to spare.
Not fixed here: the CPU variant matrix is genuinely wanted on ROCm for
partial offload. Getting it needs the build to fit in 6h, which means
trimming AMDGPU_TARGETS or splitting the job per architecture. Both are
larger changes than unbreaking the image, and neither should ride along
with a build that is currently not shipping at all.
Verified: make test-build-scripts passes, including the extended
llama-cpp-build-target_test.sh. bonsai is unaffected (own compile script,
ROCm builds in 1h52m) and turboquant has no hipblas variant.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): enable and vendor the MLX GEMM provider on darwin/metal
The darwin vllm-cpp image built the Metal backend with vllm.cpp's native MSL
GEMM only. vllm.cpp also ships an optional MLX provider for the dense GEMM,
kept OFF upstream because it costs a ~19 MB libmlx.dylib plus a ~105 MB
mlx.metallib, on the stated position that it must earn that cost by
measurement.
Measured on an Apple M4 (16 GiB, macOS 26.5.2) it does. One binary, arms
toggled with VT_OP_PROVIDER_DISABLE=mlx so there is no build-difference
confound, Qwen3-1.7B-bf16 p=512 g=128, 2 reps, arm order alternated per rep:
B=1 5.79 vs 3.08 agg tok/s (1.88x) TTFT 3.32 s vs 7.68 s
B=8 25.70 vs 13.69 (1.88x) TTFT 13.95 s vs 34.38 s
B=16 38.65 vs 17.69 (2.19x) TTFT 18.33 s vs 54.48 s
Peak RSS is unchanged (6.65 to 7.50 GB in both arms) and the output is
bit-identical: vllm.cpp's three-way parity test measures mlx-vs-msl NMSE of 0
on all six shapes, and mlx-vs-cpu equal to msl-vs-cpu, against a 5e-4 bar. MLX
serves the dense GEMM alone; paged attention stays vllm.cpp's own kernel
because MLX has no paged-KV primitive. Full disposition, including the
INDICATIVE status and the isolation actually achieved, is in vllm.cpp
docs/BENCHMARKS.md "MLX GEMM provider A/B on Apple M4".
Build: MLX comes from the pinned prebuilt pip wheel (MLX_VERSION, default
0.29.3) into a venv under the backend dir. Building MLX from source needs
`xcrun metal`, i.e. a full Xcode the macOS runners do not have, while the wheel
ships include/, lib/libmlx.dylib and the compiled metallib ready to link. The
install is a stamp FILE rather than a phony target, because a phony
prerequisite is always newer than libvllm and would re-link it every
invocation. VLLM_CPP_MLX=off restores the previous Metal build.
Packaging vendors libmlx.dylib, mlx.metallib and MLX's MIT license into
package/lib/. Three things this had to get right, each verified on the M4
before it was written rather than after:
1. libvllm.dylib links @rpath/libmlx.dylib and its build-time LC_RPATH points
inside the build venv, a path no user has. Every build rpath is deleted
and replaced with @loader_path/lib.
2. MLX loads its metallib from beside its OWN dylib, so both files must land
in the same directory or every Metal op fails with "Failed to load the
default metallib".
3. install_name_tool invalidates the code signature and macOS refuses to load
an arm64 image with a stale one, so the patched library is re-signed
ad-hoc.
Verified end to end on the M4 by building through this Makefile and running the
packaged artifact: `DYLD_PRINT_LIBRARIES` resolves libmlx from package/lib/,
`codesign -v` passes, no build-venv path survives in the load commands, and a
real generation runs with the provider selected (op=65 selected=mlx) and zero
metallib failures. A missing rpath now fails the build instead of the user's
first inference.
Cost: the darwin vllm-cpp image grows by about 124 MB.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
* fix(vllm-cpp): default the MLX GEMM provider OFF on darwin
This branch opened with VLLM_CPP_MLX=on, justified by an A/B that measured the
MLX provider at 1.88x to 2.19x against the native MSL GEMM. That measurement was
correct when taken and is now stale: vllm.cpp's own Metal kernels have improved
several-fold since, through mma prefill attention, a vectorised decode V
accumulation, vectorised attention staging, a fused qk-norm-RoPE preamble and a
simdgroup-per-row softmax. The native path MLX was compared against no longer
exists.
Re-measured on the same Apple M4, in the same binary, with the arms toggled by
VT_OP_PROVIDER_DISABLE=mlx, on Qwen3-1.7B-bf16 warm at p=512 g=128:
MLX provider ON prefill TTFT 1370 ms warm throughput 11.98 tok/s
MLX provider OFF prefill TTFT 1400 ms warm throughput 22.06 tok/s
Shipping the previous default would have halved Apple Silicon throughput.
MLX's steel GEMM is still about 20% faster than ours in isolation, but the
provider pays a per-op mx::eval synchronisation plus an output memcpy, because it
cannot write into our buffer. Across prefill's roughly 112 GEMMs that overhead
leaves a 2% gain; on decode, where the same synchronisation is paid once per
matmul per token, it costs 46%. The option is kept for prefill-dominated
workloads, where the margin is small but real.
The README section is rewritten rather than patched: it previously presented the
stale table as the reason for the default, so leaving it in place would have made
the new default look arbitrary.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): bump vllm.cpp and default MLX ON, gated to prefill
Bumps VLLM_CPP_VERSION from 9e1c9025 to eec09bed and turns VLLM_CPP_MLX back on.
These two must move together, which is why they are one commit.
Upstream now shape-gates the MLX provider to prefill: it declines m < 2, which is
exactly the decode GEMV. MLX's steel GEMM wins prefill, 524.5 ms of TTFT against
602 for the native path, but loses decode badly because the provider pays an
mx::eval synchronisation and an output memcpy on every call while decode makes
about 112 calls per token. Ungated it does both; gated it does only the good half.
Measured on an Apple M4 with Qwen3-1.7B-bf16 warm at p=512 g=128:
MLX gated to prefill (pin >= 89c46aeb) TTFT 524.5 ms 24.40 tok/s, 99.1% of MLX-LM
MLX ungated (older pins) TTFT 537 ms 12.7 tok/s
MLX off TTFT 602 ms 23.9 tok/s
This branch briefly defaulted the provider off, which was the correct call for an
ungated provider at the old pin. The gate is what makes on correct again, so the
pin and the flag are coupled: rolling VLLM_CPP_VERSION back before 89c46aeb while
leaving MLX on would select the middle row and roughly halve throughput. Both the
Makefile comment and the README state that dependency explicitly.
The bump also brings six Metal kernels landed upstream since the old pin — mma
prefill attention, a vectorised decode V accumulation, vectorised attention
staging, a fused qk-norm-RoPE preamble, a simdgroup-per-row softmax and a
simdgroup-per-head preamble — which take the non-MLX Metal path from 89.4% to
96.4% of MLX-LM on their own.
One caveat, recorded in the README: MLX's GEMM is not bit-identical to the native
kernel, so an MLX build produces a different greedy sequence than a non-MLX build.
That is a property of the provider rather than of the gate and predates this
packaging.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(vllm-cpp): correct the MLX-gated figure to 97.6%, from 99.1%
The previous commit quoted 99.1% of MLX-LM for the prefill-gated MLX build. That
figure divided by a two-run MLX-LM baseline, 27.135 and 27.744 generation tok/s
averaged to 27.44. Re-measured interleaved with ours over four ABBA blocks,
MLX-LM's decode is 27.848 with a 0.34% spread across six runs, so the 27.135 was
an outlier and averaging it in overstated us by roughly 1.5 points.
Corrected: the gated configuration is 24.37 tok/s, or 97.6% of MLX-LM, and the
MLX-off build is 23.9 tok/s or 95.9%. Prefill TTFT is unchanged at 524.5 ms
against MLX-LM's 532.6, so we remain about 1.5% faster there.
Nothing else changes. MLX still wins prefill and loses decode, the shape gate is
still the right disposition, and the pin and the flag are still coupled. The gate
is worth about 1.7 points over the MLX-off build rather than 2.7.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): pin MLX gate from mainline
The previous pin was a merge commit from the experimental C ABI v9 branch. Pin the same MLX prefill gate on upstream main so the backend build does not pull unrelated ABI v9 work into every platform variant.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): restore backend build portability
Keep the current master pin when enabling MLX so every backend variant builds against the known-good vllm.cpp revision. Suppress Apple clang’s GNU constant-folding diagnostic for Objective-C++ Metal compilation only, since upstream treats warnings as errors.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): demote MLX header VLA warning
MLX 0.29.3 headers trigger Apple clang's gnu-folding-constant diagnostic in the Objective-C++ provider. Keep the diagnostic visible while exempting only it from vllm.cpp's global warnings-as-errors policy.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): suppress MLX header VLA warning
Target-level Objective-C++ -Werror is appended after the directory flags, so a no-error demotion is re-promoted. Disable this single warning for the MLX header while keeping every other warning fatal.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): pin source-scoped MLX warning fix
Move the AppleClang warning exception into vllm.cpp where its target warning policy is defined, and pin LocalAI to that source-scoped fix.
Assisted-by: Codex:gpt-5
* fix(vllm-cpp): pin effective MLX warning suppression
The source-scoped no-error flag was overridden by the target warning policy. Pin the companion vllm.cpp change that disables only the MLX header diagnostic for its Objective-C++ translation unit.
Assisted-by: Codex:gpt-5
* fix(vllm-cpp): pin diagnostic pragma fix
Pin the companion vllm.cpp correction that scopes the AppleClang folding warning suppression inside the MLX translation unit, after command-line warning policy.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): pin remaining Darwin build fixes
Advance the MLX-enabled backend to the vllm.cpp revision already validated by the dependency update branch. This includes the feature guards and AppleClang pragma boundary needed by the Darwin build.
Assisted-by: Codex:gpt-5 [systematic-debugging]
* fix(vllm-cpp): pin MLX system dependency boundary
Pin the companion vllm.cpp change that models MLX as an imported system dependency, keeping third-party header diagnostics out of the project's warnings-as-errors policy while retaining fatal warnings for project sources.
Assisted-by: Codex:gpt-5 [Codex]
* fix(vllm-cpp): pin scoped MLX warning guard
Advance vllm.cpp to the companion fix that keeps MLX headers on a SYSTEM dependency and scopes AppleClang folding-constant suppression to the external includes.
Assisted-by: Codex:gpt-5 [systematic-debugging] [test-driven-development]
* fix(vllm-cpp): use available MLX wheel
MLX 0.29.3 is no longer available to the Darwin runner, so the backend build stopped before CMake. Pin the first available compatible wheel and keep the documented default in sync.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update antirez/ds4
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(ds4): link upstream CUDA MMQ objects
The updated ds4 CUDA object now calls into the vendored MMQ implementation. Build and link those objects into both the gRPC server and distributed worker.
Assisted-by: Codex:gpt-5 [Codex]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Wait for observable loop events instead of budgeting hundreds of milliseconds for scheduler timing. Keep a short bounded overlap observation for the two-leader exclusion check.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The blog has a deep post for 4.8 and a history post that covers the earlier
releases at summary altitude, but nothing in between. These five fill that
gap in the same shape as what-landed-in-localai-4-8: what the release was
for, runnable examples, and the limits that apply.
Every endpoint, CLI flag, env var and gallery entry is verified against the
matching release tag rather than taken from the release notes. That caught
two paths the published 3.10.0 notes got wrong: tracing is /api/traces, not
/api/v1/trace, and a stored response is fetched from /v1/responses/:id, not
/api/v1/responses/{response_id}.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* blog: rewrite the engines post without the AI tells
The HN thread on this post (item 49125065) spent most of its comments on the
writing rather than the engines. Readers quoted specific lines back as tells.
This is the same post with the same numbers, edited against the updated
no-ai-slop skill.
Every figure, table and link is unchanged, except that "27% of the memory"
is now the underlying 363 MB against 1328 MB from the table.
Two substantive framing fixes, both from the reply draft in
hn-reply-engines-post.md:
- vllm.cpp is no longer implied to be a speed win. The table is a tie, the
result is the install size, and the post now says so before a reader has to
work it out and post about it.
- Added one line on the language mix. Readers took the C++/Python/Go tree as
incoherence rather than as a Go core with per-ecosystem backends.
Cut throughout: the ledger metaphor ("what those ports buy", "not paid for in
throughput"), unearned framing ("the honest reading is", "has nothing to do
with"), the shape summary ("that is the general shape of these wins"),
confident deference ("people who are better at those models than we are"),
self-grading numbers ("a good result for a 66 MiB binary"), verbless
comparisons, three of the four exactness idioms, and the aphoristic headings
and verdicts. The double-tricolon summary is one plain clause now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* blog, website: same anti-slop sweep over the rest of the site
One-by-one pass over the other four posts and the site templates, with the
same rules used on the engines post. All figures, tables, links and PR
numbers are unchanged everywhere; the edits are to prose only.
apex-moe-quantization: ledger metaphors were the main issue, eight uses of
buy/cost/pay/spend for things that are not money. Also "the honest reading
is", "that is the comparison that matters", and two section-ending aphorisms
("Size is a speed knob as much as a memory knob", "Q6_K is the ceiling worth
paying for").
localai-since-march-2023: light touch, this one already reads like a person.
Removed "the curve is not the point", a "not the feature list, but the four
decisions" contrast, and two "X is what made / is the piece that" forms.
parakeet-cpp-asr-on-cpu: six exactness idioms across one post, "byte for
byte" twice, "character for character" twice, "byte-identical" twice and
"bit-identical" once, including in the title. Down to one, kept where the
precision is load-bearing. Also the "what end-of-utterance detection buys
you" heading and the "we say so rather than averaging it away" flex.
what-landed-in-localai-4-8: no changes. It is dense, flat and ends every
section on a PR number or a plain fact, which is the shape the other posts
should look like.
Site templates: "Most backends wrap somebody else's engine. These do not."
was the same contrast the engines post opened with. Also "Not a degraded mode
that technically runs", "A port only ships once it matches the original",
"Speed is the part we then go and win ... not a marketing run", and the last
"byte for byte" on the landing page.
Hugo builds clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* website: it is eighteen engines, not nineteen
Three places said nineteen: the /engines/ page description, the JUL 2026
timeline entry on the landing page, and the header comment in
data/engines.yaml.
Eighteen is right, confirmed two ways. The "Backends built by us" table in
the README has exactly 18 rows, and data/engines.yaml has 19 entries of which
one is apex-quant, which is a quantization recipe rather than an engine. The
two lists otherwise match name for name.
The yaml comment is the likely origin: it read "the nineteen native engines
the LocalAI team wrote, and the one quantization recipe that feeds them",
which counts apex-quant twice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
master does not compile:
vet: core/http/endpoints/mcp/localai_assistant_test.go:157:19:
method stubClient.ListScheduling already declared at
core/http/endpoints/mcp/localai_assistant_test.go:87:19
Two fixes for the same breakage landed. The four Scheduling methods were
already present at lines 87-99, in interface order after ListNodes, by
the time #11318 merged; #11318 appended its own copy after
GetRouterDecisions. The two blocks sit in different parts of the file, so
git merged both without a conflict and nothing flagged it.
Remove the appended copy and keep the one in interface order. Pure
deletion, no behaviour change.
Verified: go vet clean on ./core/http/endpoints/mcp/, and
go test ./core/http/endpoints/mcp/ passes.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The News page kept a hand-maintained "Highlights" list that had drifted:
it was missing all of 2025, duplicated the README's own news list, and
linked /features/middleware/ for a page that lives at operations/.
Both of its jobs already have owners. website/content/blog/ carries the
release write-ups and engineering notes, and GitHub Releases carries the
full changelog. Replace the list with a pointer at those two, so there is
one place to update instead of three.
The page keeps its url and front matter, so /docs/basics/news/ and the
root /basics/news/ redirect that .github/ci/gen-redirects.sh generates
both keep resolving.
Also drop the two contributor instructions in .agents that told authors
to add a whats-new.md bullet per feature: announcing a capability is the
release blog post's job, per .agents/preparing-a-release.md.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Write] [Bash]
Deploy site to GitHub Pages failed on five of the last eight master
pushes, always in the build job before Hugo runs:
Setup go version spec 1.22
...
go: downloading go1.26.0 (linux/amd64)
go: download go1.26.0: golang.org/toolchain@v0.0.1-go1.26.0.linux-amd64:
Get "https://proxy.golang.org/...": connect: network is unreachable
##[error]Command failed: go env GOPATH
The workflow pinned setup-go to 1.22 while go.mod declares go 1.26.0, so
the `go run ./.github/ci/modelslist.go` step that generates the gallery
page had to fetch the real toolchain from proxy.golang.org first. That
fetch is not reliably reachable from the runner, which is why the deploy
alternated between passing and failing rather than failing outright.
Track go.mod instead of a literal. The version the module needs is then
installed directly and there is no toolchain download to fail.
This matters beyond CI noise: the docs and the site, including the
release blog post, ship through this workflow.
Scoped deliberately to gh-pages, the workflow with the observed failure.
test-extra.yml pins 1.25.4 in a dozen places and is below go.mod for the
same reason, so those jobs also download a toolchain, but they are
currently green and rewriting twelve pins on a hunch risks more than it
fixes. Worth a follow-up.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
#11228 added ListScheduling, GetScheduling, SetScheduling and
DeleteScheduling to localaitools.LocalAIClient but did not update
stubClient, the hand-written test double in the mcp endpoints package.
The package therefore fails to typecheck, which takes out both lint and
tests on master:
cannot use stubClient{} as localaitools.LocalAIClient value in
argument to h.Initialize: stubClient does not implement
localaitools.LocalAIClient (missing method DeleteScheduling)
Red on 8f74f74b, fd4ec083 and 8a68f357; green on cd62e8ff, the commit
before.
Add the four methods with the same inert bodies the rest of the stub
uses. The real implementations are covered in the localaitools suites;
this double only exists so the holder can be constructed.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
chrome-audit.spec.js walks 25 routes in a single test, and has been the
UI E2E suite's failure on 5 of the last 6 master runs. It always dies the
same way, at the 30s per-test default:
Test timeout of 30000ms exceeded.
Error: page.waitForTimeout: Test timeout of 30000ms exceeded.
19 | await page.goto(route)
> 20 | await page.waitForTimeout(400)
The spec is new in 5cb0c1a8; the commit before it was green, and every
run since has been red on this file.
The failure is cumulative rather than one bad route. Across those runs
the clock runs out at line 19, 20 or 21 depending on where the loop
happens to be, and the timeout lands on waitForTimeout rather than on
goto, which is what running out of budget looks like as opposed to a
navigation that hangs. 30s over 25 routes is ~1.2s each, including a
deliberate 400ms settle, so there is very little headroom to begin with.
Give the test a budget proportional to its work: six seconds a route.
That absorbs a slow runner and still fails promptly if a route genuinely
hangs.
Verified: the spec passes on the current UI in 12.2s solo, and the full
suite passes 418 at 8 workers locally. What I could NOT do is reproduce
the CI timeout on this machine, which has 20 cores against the runner's
2 to 4; under synthetic CPU load it still finished in 13.5s. So the fix
is argued from the CI signature and the arithmetic, not from a local
repro, and the proof is this spec going green on the hosted runner.
Note test.setTimeout() has to be called inside the test body. At module
scope Playwright rejects it with "test.setTimeout() can only be called
from a test".
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Every CUDA sglang image failed in the 2026-08-02 full-matrix rebuild:
-gpu-nvidia-cuda-12-sglang, -gpu-nvidia-cuda-13-sglang and
-nvidia-l4t-cuda-13-arm64-sglang, all with the same build error.
Building cuda-tile==1.6.0rc3
x Failed to build `cuda-tile==1.6.0rc3`
ModuleNotFoundError: No module named 'wheel_stub'
hint: `cuda-tile` (v1.6.0rc3) was included because `sglang` (v0.5.16)
depends on `flashinfer-python` (v0.6.14) which depends on `cuda-tile`
This is the failure mode requirements-cublas1{2,3}-after.txt already
carries an nvidia-modelopt bound for, arriving through a different
package. install.sh passes a global --prerelease=allow, which is
load-bearing for flash-attn-4, so an unbounded dependency resolves to a
prerelease; cuda-tile 1.6.0rc3's build backend imports wheel_stub without
declaring it in build-system.requires; --no-build-isolation means nothing
provides it, and the build dies.
Nothing in this repo changed. cuda-tile published 1.6.0rc1 and rc3 and
the weekly cron picked them up, which is the drift that job exists to
catch.
Bound the one package rather than dropping the global flag, matching the
existing precedent. 1.5.0 is the newest stable release, so <1.6 takes the
last good one. l4t13 gets the same bound: it installs plain sglang rather
than sglang[all], but flashinfer-python is a dependency of both.
NOT VERIFIED LOCALLY: reproducing this needs a CUDA docker build, which
this machine cannot run. The diagnosis is from the CI log and the
resolver's own hint, and the change follows a fix already proven in these
same files. CI on this PR is the check that matters.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Since #11255 and #11276 every GPU image also builds ggml's CPU_ALL_VARIANTS
matrix, so a partial offload uses the host's SIMD kernels. That works
everywhere except SYCL, where the Makefile compiles the whole tree with
icpx -fsycl: icpx never finishes ggml-cpu/arch/x86/repack.cpp at
-march=sapphirerapids. In run 30765516644 both sycl_f16 and sycl_f32 stopped
at that translation unit and sat there for 5h30m with a single compile in
flight until GitHub killed the job at its 6h limit, and turboquant's f16 job
lost its runner outright. gcc compiles the same file in seconds in the vulkan
and CPU jobs of the same run, so the CPU variant matrix is only unbuildable
under icpx.
Route SYCL back to the portable fallback binary, which is what these images
shipped before #11255. run.sh already prefers *-cpu-all when present and falls
back otherwise, so nothing else has to change.
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(api): add text moderation endpoint
Add an OpenAI-compatible /v1/moderations endpoint backed by constrained local text generation. Register its auth and discovery surfaces, document the text-only MVP, and cover response shaping and access control.
Assisted-by: Codex:gpt-5
* test(mcp): update assistant client stub
Keep the LocalAI Assistant holder test stub aligned with the scheduling methods added to LocalAIClient so repository-wide type checking succeeds.\n\nAssisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(api): add POST /v1/images/upscale endpoint
Add a new image upscaling endpoint that accepts a source image and
returns an upscaled version. Supports selectable upscaler models
(e.g. realesrgan) and a configurable scale factor (2x or 4x).
- backend.proto: add UpscaleImage RPC and UpscaleImageRequest message
- pkg/grpc: implement UpscaleImage in Backend interface, client, server
and embed shim
- core/backend/upscale.go: new backend helper (mirrors ImageGeneration)
- core/http/endpoints/openai/upscale.go: new multipart/form-data handler
- core/http/routes/openai.go: register POST /v1/images/upscale
- core/http/auth/features.go: gate upscale routes under FeatureImages
- backend/python/diffusers/backend.py: implement UpscaleImage — uses
diffusers upscale pipeline when loaded, falls back to Lanczos resize
* fix(grpc): add UpscaleImage stub to Base backend
All Go backends embedding Base now satisfy the AIModel interface
without needing to implement UpscaleImage explicitly.
* fix(images): complete upscale endpoint integration
Store generated upscales under the served images directory, validate scale factors, document and advertise the endpoint, and add a functional Stable Diffusion x4 gallery model.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Give gallery operations distinct pause and cancel paths. Pause preserves partial download data so reinstalling the same model or backend resumes through HTTP Range, while cancel keeps its destructive semantics. Surface the action in the Activity UI and document the API behavior.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add multilingual 1B and 8B Q4_K_M GGUF embedding entries and link them as variants for automatic memory-aware selection.
Assisted-by: Codex:gpt-5 [web]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
fix(gallery): repair DeepSeek V4 0731 entry
Use the official single-file ggml-org MXFP4 artifact with its verified SHA256 and route it through llama.cpp instead of treating an unsloth repository page as a ds4 model file.
Assisted-by: Codex:gpt-5 [Hugging Face API]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add Q4_K_M and Q8_0 GGUF builds for the trending Instella-MoE-16B-A3B-Think model, with host-selectable variant metadata and verified Hugging Face LFS hashes.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(ui): give the Operate overview real numbers and traces a latency shape
First two items from a component-by-component comparison against the mocks.
The pattern that audit found: everything newly built matched, everything
pre-existing got the palette but not the layout, and an "absent rather than
empty" rule hid most of the overview exactly when someone was looking at an
idle installation.
**The headline grid is always rendered**, including at zero, with a fourth cell
for host memory. Hiding it removed the page's structure precisely when it was
most likely to be read, and "0 failed" is information — an absent panel is not.
The quiet case is now said in a line underneath instead of by showing nothing.
**The sections state counts** rather than listing their destinations: backends,
models, updates and running operations instead of the words "Usage and traces".
That needed installed backend and model counts in the summary context, which
are two more cheap reads on the poll that was already running.
**Traces rows carry latency as a bar as well as a figure**, scaled against the
slowest request currently in view and turning amber past two seconds. The table
had no latency column at all — the number was buried in the expanded detail, so
the shape of the tail was invisible while scanning. Scaling against the view
rather than an absolute ceiling is deliberate: what matters when reading a page
of traces is which of these are the outliers, and an absolute scale flattens
every row on a fast installation into nothing.
Full e2e suite: 409 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): name the engine on Home's resident models, and add jump-back-in
Third item from the mock comparison.
The mock showed each resident model with the engine serving it. /system carried
only the id, so the audit recorded this as blocked on a server field — but the
config loader is already in scope where that response is built, so it is one
lookup. SysInfoModel gains an optional `backend`, resolved from the model's
config and omitted rather than guessed when there is none (a loose file, or a
config since removed). Home renders the column blank in that case; the test
pins both halves of that.
Memory per model stays out. It is not one lookup — it would mean asking each
backend process — and inventing a number beside a real one is worse than
leaving the column off.
"Jump back in" is the block the mock had and Home did not. The quick-links row
above it is a set of first-run actions; these are the three places someone
returns to, each stated with what it currently holds rather than as a bare
label.
Go: routes suite passes. Full e2e suite: 412 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): rank the recommended models as lanes instead of equal cards
The hardware recommendations were a grid of equally-weighted cards. The list is
already sorted by fit, and a grid throws that order away: three cards side by
side say "pick one", when the page has actually formed an opinion about which
one.
They are lanes now, read top to bottom in fit order, with the leader carrying
the single amber "Best fit" label and the rest marked "Also fits". One opinion
per page — the alternatives are alternatives, not runners-up each worth their
own colour, which is how a strip of coloured badges ends up meaning nothing.
Below 720px the size and VRAM columns drop and the lane keeps the name and the
install action, which are the two things a narrow screen needs.
The existing panel spec moves off .rec-models-item onto .lane rather than being
deleted; dismissal, collapse, keyboard operation and install all still pass
unchanged, and there is a new assertion that exactly one row is called out.
Full e2e suite: 413 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): drop capsule chips app-wide, and un-break the empty voice library
**Pills are gone.** A capsule radius reads as a tag floating on the surface,
which fights a system whose structure is hairlines and square corners — and
with chips on Discover, Host, Activity and the biometrics pages, "some pages
have pills" was the real inconsistency rather than any one page.
Sixteen selectors move to the small radius: filter buttons, tab pills, activity
and biometrics chips, file and count badges, the jump-to-latest control, the
nav badge. Round *buttons* keep their circle — .lightbox__nav and
.home-send-btn are circles, not capsules — as do every progress track, status
dot and avatar, which are round because they are round, not because they are
tags.
**The empty voice library was unusable.** `.voice-library-empty` sets
min-height: 430px, border: 0 and background: transparent — a description of the
empty PANEL — and it had been attached to the action instead. The create button
was therefore a 430px transparent box that pushed itself out of the panel and
could not be seen. Moved onto the container it describes, which now centres its
action rather than letting it fall off the bottom. Same class-mangling shape as
the Agents header fixed earlier.
Full e2e suite: 416 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): put Host's headline figures on the shared hairline strip
Host had shadowed, clickable StatCards above a page that already has a rail, a
pane and a tab bar — a second dashboard language on one screen, and a different
one again from the figures inside its own detail pane.
The Operate overview's figure grid is generalised into a shared `.stat-strip`
and Host adopts it, so the two pages read as one system: same cell, same figure
scale, same tone vocabulary, and the same hairline grid the split-view StatGrid
already uses. The cells stay clickable and still route into the tab and filter
they describe, because a count is worth more when it is also the way to the
thing counted.
Tone is spent only where the number means something — running and updates when
non-zero — since a strip where every cell is coloured has no emphasis left.
Two bugs made on the way, both now covered:
- The first version put `<button>` elements inside a `<dl>` with `<dt>`/`<dd>`
inside the buttons. Neither is valid, the browser re-parents both, and the
cells collapsed. These cells are a set of controls, so a plain container of
buttons is also the honest markup.
- Even correct, the strip rendered 2px tall: `.page--app` is a flex column
whose split view takes flex:1, so a child with no intrinsic minimum is shrunk
away. The old cards survived only because `.stat-card` carried
min-height:96px. The strip now declines to shrink, with a test pinning it.
The stat-card specs are retargeted rather than deleted: they were written to
guard a class collision on a page that no longer uses cards, so they now guard
the strip's labels and its height.
Full e2e suite: 417 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): make Backends notices an edge rather than a filled card
The install and upgrade banners were tinted cards with a full border. A filled
panel makes every notice shout at the weight of an error, which is how notices
stop being read — and Backends shows one on most visits, so it was shouting
routinely.
They are now a hairline with a coloured left edge, the same treatment the
Operate overview gives rows that want a decision, so "this needs you" looks the
same wherever it appears. Counts in the notice take the monospace tabular
figures the rest of the console uses.
Also drops the last inline style on the page, and refreshes the inline-style
baseline, which has read 624 against a real count since #11288 landed. The gate
exits 0 either way, so nothing was failing — but a baseline 86 above the truth
would have let that many inline styles back in unnoticed. Now at 538, which
tightens the ratchet rather than loosening it.
The spec creates the upgrade it asserts on rather than skipping when the mock
has no notice: a test that skips is a test that proves nothing.
Full e2e suite: 418 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): finish the mock parity list, and stop hiding the recommendations
The last two items from the audit, plus a correction.
**Discover's use-case shelf is lanes.** These are a list of ways in, read in
order; a grid of equal cards asks the reader to compare them, which is not the
choice on offer.
**The request panel reaches every generator.** Video, 3D, Sound and Audio FX
join Images and Speech, so each one teaches its own endpoint rather than two of
six doing it. Audio FX records the fields that shape the request rather than
the bytes, since its payload is multipart.
**Recommendations no longer collapse themselves.** They were folded away by
default once anything was installed. That is the page's one opinion about this
host, and an opinion hidden by default is one the reader never gets. Someone
who disagrees can still collapse it and that choice is remembered — the
difference is that we no longer make it for them. Three specs asserted the old
default and now assert the new one.
The use-case heading also sat a line's width from the text it introduces, so
the two read as one paragraph. It has air under it now, and the shelf is
separated from the recommendations above it.
Two tests removed rather than kept: a generator loop whose only real assertion
was `expect(endpoint.length).toBeGreaterThan(0)`, and an earlier card-gap guard
that could only skip. A test that cannot fail is worse than no test, because it
reads as coverage.
Full e2e suite: 418 passed, 4 skipped. Inline styles at baseline.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): make the Host figures legible and give the strip its spacing back
Three defects introduced by the Host redesign, all found by looking at the
running app rather than by the suite.
**The figures were invisible.** "Running now" and "Updates available" rendered
pure black on the dark ground. Two causes compounding: the `--muted` tone alias
never landed, because the source rule has extra spaces before its brace and the
exact-match edit missed it silently; and a `<button>` does not inherit colour,
so with no tone rule the value fell back to the user agent's `buttontext`.
Both fixed, and a test now fails on any figure computing to pure black.
**The strip sat flush against the resources panel.** `.stat-strip` declares
`margin: 0 0 ...` and is declared later in the file than `.manage-summary`, so
the shorthand quietly won and the top margin became zero. Raised to
`.stat-strip.manage-summary` so it beats the shorthand on specificity rather
than on declaration order, which is the kind of thing that breaks again the
next time a rule moves.
**Discover's use-case heading had a doubled gap.** `.zero-pane` is a flex
column that already separates its children; adding a margin on top of the gap
stacked the two. The margin is gone and the heading keeps only its own breathing
room.
Full e2e suite: 420 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): make Studio's tabs path segments rather than a query parameter
`/app/studio?tab=images` reads like a filter applied to a page. It is
navigation: a different generator, with its own state and its own deep link. It
is now `/app/studio/images`, with the overview at `/app/studio`.
Legacy `?tab=` links are redirected once to the path form, replacing the
history entry so Back does not bounce between two spellings of the same place.
Bookmarks and older links keep working and land on the canonical URL rather
than a second version of it, which is the part worth having a test for.
The nine `?tab=` references were all in specs, none in docs, so the migration
is contained. They move to paths, and a new spec pins the redirect.
Full e2e suite: 421 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): make the hardware recommendation a section, not a dismissable card
It was a bordered card with a collapse control and a close button, sitting
inside a pane that is otherwise hairline sections. Two problems: it read as
something bolted onto the page rather than part of it, and treating it as an
interruption to be shut is the wrong frame for the one thing the page has to
say about the machine it is running on.
It is now a plain section with the same heading treatment as the shelves below
it. The collapse state, the dismissal, their storage keys and the legacy key
read for backwards compatibility all go with it, along with the installedCount
prop that existed only to pick a default collapse.
Five specs described behaviour that no longer exists and are removed rather
than adjusted — collapsing, dismissing, persistence of both, and the toggle's
keyboard handling. One new spec asserts the replacement contract: no control
with aria-expanded, no dismiss, and no card border.
Full e2e suite: 416 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): restore every stripped icon and every default-chrome control
You reported two broken icons. They were not two: an earlier automated edit had
stripped the `fa-*` class from twenty `<i>` elements across eleven pages, and an
`<i>` with no icon class renders nothing at all. Settings' save button, the
voice-profile back link, and eighteen others — agent row actions, task and job
buttons, import and create actions — were all drawing empty space.
Each is restored from its own context rather than a blanket icon: the agent row
gets pause/play, pen, comments, file-export and trash; the fine-tune toggle
swaps plus for xmark as it opens; the P2P documentation link gets the
external-link glyph.
The same edit left controls without their classes. Fine-tune's "Import config"
was rendering in the browser's own chrome, and `.p2p-cmd__copy` set a border
but no background, so it fell back to `buttonface` — a pale grey chip on a dark
command block. FineTune's "New job" also had its icon classes folded into the
button's className, the same mangling already fixed on the Agents header.
Rather than fix the reported two and wait for the next report, this adds a
standing audit: twenty-five routes are walked and the test fails on any visible
control rendering with user-agent chrome, or any `<i>` without an `fa-*` class.
It found the three remaining cases after the first sweep, and it is the reason
the next one cannot ship quietly.
Full e2e suite: 417 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): give Operate a front door and fold six rail groups into four
Opening Operate ran firstVisiblePath() and landed on Backends, because
Backends happens to be written first in operateConsole.groups. The section
that should answer "is anything wrong" opened on a package manager, and
nothing was reported until you visited it.
Adds /app/operate. Its one irreplaceable block is "Needs attention", which
is empty when nothing is wrong and says so in a line rather than rendering a
reassuring green panel. It collects stale backends, failed operations and
unhealthy nodes. Everything else on the page is a summary you could already
assemble by visiting four others.
The rail regroups from six headings to four: Inference and Activity were both
"the runtime right now", Access and System were both administration. No
destination is removed and no gate changes, so isConsoleItemVisible and
consolePaths are untouched. Overview leads the first group, which is what
makes firstVisiblePath() return it without knowing it exists.
Rail entries now carry a signal beside the label. This does not replace the
sidebar badge and is not built as if it does: the badge stays on the
always-visible sidebar entry for the reason recorded in Sidebar.jsx, that the
rail exists only on Operate routes and can be collapsed. The signals are
orientation while inside Operate, so they are aria-hidden and nothing urgent
depends on them alone.
OperateSummaryContext polls once for the whole console, following
OperationsContext, which exists because per-consumer setInterval against one
endpoint was the defect it fixed. It is mounted by ConsoleLayout for the
Operate console only, so "poll only while in Operate" needs no route check.
Built on usePolling, so it pauses on a hidden tab. Operations are read from
OperationsContext rather than polled a second time, and each source degrades
to no-signal on its own so one dead endpoint cannot blank the rest. It reads
the cached GET /api/backends/upgrades and never the POST that forces a real
registry check.
Traces and Usage get no signal yet: /api/traces returns the list, so a count
would mean fetching every trace to render one number. A counts endpoint is
the honest fix and is scoped separately.
Full e2e suite green (369 passed, 4 skipped), including a render-smoke entry
for the new route.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): open Studio on what this machine can actually make
Studio was a tab strip over six generators that opened on Images, which was
never a decision, only the first entry in BASE_TABS. Nothing said which
modalities this installation could run, so the way to learn that video had no
model was to pick the tab and find an empty select.
Adds an overview tab and makes it the fallback. Explicit tabs still win, so
existing deep links keep working; anything unrecognised or gated now lands on
the overview rather than Images.
Each tab carries a dot: filled when an installed model advertises that
modality, hollow when nothing serves it. That is the feature in one detail,
turning the strip from navigation into a report of what the machine can do
before anything is clicked. The dot is aria-hidden because the overview states
the same facts in words and the dots change as models load.
Two kinds of unavailable, which had to stop looking alike:
- switched off, via a permission: no tab and no lane, unchanged
- available with no model: a lane, and a route to installing one
Studio now owns one MODALITIES table so the tab strip and the overview cannot
disagree about what exists, and calls useModels() once, unfiltered, grouping in
the browser. useModels(capability) fetches the whole list and filters locally,
so a hook per modality would have been six identical requests to
/api/models/capabilities on every mount. There is a test for that.
Recent outputs read every localStorage store through a new
readAllMediaHistory(), which avoids mounting five hooks that carry save timers
the overview has no use for. 3D is read separately through use3DHistory rather
than folded in: its entries are GLB blobs in IndexedDB, so they cannot come
from the same synchronous read.
Typical cost is the median of this machine's own history, not a guess, and
renders as a dash when there is nothing to go on.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): stop the stat cards and the console rail breaking on small screens
Two unrelated causes behind one report that /app/manage looks wrong when the
window is narrow.
The stat cards were being laid out by the wrong rule at every width. Two
different components both claimed `.stat-grid`: the dashboard card strip that
holds .stat-card children, and the detail-pane StatGrid the split views
introduced further down App.css. Being later, the second won every shared
property, so the cards got its 120px columns and its 1px hairline gap in place
of their own 180px columns and spacing-md. Four cards were packed onto a row
that fits two, labels wrapped to three lines and clipped, and the icon crowded
the value. Renamed the strip to `.stat-cards`, after the children it actually
holds, which also removes the mismatch of a `.stat-grid` container full of
`.stat-card`s. The split-view component keeps `.stat-grid` and its BEM parts.
The expanded console rail had no bounded height. Thirteen destinations stacked
in one column is taller than a phone, so opening the menu pushed the page's own
heading past the fold: the menu replaced the page rather than annotating it.
Capped at 55vh with internal scrolling below 768px, so the content behind stays
reachable.
Both are asserted on behaviour rather than markup: no stat-card label may be
clipped, the card gap must not be the detail pane's hairline, and expanding the
rail must leave the page heading on screen.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): retemper the palette to localai.io and add the lane primitive
The token half of the style transfer, plus the shared list idiom the two
overviews had each grown their own copy of.
theme.css moves from Nord to the website's palette, variable names preserved so
every consumer moves with it: ground #13171f -> #0d1117, accent frost cyan
#88c0d0 -> action blue #4f8cff, success sage -> mint #56d6a4, warning -> the
amber #f1b95d the site spends only on the thing asking for a decision. Eyebrows
go mint. Dividers become an opaque #29384a hairline rather than alpha over a
varying surface, which is what makes stacked surfaces read crisply on the site.
Light is derived, not inverted. The site ships one theme and never had to
answer this, but the app does: blue darkens to #2f62d8, mint to #0d8b60 and
amber to #8a5d0b, all clearing 4.5:1 on a cool paper ground, where the
dark-mode values sit near 2:1. Same three roles, different values.
Three files restate the palette because CSS variables cannot reach them:
cmTheme.js (the whole CodeMirror theme), VoiceVisualizer and WaveformPlayer
(canvas). Left alone they would have quietly kept the app half-Nord.
The `.lane` primitive replaces the near-identical row CSS that OperateOverview
and StudioOverview had each written: a full-bleed row on a hairline that insets
on hover, with no card and no shadow. Callers supply only the column template.
Both pages now use it, along with `.lane-head` for section rhythm and a
`.page-pad` container for top-level pages outside a console shell — without
which Studio sat flush against the sidebar with its eyebrow clipped.
Studio's tab strip wraps rather than running off the edge at narrow widths.
Full e2e suite: 386 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): put Home's resident models on lanes and give the footer one line
Home's status line was three chips saying a thing was true. It now reports
figures: how many models are resident, how many nodes are healthy, what share
of memory is in use, set in tabular monospace so the digits line up. A chip
answers whether; a figure answers how much, which is what someone opening the
page at a glance is after.
Resident models move from status chips to lanes, with the id set in a new
`.lane__name--id` because an id is something you might type or paste and the UI
face makes it read as a label. /api/system-information carries only the id, so
there is deliberately no backend or memory column: inventing one would mean a
server change this does not make.
The footer was three centred rows and cost the bottom sixth of every page for
chrome. It is one line now, version left and links right, wrapping to centred
when the viewport is too narrow to hold both. Every link it had, it keeps.
Full e2e suite: 392 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): correct three contrast failures and stop a guaranteed-404 poll
A contrast audit of the new palette found three values below WCAG AA, one of
which the previous commit message claimed was fine:
- White on the #4f8cff button is 3.22:1, which is large-text only. The website
does exactly this, but a button label in an app is not large text, so the
label goes to dark ink at 5.88:1. Light mode keeps white, which is 5.44:1 on
its darker blue.
- Light-mode success was 4.08:1 on paper, not the 4.5 claimed. Darkened to
#0a734f, 5.56:1.
- Nord red was already 4.28:1 on raised surfaces, a pre-existing miss carried
over unexamined. Lifted to #c96f78, 5.02:1.
Lanes gain the two states they were missing: a 44px target on coarse pointers,
matching what EntityRail already does so the two list idioms feel the same
under a thumb, and a reduced-motion variant that keeps the background feedback
while dropping the hover inset, which is a position change.
The Operate summary no longer asks for /api/nodes on a single-node install. The
cluster API answers 503 when distributed mode is off, so it was a guaranteed
miss every fifteen seconds; it is now gated on useDistributedMode, the same
condition the rail already uses for the Nodes entry.
Full e2e suite: 392 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): restore the gap between overview blocks, and stop claiming zero nodes
Two defects a design review surfaced.
`.lane-head:first-child { margin-top: 0 }` was meant to stop the first block on
a page carrying a top margin. But every <section> makes its lane-head a first
child, so the reset applied to all of them and the gap between blocks vanished:
"Sections" sat flush against the attention row above it. The header supplies its
own bottom margin, so a uniform top margin is correct everywhere.
The Cluster summary read "0 nodes" on a single-node install, which looks like a
fault when the cluster API is simply switched off. It now says "Single node".
Full e2e suite: 392 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): open dark by default, and stop clipping the collapsed sidebar footer
Dark is the identity rather than a preference: localai.io ships one theme and
it is this one, so an install should look like LocalAI before anyone has chosen
anything. The OS setting no longer selects light on first load. The toggle
still does, and a stored choice wins forever after, which the tests assert
both ways.
The collapsed sidebar footer stacked its controls but kept the expanded row's
inline padding, so their edges were clipped against the 51px rail.
Full e2e suite: 394 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(api): count traces server-side and give the Operate overview real totals
The overview's headline block had no source. /api/traces returns the trace
list, so "37 errors in 24h" meant fetching every buffered exchange to count it
in the browser — waste that grows with the buffer, to produce three integers.
Adds GET /api/traces/summary: totals, failures, p95 and a bucketed series for
sparklines, over a window that defaults to 24 hours and is capped at a week.
Deliberate calls, each with a spec:
- A 4xx is the caller getting it wrong, not the installation being unhealthy,
so only 5xx and transport errors count as failures.
- p95 is a nearest-rank percentile rather than the slowest request, which is
what a max would report and what makes latency panels lie.
- Buckets are oldest-first so a sparkline reads left to right, and the slice is
never nil: nil serialises as null and breaks .map() on the other side, which
is a silent runtime error rather than an empty chart.
- Exchanges outside the window are not counted at all.
The route is registered before /api/traces/:id so "summary" is not captured as
a trace ID.
On the client, Traces and Usage gain the rail signals they were shipped
without, the Observability section summary now states counts instead of listing
its destinations, and an installation that has served nothing says so rather
than showing three zeroes dressed as telemetry.
Sparkline is a bare stroke with an emphasised endpoint and no axes: the figure
above it already states the value, so its only job is the shape.
Go: 185 middleware specs pass. Full e2e suite: 396 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): stop the memory chart calling a trade-off an error
The VRAM-by-context chart rendered any build over the limit in error red, and
escalated the verdict to the error tone as soon as two context sizes crossed
it. But an over-limit build still installs — #11288 keeps a test on exactly
that — so red overstates what is happening. A model that fits at 32k and not
64k is a trade-off, not a fault.
Over-limit bars and the limit line now use the warning tone, which is the
constraint colour used everywhere else in this branch: know what you are doing,
not you may not. The error tone is reserved for "fits nowhere", where the model
genuinely cannot run on this host.
Full e2e suite: 397 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): give the new surfaces orchestrated motion
Uses the reveal system already in the codebase rather than adding a library:
pageReveal, .reveal-stagger and staggerStyle() were built for exactly this, and
anime.js would be ~17KB duplicating four lines of CSS for list reveals.
The overview's headline figures, attention rows and section lanes stagger in,
as do Studio's modality lanes and recent outputs, so a page assembles in the
order it is read instead of appearing all at once.
Two additions beyond stagger. Rail signals transition on opacity when a poll
lands, so a number changing reads as an update rather than a jump cut, and it
stays on the compositor so it cannot reflow the rail. The attention block
animates its left edge in — the one thing on the page that should announce
itself, and on the border rather than the text so nothing moves under a reader.
Both are dropped entirely under prefers-reduced-motion, alongside the lane
hover inset already handled.
Full e2e suite: 397 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): put the generators on a hairline field stack and record the request
The workbench treatment from the mocks, applied where it costs least: both
changes land on shared surfaces, so all six generators get them at once rather
than drifting apart page by page.
The control column stops being a shadowed card of boxed groups and becomes a
hairline field stack — the panel is the page's left half, not an object
floating on it — with uppercase micro-labels matching the eyebrow treatment
used elsewhere. Because .media-controls is shared, Images, Video, 3D, Speech,
Sound and Audio FX all move together.
RequestPanel shows the request the form actually built, with a copy-as-curl.
LocalAI is API-first and Studio is the best place in the app to teach its own
endpoints: the form stops being a black box, and a result worth keeping can be
reproduced from a shell without reverse-engineering which fields the page sent.
It records what was sent rather than what the form currently holds, and renders
nothing until a request has been made — a panel describing a request nobody
made is a tutorial, not a record. Wired into Images and Speech.
Full e2e suite: 401 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): make Chat a transcript instead of a bubble thread
Rounded, filled, asymmetric bubbles fight a system built on hairlines, and they
carry the speaker in shape and side rather than in words. The assistant side
had already given up its bubble; this finishes the job.
Both roles now run full width down one column, separated by a rule, each with a
mono role label. The user turn keeps a left edge in the action tone so the two
are still told apart at a glance, without a fill or a corner radius. The
avatars go: the accent and the label carry the speaker, so the glyph was
decoration once neither side had a bubble.
Saying who is speaking in words rather than in geometry is also what survives
being read aloud, printed, or looked at by someone who cannot pick the sides
apart by colour.
Full e2e suite: 404 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* feat(ui): dress the API reference in LocalAI's palette
The Swagger page was the last surface still shipping in someone else's colours,
which is conspicuous now that everything it links from is dark.
Swagger UI has no theming hook, so rather than fork it we serve our own index
ahead of the library's wildcard and restate the palette over its stylesheet.
The library's own bundle and assets are still what load, so a swagger-ui
upgrade cannot silently break the page — this is a skin, not a fork.
Two things needed real care. Swagger tints the entire operation row per method
via .opblock.opblock-post and friends, so the palette had to match that
specificity rather than reach for !important; the method now lives on one edge
instead of washing across the row, because a page where every row is a status
colour has no status colour left. And the filled method chip put white on pale
green, which was the least readable thing on the page — it is an outlined mono
chip now, carrying the method in its border and text.
Palette values are copied from theme.css rather than referenced: this page is
served by Go and never sees the app's CSS. The comment says so, and says to
keep them in step.
Go: routes and middleware suites pass. Full e2e suite: 405 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
* fix(ui): make tall split-view pages reachable, repair the Agents header, scale titles
Three things found by actually using the app rather than measuring it.
**Host was unusable.** The shell above a split view is overflow:hidden so the
document cannot grow, which left anything taller than the viewport simply
unreachable — and Host stacks a resources card, four stat cards and a tab bar
above its split, so the bottom of the pane fell off at every window height with
nothing to scroll. Every sweep I ran for this was horizontal, which is why it
kept coming back clean.
The page now scrolls inside the pinned shell. The pane keeps its own scroller:
letting it grow instead pushes the document taller and stretches the rail to
match, which is the regression e2e/discover-height.spec.js exists to catch, and
which the first version of this fix duly caused.
**The Agents header controls were unstyled** — "Create Agent" was rendering
with the browser's default chrome. The markup had been mangled at some point:
six unrelated classes merged into one string on the link, and the label and
button left with none at all and empty icons. Repaired, with the inline flex
replaced by a shared .header-actions class.
**Page titles take the editorial scale from the site**: larger, tracked at
-0.04em, on a line height near 1, so a two-word title reads as a statement
rather than a label. The typeface is unchanged — DESIGN.md keeps the existing
type system — so the whole difference is scale, tracking and leading, which is
where the site gets its voice from. This was the biggest reason the running app
still did not look like the mocks.
Full e2e suite: 404 passed, 4 skipped.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Follow-up to #11288, which warmed the VRAM estimate caches at startup and left
the variant picker paying its own way.
Describing an entry's variants probes the weight files of every build it
offers, so the first time a model is opened costs 1.2-1.9s against a cold
cache. That is the same cost as an estimate wearing a different hat, and it
lands in the same caches underneath, so it belongs in the same pass rather than
in a second mechanism.
The warm-up now describes variants for the entries it walks. Entries that
declare none cost nothing: the call is gated on HasVariants rather than
attempted and discarded. The host resolve env is derived once for the run,
since it describes the machine rather than the entry.
Failure handling matches the estimate half. An entry whose variants cannot be
described is logged at debug and skipped, and the estimate for that same entry
is unaffected, because neither half is allowed to fail the other.
Measured against a live instance with 1,595 models, first ever call to
/api/models/variants/:id after a cold boot:
before 1.2-1.9s
after 2ms
The warm-up's own cost barely moves: 3m0s to 3m19s for 300 entries, of which
40 declared variants. It stays bounded by the same knobs, and
LOCALAI_VRAM_WARM_LIMIT=0 still turns the whole thing off.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): rename the Install Models nav entry to Discover
"Install Models" named the action rather than the destination, and it was
the only multi-word entry in a rail of one-word ones (Home, Chat, Studio,
Talk, Build, Operate). A bare "Models" was the obvious fix but it collides
with the installed-models view under Host, which is a different page for a
different job.
"Discover" keeps the rhythm and says what the page is for. The icon moves
from a download arrow to a compass for the same reason: the page is browsed
before it is installed from.
Translated in all seven locales rather than left to fall back, so a locale
switch does not leave the entry in English.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(ui): replace the gallery table with a rail and a detail pane
The eight-column table was not the real problem; the click-to-expand row
underneath it was. Variants, files and a VRAM estimate never fitted inside a
<tr>, so they were pushed into a drawer that could hold one model at a time,
could not be linked to, and had no room to say anything useful.
The gallery is now a rail to scan and a pane that answers. The pane has two
states and no third: with nothing selected it is the discovery page, and with
a model selected it is that model's detail. Selection lives in the URL, so a
model is linkable and Back steps out of the detail instead of off the page.
The rail groups by capability while browsing and flattens to results the
moment a term is typed. That is a rule rather than a toggle: once someone has
said what they are looking for, the buckets are between them and the answer,
and making the user choose would be handing them our problem.
The detail pane plots VRAM against context length with the host's own limit
drawn across it. This is new information, not a restyle. A single number
invites "so will it run?", and the honest answer is usually "yes, up to a 32k
context", which is a shape rather than a number. The estimates were already
fetched for every context size, so it costs no new request. Backends that
take no context length say so instead of being given a meaningless chart, and
a host with no GPU gets no chart at all rather than bars with nothing to
compare against.
The split-button variant menu goes with the actions column. The pane lists
every build with its backend, quantization, size, fit and a details
disclosure, each installable, which is what the dropdown was a cramped
substitute for. Its tests move onto that list; the three contracts it alone
carried (fetch-once caching, the loading state, an unfit build staying
installable) are backfilled against the pane.
RecommendedModels moves inside the pane, where it has the width to argue for
a model instead of listing one, and keeps its own dismissal and collapse.
Rail entries carry no description. Two lines is the budget and the second is
better spent on whether the thing will run; the stripped-Markdown contract
moves to the pane's lede, tooltip included.
e2e: 123 passing across models-gallery, navigation, recommended-panel,
model-artifact-operation, operations-strip and page-render-smoke. Inline
styles in Models.jsx drop from 82 to 41.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* refactor(ui): extract the split view into shared components
Discover shipped its rail, pane and detail header as private functions inside
Models.jsx. Backends and Host have the same defect and want the same shape, so
leaving them there guarantees three rails that drift.
SplitView, EntityRail, DetailHeader and StatGrid now live under
components/split/. EntityRail is deliberately data-driven: a surface maps its
own entity onto { id, name, icon, meta, stripe, groupId } and keeps its
vocabulary to itself, which is what stops the rail learning about models,
backends and loaded state all at once.
The CSS moves with it. What was .discover__rail is .entity-rail, .discover__
pane is .split-view__pane and so on, because a class named after one page is a
lie on the next two. Only what is genuinely Discover's stays behind the old
prefix: the shelves, the hero and the VRAM-by-context chart.
Two additions the shared rail needs and Discover did not: a state stripe, for
surfaces read by condition before they are read by name, and an empty label.
Discover passes neither.
No behaviour change. e2e 100 passing across models-gallery, navigation and
models-recommended-panel.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(ui): put the backend gallery on the split view
Same defect as the model gallery, so the same shape: a seven-column table over
a click-to-expand row that was the only place the repository, licence, tags and
links could go.
The rail groups backends by the use case they serve, sharing Discover's
taxonomy on purpose: a backend is the runtime a use case needs, so "vision"
ought to mean the same thing one level down. It flattens on a query for the
same reason it does on Discover.
The zero state is the one real departure. A backend's fitness is not free
memory, it is the accelerator and platform it was built for, so the pane leads
with what this host is, then what is not installed yet, then whether anything
installed has gone stale. The table listed 37 runtimes and left "which of these
can even run here" entirely to the reader.
Distribution moves into the pane, which is the one thing a row could never
carry: which nodes hold a copy and which do not, with the install-on-more
control next to it rather than squeezed against a chip.
The distributed and target-node action logic is unchanged, including the guard
that keeps a hardware-specific build off the fan-out path. The split-button
popover loses its per-row anchoring because there are no rows; one pane, one
anchor.
Selection lives in ?backend=, preserving the ?target= scope rather than
clobbering it.
e2e: 139 passing across models-gallery, navigation, backends-management,
models-recommended-panel, nodes-per-node-backend-actions, page-render-smoke,
operations-strip and model-artifact-operation. The backends spec gains six
split-view tests; its three description-cell tests move onto the pane lede.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(ui): put the Host inventory on the split view
The last of the three surfaces, and the one that is not a catalog. Both tabs
had the same click-to-expand row, so the shell transfers; what does not
transfer is the zero state, because there is nothing to discover in your own
inventory.
With nothing selected the pane reports what is happening: how many models are
loaded, what failed, what has an update, and which models are holding VRAM
right now. Every number was already on the page. None of them had been
assembled into one statement, so "what is going on" was a question the tabs
could not answer however long you looked at them.
The rail buckets by state rather than capability - Running, Idle, Disabled for
models; Update available, Installed for backends - which is the opposite of the
galleries and deliberately so: nobody opens Host wondering which of their
models does vision. Entries carry a state stripe for the same reason.
Load and Stop are promoted out of the kebab, because that is what an operator
came for; the rest stays behind the menu rather than diluting it. Adopted,
pinned and alias badges follow the model into the pane: they are facts about
the thing, not about its state, and the rail line is spent on state.
Deliberately NOT done: folding the two tabs into one rail, as the mock had it.
It costs five URL parameters, the manage-tab localStorage key and the
stat-card shortcuts, all of which are live deep-links today. The tabs stay as
the group selector; merging them is a follow-up with its own migration.
e2e: full suite 355 passing. New host-split-view spec; alias-template,
manage-logs-link, manage-action-menu-position and model-editor-back-nav move
off `.table` and the row kebab onto the rail and the pane.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* polish(ui): accessibility and consistency pass over the three split views
Findings from a pass over what the previous four commits actually shipped,
rather than what they were supposed to.
The rail was not a listbox. ARIA lets a listbox contain options and groups,
and nothing else, but each group's collapse control is a button that has to
sit inside the scroller with the entries it folds. It is now a labelled group
of buttons, which is the honest description; selection is announced with
aria-current and the arrow keys are unaffected.
Every entry was its own tab stop, so tabbing past a forty-entry rail to reach
the pane took forty keystrokes. Roving tabindex makes the rail one stop, and
arrowing now moves focus with the selection instead of leaving it behind on an
entry Tab can no longer reach.
The rail rounds its corners with overflow:hidden, which was clipping the focus
ring off the first and last entries entirely. Inset outlines fix it.
A 30px row is fine under a mouse and too small under a thumb, so coarse
pointers get a 44px target without costing density on a desktop.
One slot said three different things: "9 models loaded" on Discover, "12
loaded" on Backends, "3 of 9" on Host. All three lists are a page of a larger
set, so all three now say so the same way.
Also removed: an emptyLabel prop on EntityRail that nothing passed, its dead
CSS rule, and MODELS_COLSPAN and ResourceRowDesc, which died with the tables.
e2e: full suite 355 passing.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(ui): correct three defects only a real gallery exposed
Running the branch against a live instance with 1,595 models and 1,017
backends, rather than against mocked fixtures, surfaced three things the e2e
suite could not.
Grouping did nothing. The rails matched on the use-case keys the filter chips
send (`chat`, `tts`, `transcript`), but those are a server-side vocabulary the
handler maps onto entries. What entries actually carry is free-form and
inconsistent: models come back tagged `llm`, `gguf`, `vision`, `coding`, and
backends `LLM`, `text-to-text`, `audio-transcription`. Nothing matched, so
every model landed in "Everything else" and the feature was decorative.
Grouping now lives in utils/entityGroups.js, shared by both galleries, matching
case-insensitively against the vocabulary the API really uses, with the entry's
backend as a fallback signal - a backend named `whisper` is a speech backend
whatever its tags say. Order is specific before general and that is
load-bearing: a vision model is tagged `llm` too, so testing text first would
swallow it.
The zero state claimed GPU memory on a machine with no GPU. The resources
endpoint reports system RAM in the same field when gpu_count is 0, so the hero
read "84.4 GB of GPU memory" next to the recommendations panel correctly
saying "No GPU detected". The number was never wrong, only its label; it now
says system memory unless a GPU is actually present.
The page title still said "Install Models" under a nav entry saying Discover.
Also: the keyboard test named the model it expected to arrive at, which made it
a hostage of the grouping table and broke the moment the buckets were fixed. It
now asserts that the selection moves and returns.
e2e: full suite 355 passing.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(ui): the filters and the rail were fighting over the same job
Four things you find odd on Discover, and they turn out to be one mistake seen
from four sides.
The rail grouped the current page. The listing is paginated at nine rows, so
those bucket headers described nine entries out of 1,595, and turning a page
reshuffled the sections under the reader. The structure was never stable
because it was computed over the wrong set.
The chips were redundant for the same reason, seen from the other side. They
send tag= and filter all 1,595 server-side. The rail grouped nine of them
client-side by the same axis. Two controls for one job, and the weaker one was
the one this branch added, so it goes. Grouping stays only on Host, where the
list is complete, local, and bucketed by state rather than capability.
The search bar felt odd because it sat in a full-width band while the thing it
narrowed was a 290px rail below and to the left. The whole band now lives in
the rail column: search, backend, use cases, refinements, then the list it
narrows. One column to say what you want, one to show what you got. Nineteen
chips do not fit at that width, so they fold into a disclosure that states the
selection. A disclosure and not a popover, deliberately: picking use cases is
multi-select and interleaves with the backend select and the toggles below,
and a popover dismisses itself the moment you touch either.
The header held two counts and two buttons at arm's length from all of it. The
counts were the third statement of the same number on one screen, after the
rail's "9 of 1,247" and the pane's own headline, so they go. The buttons move
into the pane's zero state, which is the surface that answers "what do I do
here".
Also: the two first-run empty states wore .loading-center, which is
display:flex in the default row direction because it exists to centre one
spinner. With four children that put the icon, the heading, the sentence and
the buttons on a single line with no gap. They are now a proper full-height
empty state.
e2e: full suite 353 passing. Grouping tests are replaced by ones asserting the
rail stays flat; chip tests open the disclosure first; two filter-layout tests
that asserted the old three-band arrangement now assert the column.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* polish(ui): make Discover a full-height view, group the chips, name the refinements
Four things, all of them the same complaint: the page read as a document with
controls scattered on it rather than as one view.
The header is fused. A title block with its own padding, a subtitle and two
counts made the split view look like an attachment to a document that happened
to sit below it. It is now a slim bar carrying the title, the count and the two
page-level actions, and the split fills the rest of the window. Rail and pane
scroll independently, so the filters and the pane's headline stay put while a
long list moves under them.
The chips group. Nineteen in a flat row is a lot to scan even behind a
disclosure, and they already belong to the four families the rest of the UI
speaks, so they are bucketed by those. "All" sits on its own above them without
a heading, because it is a reset rather than a use case.
The refinements stop looking dumped. When the band became a column they were
three controls left where they landed; they now read as a named section with
one control per row.
The zero state suggests again. It had decayed into a "Browsing / 9 of 1,247 /
select a model" line that restated the count for the third time on one screen.
It now offers the four use cases as tiles that set the filter, which is the
shelf idea from the mock without inventing curation or paying for a second
fetch.
Two bugs found by looking at it rather than at the tests: the disclosure was
clamped to 190px, which cut it off partway through its third section so two of
the five never appeared at all; and the creation actions rendered twice, once
in the new bar and once in the pane hero a few pixels away.
e2e: full suite 353 passing. The chip-row test now holds its contract across
the per-family rows rather than a single one, and additionally asserts every
family is present and non-empty.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(ui): pin the split view's height so a long detail scrolls the pane
Selecting a model with a long description grew the whole page and dragged the
rail down with it, which is the opposite of what "full height" was supposed to
buy.
The flex chain was right and the ceiling was missing. .app-layout and
.main-content are min-height:100dvh, which is a floor: flex distributes free
space but nothing caps growth, so a pane taller than the viewport expanded the
column, the document scrolled, and the rail stretched to match. height:100% on
the pane then resolved against an auto-height parent and did nothing.
The chat route already solves this by pinning .main-content to 100dvh. The
same treatment now applies to any route containing a .page--app, selected with
:has() so the shell does not have to learn which pages happen to be split
views. Below the stacking breakpoint the pin is lifted, because two stacked
halves in two short scrollers is worse than a page that scrolls.
Measured on a live instance: document height stays at the viewport across
selection (950px either side) and the pane overflows internally instead.
Adds discover-height.spec.js, which asserts the page height and the rail height
are unchanged by selection and that the pane is the thing that scrolls. The
existing specs could not have caught this: they mock short descriptions, and
the bug only appears when the pane has more content than the viewport holds.
e2e: full suite 355 passing.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(ui): give Backends and Host the full-height view, and fix the Update button
Backends now matches Discover: the header fuses into a slim bar carrying the
title, the count and the page-level actions, the filters move into the rail
column where they narrow the rail and nothing else, and the split fills the
window. Its seven chips fit at rail width, so unlike Discover's nineteen they
need no disclosure. Host gets the bar and the height; its resource monitor,
summary cards and tabs stay above the split, because those are read once while
the rail and the pane are worked in.
Two things the height change surfaced.
The console layout is a flex row with align-items:flex-start, so its body sizes
to content. Right for the pages it was built for, wrong for a split view, which
needs a ceiling to scroll inside: without it the Backends rail ran past the
viewport and over the footer. Pinned with :has() so only split-view routes are
affected.
The filters vanished when nothing matched. Both galleries swapped the whole
shell for an empty state, which took the search box and the chips with it, so
the page said "try adjusting your search or filters" while offering neither.
The shell now stays and the empty state moves into the pane.
Also fixes the Update control on Host, which had no className at all and
rendered as bare text, next to a status span that had picked up btn classes and
two copies of `fas` and so rendered as a button you cannot press. They have
swapped appearances back.
e2e: full suite 355 passing. The render-smoke selector learns .view-bar__title,
since the pages it checks no longer all use PageHeader.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(ui): keep the view mounted while searching, and bring rail grouping back
Searching replaced the whole view with a loader. The search box lives in the
rail column, so every debounced refetch unmounted the field being typed into
and dropped its focus with it. The list, the filters and the pane went too.
The shell now stays and the rail says it is busy: a sweep bar under its header
and the stale list dimmed, so the eye knows the answer is being replaced
without losing its place. A cold start still gets the skeleton, because there
is nothing to keep.
The condition for that is "nothing has loaded yet", not "the list is empty".
Those differ exactly when someone is editing a query that matched nothing, and
getting it wrong there would unmount the view on the keystroke after a
no-results search - the worst possible moment.
Grouping comes back on both galleries. It was removed because nine rows could
not fill five buckets, so a page turn rebuilt the rail's whole structure. That
was a symptom of the page size rather than of grouping: the rail now asks for
30 rows instead of 9 (Backends 60 instead of 21), which is enough for the
sections to read as structure and turns five times fewer pages. The order of
the sections is fixed, so what changes between pages is membership, not
arrangement.
Grouped while browsing, flat while searching, as before: once a term is typed
the buckets stand between the reader and the answer.
Also gives GalleryLoader a class and a testid instead of six inline style
declarations on a bare div, which is why nothing could select it.
e2e: full suite 359 passing, including a new spec asserting the search box
keeps its focus and its value across a refetch, and that a cold start still
shows the skeleton.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* perf(gallery): stop invalidating the VRAM estimate caches on every request
Searching or turning a page felt slow. It was not the search and not the
listing: /api/models answers in 3-9ms. It was the VRAM estimate, which the
gallery asks for once per row, and which took ~2.3s every single time however
often the same model was asked about.
pkg/vram already caches what makes that expensive - the remote content-length
probes, the GGUF metadata reads and the HF repo sizes. Those caches key on a
gallery generation counter, and AvailableGalleryModelsCached triggered a
background refresh on every call, with each refresh bumping the counter. One
page view is one listing request plus thirty estimate requests, each of which
re-read the gallery and started another refresh, so the generation moved
constantly and every cache entry was stale before it could ever be read. The
caches were dead in production.
Three changes, each doing one thing:
A refresh interval. The cached list is still served immediately; this only
decides how often re-fetching from upstream is worth starting. Five minutes,
as a package variable so tests can drive it without waiting.
A generation bump only when the gallery actually changed. An unchanged gallery
re-fetched on schedule must not throw away work that is still valid, which is
the difference between an estimate costing nothing and costing a network round
trip.
A separate "loaded" flag. The cache engaged on `cached != nil`, so a gallery
that legitimately holds nothing read as never-loaded and took the blocking path
on every call, bumping the generation each time. Found by the test for the
interval, which could not pass while this was true.
Measured against a live instance with 1,595 models:
one estimate, repeated 2.3s -> 2ms
a page of 30, in parallel 10s -> 0.04s
A first, genuinely unseen model still costs its remote probe. That is inherent;
what changed is that it is now paid once per model per gallery version rather
than once per request.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* perf(gallery): warm VRAM estimates at startup, and stop the UI waiting on them
Two halves of the same complaint: the gallery stalls on VRAM estimation.
Server side, the estimates are now warmed in the background at startup.
Estimating an entry nobody has asked about costs a remote probe of its weight
files, and the gallery needs one per row, so the first visitor was paying for
the whole page. The warm-up walks the gallery in the order the UI lists it, so
the first page is ready before anyone reaches it.
It is bounded and it never blocks: 300 entries at 4 at a time by default, on
its own goroutine, stopping with the server's context. Warming the whole
gallery would be thousands of probes on every boot, which is rude to the
upstream and slow to finish; warming nothing leaves the first page paying two
seconds a row. Anything past the limit still warms itself on first view.
LOCALAI_VRAM_WARM_LIMIT=0 turns it off for an air-gapped host,
LOCALAI_VRAM_WARM_CONCURRENCY=1 slows it for a metered link.
Client side, the page no longer waits on estimates it does not need yet. It
fired one request per row at once; a browser allows about six connections per
host, so thirty estimates took every slot and the request behind a click - the
variant list, an install - queued behind work nobody asked for. That is the
freeze: the list was already usable, and the UI was busy fetching sizes. Four
at a time leaves room for the interactive request to overtake, and a row whose
estimate is still in flight says "sizing…" rather than leaving a blank where a
number will appear.
buildEstimateInput moves to core/gallery as EstimateInput, since the handler
and the warmer both need it.
Measured against 1,595 models, from a cold boot:
page 1, 30 estimates in parallel 10s -> 0.04s
full warm-up (299 of 300 entries) 3m, in the background
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* chore: untrack data/.local_user_id and ignore the runtime data dir
`local-ai run` writes its instance state under ./data when started from the
repo root, which is exactly what a contributor testing a build does. The
identity file ended up committed on this branch by a `git add -A` while
verifying the gallery changes against a live instance.
Anchored, so it matches the runtime directory at the repo root and not a
`data` directory nested inside some package.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Add the recommended Q4_K_M build and an MTP-enabled variant with the shared vision projector. Tag the existing Qwythos 9B MTP entry so serving-feature ranking recognizes it.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* chore(deps): bump cogito to v0.11 ahead of the nib harness
nib is the agent harness that becomes 'local-ai chat'. It requires cogito
v0.11, so pull that bump forward on its own: minimal version selection would
apply it to LocalAI anyway, and both repos use cogito and cogito/clients.
Landing it separately keeps the harness change reviewable.
nib itself is not pinned yet. Nothing in LocalAI imports it, and 'go mod
tidy' runs as a goreleaser before-hook in CI, so an unimported require line
does not survive. It lands with its first importer.
No LocalAI call site needed a change. Both cogito.WithMaxAttempts callers
guard the argument above zero, so v0.11's new clamp is unreachable, and
LocalAI's Multimedia values implement only URL(), so v0.11's new
TypedMultimedia routing treats them as images exactly as v0.10 did.
Binary size (cmd/local-ai): 200,301,381 -> 200,336,045 bytes (+34,664).
A throwaway probe that links nib measured 201,042,243 bytes (+740,862 over
the pre-change baseline).
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): resolve and seed the agent state directory
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): write the agent config atomically and tighten its modes
Replacing config.yaml in place truncated it first, so an interrupted write
would have destroyed the api_key nib keeps in the same file. Stage through a
sibling temp file and rename over the target instead, and match nib's 0700
directory mode.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): probe the endpoint and classify failures
Probe lists what a LocalAI endpoint advertises and separates the two
failures that need different advice: nothing listening, and rejected
credentials.
go-openai reports a rejected key as one of two concrete types depending
on the error body, and both occur against a real LocalAI. The normal
error handler sends an OpenAI error envelope, which arrives as
*openai.APIError; the opaque-errors handler replies with a bare status
and no body, which arrives as *openai.RequestError. Classifying on only
one of them misses half the cases, so the status is read from either.
A cancelled probe is not reported as an unreachable server, because it
learned nothing about the endpoint, and neither is a reply that could
not be parsed, because something did answer. Both would otherwise send
the user off to start a server that may already be running.
The model list is returned verbatim and in server order. LocalAI lists
whatever it finds in the models directory, including stray archives and
dotfiles, and deciding which advertised ids are real belongs to whoever
presents them.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): resolve the model from flag, config, or the server
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(chat): pin that model resolution sorts a copy of the caller's slice
The sort spec asserted only on what the chooser was offered, so replacing the
defensive copy with an in-place sort of req.Available still passed all 37
specs. Assert the input slice's order after the call, so the guarantee cannot
be dropped silently by a later refactor.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): offer to start a server when none is reachable
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): bound the server wait and pin readiness and stop semantics
Set cmd.WaitDelay so a backend subprocess holding the child's stderr pipe
cannot block cmd.Wait forever, which would leave exited unclosed, burn the
whole shutdown grace on a clean exit, and leak the waiter goroutine.
Two test gaps closed alongside it: the readiness spec now counts polls, so
treating 503 as ready is observable, and Stop's single-interrupt contract is
pinned by giving StartedServer interrupt/kill hooks that a spec can count.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactor(chat): drive Stop through one process interface, hide exec plumbing
Two independent interrupt/kill func fields plus a nil check admitted wirings
no test could distinguish: the pair swapped, so a SIGKILL would strand the
backends SIGINT exists to let local-ai run clean up, or kill left nil, so a
wedged server never escalates. One two-method interface that *os.Process
already satisfies leaves nothing to swap and nothing to nil.
Also translate exec.ErrWaitDelay, whose text names an os/exec struct field,
into what the user can act on. os/exec only substitutes that sentinel when the
process exited without an error of its own, so no exit status is swallowed.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): replace the REPL with the built-in agent
local-ai chat is now the nib agent harness compiled into the binary: tool
use behind an approval gate, sub-agents, MCP, plugins, and skills, all
auto-configured against the local server.
The REPL goes with it. Its model listing and its 401 classifier were
duplicates of the ones Probe now owns, and the classifier was the version
that misreads a bare 401 with no OpenAI error envelope, so keeping either
would leave the package with two divergent answers to the same question.
github.com/mudler/nib lands in go.mod in this commit rather than earlier:
go mod tidy runs as a goreleaser before-hook on every PR, so a require
line with no importer is stripped before it reaches CI.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactor(chat): split the pre-agent phase out of Run and pin it
Everything before the handoff is testable and nothing after it is: once
app.Run owns the terminal there is no seam left. prepare draws that line,
takes interactivity as a parameter so the prompts can be driven over a
pipe, and hands Run the state dir, the model, and any server it started.
The questions move onto one prompter that owns its buffered reader. A
fresh bufio.Reader per question reads ahead and discards what it buffered,
so the model choice typed behind an answer to "start a server?" was lost
and the next question saw EOF.
choose answers with a list index and refuses an empty offer, so a value
that was never on the list cannot reach ResolveModel, which persists it
and starts every later run against it.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): bound each server check with a deadline
Nothing bounded the model listing, so pointing chat at an address that
accepts the connection and then never replies left the user with no
output and no offer to start a server.
The budget is context.WithTimeout rather than a cancel plus a timer.
Probe deliberately refuses to call an endpoint unreachable on a
context.Canceled, since a caller who gave up learned nothing about the
server, and only honours a deadline. A cancel-based budget therefore
expires as the one error that suppresses ErrUnreachable, exactly for the
hung servers the offer exists to rescue.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): tell the user when their model choice cannot be saved
The choice is meant to be asked for once. When saving it fails the user
is silently asked again on the next run, and the only trace was an
xlog.Warn: the agent runs at log level error, and a --log-level=error run
swallows it entirely.
ModelRequest gains Notify for exactly this class of problem, one that is
worth telling the user about but not worth failing over, and the chat
wiring points it at the same writer the question was asked on.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): stop a session's server when the process is signalled
A server started for the session is stopped by a deferred call, and a
signal skips deferred calls: a SIGTERM between the spawn and the exit
left 'local-ai run' reparented to init with nothing left that knew to
shut it down. Ctrl+C was already safe, but only incidentally, because the
child shares this process' foreground process group.
A signal handler rather than Pdeathsig on the child. Pdeathsig is
Linux-only and, in Go, is delivered when the OS thread that forked exits
rather than when the process does, so it can fire on a healthy parent.
Setpgid would break the Ctrl+C that works today by taking the child out
of the foreground group.
SIGHUP joins SIGINT and SIGTERM: a terminal program whose terminal is
gone has nobody left to talk to. The same context is what cancels the
agent, which nib leaves to its embedder.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): only skip the server checks for work that stays local
Two argument shapes were classified wrongly. Every 'mcp ...' invocation
counted as management, so 'local-ai chat mcp --stdio', which serves the
agent over MCP and needs a model like any other session, was handed an
empty one. And --init, whose shell snippet a user pastes into an rc file
long before any server exists, went the other way: it demanded a running
server to print a static string.
The mcp split is asked of nib's own IsMCPManageSubcommand rather than
restated here, so a verb added upstream cannot drift out of this list.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): exit with the agent's status instead of reporting it twice
nib writes what went wrong to stderr and returns nothing but an exit
code, so returning that error unchanged had main log "Error running the
application error=exit status 1" underneath the message the user had just
read. The refusal to render the full-screen interface into a pipe is the
one they meet in practice: it names --cli, and burying that hides the fix.
ExitCodeError says "already reported, exit with this status". main
honours it and prints nothing more, so a piped or redirected chat still
fails a script the way it should.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* style(chat): route interactive chatter through one writer helper
The prompts and notices all write to a terminal, where a failed write is
not worth failing the session over and the read that follows the question
reports the real problem. say says that once instead of five discarded
error returns.
The command's one-line help comes along: chat is no longer "an
interactive chat session".
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(chat): record why the agent gets this process' streams
Injecting them is what makes nib refuse to draw its full-screen interface
into a pipe and name --cli, instead of rendering onto a terminal the
caller may not own. The tradeoff is worth stating where the wiring is.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): stop the session's server on cancellation, not on the way out
The deferred Stop is reached only if the agent returns, and cancelling
the context does not make it: nib hands the TUI to bubbletea without the
context, so what actually unwinds a running session today is bubbletea's
own SIGINT and SIGTERM handler. SIGHUP has no such backstop, and
registering for it removed the default disposition that used to end the
process outright, so kill -HUP left a live TUI with a cancelled context
and the started server still running.
runSession watches the context alongside the agent and stops the server
the moment it is cancelled, so the guarantee no longer depends on what
the agent does with cancellation. Stop is idempotent, so the deferred
call stays correct and free.
The doc comment on shutdownContext described the mechanism it was
supposed to work by rather than the one that does. Corrected, bubbletea's
handler included.
ResolveModel now checks the chooser's answer against what it offered.
The shipped chooser answers by list index and cannot be wrong, but
ModelChooser is exported, the answer is persisted, and every later run
starts against it, so the invariant belongs at the consumer.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(chat): bump nib to v0.5.1
v0.5.1 carries four fixes that matter to 'local-ai chat':
- --init now names the embedder's command, so the emitted widget invokes
'local-ai chat' rather than a bare 'nib' the user does not have.
- A piped CLI session that succeeds exits 0 instead of failing with EOF.
- EOF at a tool-approval prompt denies the call rather than approving it,
and the session exits 3 (app.ExitCodeApprovalNoInput) so a script can tell
"answered" from "refused to act" without reading stdout. Read-only tools
are unaffected and still run. ExitStatus already unwraps app.ExitError,
so the code propagates with no change here.
- RunTUI passes the context to bubbletea and gives up bubbletea's own signal
handler, which makes shutdownContext the single owner of the signal and
stops a SIGHUP leaving a wedged TUI behind.
Verified against a live server on 127.0.0.1:8080: the three --init shells,
a piped prompt exiting 0, a denied 'touch' that left no file and exited 3,
a read-only 'ls' that still ran and exited 0, and a SIGHUP that unwound a
TUI running under a pty.
Two comment blocks in run.go described the old TUI behavior and are now
wrong, so they are corrected in the same change. No behavior change: both
shutdownContext and runSession are untouched, and stopping the server on
cancellation is still worth keeping independent of how promptly nib unwinds.
One known gap, not addressed here. The widget --init now emits runs
'output=$(local-ai chat --height 50%)', and runAgent injects Stdout
unconditionally, so under $(...) nib refuses the TUI for a non-terminal
stream. This is the cost the runAgent comment already anticipated, now that
the snippets no longer hardcode standalone nib. Ctrl+Space should not be
documented until that is decided.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): let nib own stdout, so the Ctrl+Space widget works
The widget 'local-ai chat --init' emits runs
'output=$(local-ai chat --height 50%)', which puts a pipe on stdout by
construction. runAgent injected os.Stdout unconditionally, and nib refuses
every mode but --cli when a stream it was handed is not a terminal, so
Ctrl+Space printed "Re-run with --cli to use the injected streams" and
inserted nothing. Verified against a pty before and after.
nib reads a nil stream as "not injected" and falls back to the process
stream, which is how an embedder asks for nib's own behavior. That is what
stdout needs: the interface renders on /dev/tty but writes the chosen
command to stdout even when stdout is a pipe, and that write is the whole
of the shell-capture idiom.
Stdin is deliberately left injected. A piped or redirected stdin really is
ignored by the interface, so the refusal is the honest answer there, and it
is the one users meet: 'echo q | local-ai chat' still says to re-run with
--cli, once, exit 1. Nilling stdin the way stdout is nilled would delete
that silently. Stderr is not gated by nib at all and is unchanged.
One case does change and cannot be kept: 'local-ai chat > out.txt' from a
terminal no longer refuses, because it is indistinguishable from the
widget. It renders on /dev/tty and writes the capture line to the file,
which is what standalone nib does.
The app.Options literal moves into agentOptions so the decision is
reachable from a spec rather than being a detail of a function that takes
the terminal. Both sides of the asymmetry are pinned: reinstating
'Stdout: opts.Out' fails "hands nib nothing for the process stdout", and
nilling stdin fails "hands the process stdin over".
Also rewrites the last comments describing the pre-v0.5.1 behavior.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(chat): say what the stream refusal actually keys on
Two comments still called it the refusal to render the interface "into a
pipe". That was true when both stdin and stdout were injected, but a pipe on
stdout no longer refuses, so the wording now points at precisely the case
that was un-refused to make Ctrl+Space work. Only a stdin that cannot be
read triggers it, and both comments now say so and name the command a user
meets it with, 'echo q | local-ai chat'.
The agentOptions doc also said a "file a caller chose" stays injected and
refused, which reads as though 'local-ai chat > out.txt' still refuses. It
does not: a shell redirect arrives as os.Stdout and is nil-ed like the
widget's pipe, because the two differ only in being a regular file rather
than a FIFO and nib's gate does not look at that. What stays injected is a
writer an in-process caller chose for itself. Says that now, in the doc and
in the spec comment that had the same ambiguity.
Comments only. No behavior change.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: local-ai chat is now the built-in terminal agent
`local-ai chat` was a plain chat prompt and is now an agent that runs
shell commands behind an approval gate, so the pages that described a
REPL were wrong rather than merely thin.
Adds a Terminal agent feature page at /features/terminal-agent covering
the approval gate, piped runs and their exit codes, Ctrl+Space, model
resolution, state directory, and the pass-through management commands
(including the `--yes` caveat that leaves a plugin installed but
disabled in a script).
The three-way "looking for something else" notice becomes four-way and
moves into an agentic-routing shortcode. Four hand-kept copies of the
same paragraph is what produced the drift the new page would otherwise
have added to; the shortcode takes `current=` so each page still marks
itself, and errors the build on a name that is not one of the four.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* website: the agent is in the binary, not a second install
The nib section sold a separate tool you also install, with a GitHub
link as the only way in, which is now the wrong order: the agent ships
compiled into local-ai, and the standalone binary is the second reason
to care rather than the first.
Leads with `local-ai chat`, keeps nib as the SSH-anywhere story, and
adds a docs CTA pointing at the new Terminal agent page. id="nib" is
left alone because localai.io/#nib is linked from outside.
The two credits on the demo clip named nib as the thing that drove the
machine; they now credit the agent in LocalAI, which is the same agent.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* website: fix the exit keys, the plugin warning, and the redirect gap
Three claims on the chat-agent pages that the code does not back.
try-it-out told readers to press Ctrl+D. nib has no Ctrl+D handler: the
full-screen interface quits on Esc or Ctrl+C, and Ctrl+D is only an exit
in --cli, where it arrives as ordinary tty EOF. That sentence had
replaced the removed /exit and /quit text, so the page was left with no
working way to leave a session. Document both modes, since they differ.
The plugin warning said nothing tells you the install stopped short. It
does: the command prints that the plugin was left disabled. What it does
not do is say so in its exit code, which is 0 either way. That is the
part a script cannot work around, and it is the reason to pass --yes.
Overstating it in the paragraph that gives the advice only makes the
advice easier to dismiss.
Redirecting stdout no longer refuses; the interface goes to /dev/tty and
only the yanked command reaches the file. It is what lets the Ctrl+Space
widget capture a command at all, since a redirect and out=$(...) are the
same thing to the stream gate. It was documented nowhere. A non-terminal
stdin is still refused, and the new text says which of the two it is.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): make the CLI flags outrank the agent config file
local-ai chat routed --endpoint, --model, --api-key, --trace-dir and --yolo
through nib's app.Options.Defaults. Defaults are seeds: they sit beneath the
config file, so the file silently undoes them. That made the flags accepted and
inert, and not in an edge case, since EnsureStateDir writes base_url on the
first run and the interactive picker writes model, so from the second run on
the file carried a value for both.
Observed against a live server: with base_url: http://127.0.0.1:9999/v1 in the
config and --endpoint http://127.0.0.1:8080 on the command line, the probe hit
8080 and every agent turn posted to 9999. With model: gemma-4-e2b-it-qat-q4_0
in the config, --model lfm2.5-8b-a1b was ignored on the wire.
nib v0.6.0 adds app.Options.Overrides, applied above the config file and above
the bare environment block. Move the whole block there: all five values are
decisions this invocation already made on the user's behalf, and a flag the
config file can undo is not a flag. Nothing is left in Defaults, because
LocalAI's one genuine seed, the initial base_url, is written into the config
file by EnsureStateDir rather than handed to nib.
Two limits come with the channel and are documented on agentOptions rather than
worked around. An override can only raise a field, since nib cannot tell "set
to the zero value" from "not set", so --yolo can turn approval off but nothing
on the command line turns it back on over an approval_mode: auto in the file.
And nib's own NIB_TRACE_DIR and NIB_YOLO are resolved after the config load and
still outrank these, deliberately, upstream.
The existing spec pinned that the right values reach app.Options, which they
always did, which is exactly why it could not see nib discarding them. The new
specs resolve the config the way app.Run resolves it, against a real config
file that disagrees with every flag, and one asserts Defaults stays empty.
docs/content/features/terminal-agent.md already documented --model as winning
over the saved model; that was false before this change and is true now, so no
docs edit was needed.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(chat): document intentional config file read
Assisted-by: Codex:gpt-5 [gosec]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The post was written against the first draft of the release notes, when the
cycle stood at 214 PRs over thirteen days. It closed at 321 PRs over eighteen
days, and three of the larger user-facing changes landed after it was written.
- Correct the counts throughout: 321 PRs, eighteen days, 24 contributors
(11 first-time), gallery 1,221 to 1,505.
- Add sections for the three new capabilities: 3D generation as a modality
(Generate3D, FLAG_3D, /v1/3d/generations, trellis2cpp), audio.cpp serving
six audio endpoints from one process, and the operations bar becoming the
Activity page.
- Cover the two further hardening fixes (tar hardlink escape, cyclic $ref
stack overflow) alongside the TRL one.
- Note the Valkey store, systemd socket activation, persistent trace history,
in-place chat edits, the self-contained SYCL backend and the site split.
- Group the new-engine sections together rather than splitting them across
the operational ones.
Embeds the existing vllm-race and magpie clips, and adds a 3D generation clip
cut from the demo recording to the conventions in .agents/preparing-a-release.md
(no audio track, 14s, named for the feature). blog.css styled figure img but
not figure video, so a clip in a post rendered outside the card; both selectors
now share the rule.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
tests-qwen3-tts-cpp has been failing on master since 2026-07-31. The suite
loads every component fine and then stops: TTS() never returns from the
native call, so a job that takes ~5 minutes runs into the 20 minute Go test
timeout instead.
goroutine 74 [syscall, 19 minutes]:
github.com/ebitengine/purego.RegisterFunc.func4
qwen3-tts-cpp.(*Qwen3TtsCpp).TTS goqwen3ttscpp.go:154
qwen3-tts-cpp.init.func2.4 e2e_test.go:90
Not a flake: reproduced on master and again on an explicit re-run.
Bisected across this cycle's eight qwentts.cpp bumps by their own check:
10832, 10850, 10902, 10964, 11006, 11039 and 11127 all pass in ~5 minutes;
11241 (abab6b3) fails at 1h58m. That PR was merged with this check already
red, which is how the hang reached master.
35ebe537..abab6b3 is three upstream commits, and the only functional one is
26dd8adb, "predictor: unroll the frame into one cgraph and sample in standard
ops", which is consistent with a generation loop that never reaches its stop
condition.
Hold the pin at the last known-good commit. The bump entry is commented out
rather than left in place, because it tracks upstream master and would put
the hang straight back on the next nightly run. Both spots carry a pointer to
the other so the hold is discoverable, and restoring it is uncommenting four
lines once upstream is fixed.
Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Select the CPU_ALL_VARIANTS target for x86 GPU images so partial offload uses runtime-selected host kernels. Keep GPU arm64 builds on the portable fallback until their toolchains consistently provide gcc-14.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): drop merged MiniMax-M3 patch
The bumped llama.cpp revision includes the MiniMax-M3 parser and template detection, so the carried patch now rejects during backend preparation. Remove the obsolete patch while retaining the independent score-task patch.
Assisted-by: Codex:gpt-5 [systematic-debugging]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add the new 9B Fara computer-use model alongside its existing 27B sibling, with Q4_K_M and Q8_0 llama.cpp variants plus the required vision projector.
Assisted-by: Codex:gpt-5 [web]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Build the runtime CPU variant set alongside x86 GPU backends so partial offload uses the host's SIMD kernels instead of the scalar fallback. Keep arm64 GPU images on the portable binary until their builders consistently provide gcc-14.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(utils): reject tar hardlinks that escape the extraction root
ExtractArchive pre-scans archive members and rejects symlinks, but tar
hardlink entries carry a regular file mode and so pass that check.
Header.Linkname was never validated, so an archive could create a link
to a path outside the destination directory.
Validate Linkname with the same path check already applied to member
names. Hardlinks that resolve inside the extraction root still extract,
so ordinary archives are unaffected.
pkg/oci/image.go already resolves tar.TypeLink targets before using
them; this brings the archive extraction path in line with it.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Zelys-DFKH <zelys@dfkhelper.com>
* test(utils): cover hardlink overwrite and in-root hardlinks
The existing hardlink test names a link target two levels above the
extraction root, so its final assertion checked a path the link never
resolved to and could not fail. Point the target one level up instead,
at the path that assertion already names.
Add two cases. The first uses a .tar.gz, where ExtractArchive binds a
Tar config with OverwriteExisting set, and follows the link entry with a
regular entry of the same name. Before the fix that pair linked to a
file outside the root and then truncated it through the link, which the
plain .tar case does not reach. The second extracts a hardlink whose
target is an earlier member of the same archive, covering the claim that
ordinary archives are unaffected.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Zelys-DFKH <zelys@dfkhelper.com>
---------
Signed-off-by: Zelys-DFKH <zelys@dfkhelper.com>
Use the case-sensitive Hugging Face filenames and refresh the linked SHA256 values for both gallery variants.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Detect text-to-speech MLX repositories during model import and emit a TTS-ready mlx-audio configuration. Expose mlx-audio in the backend preference dropdown for repositories without complete metadata.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(sycl): make the intel llama.cpp backend self-contained on any host
The SYCL backend shipped an incomplete oneAPI runtime AND relied on a
host-provided GPU driver, so it only ran inside the build container. On a
bare host it died with "libze_loader.so.1 / libdnnl.so.3: cannot open
shared object file", and even with the host's Intel driver installed it
SIGSEGV'd during SYCL init when the host driver was built against a newer
glibc than the backend's bundled loader (rolling-release distros).
package_intel_libs now bundles the complete, coherent oneAPI runtime
(the missing MKL ILP64 / sycl_blas / tbb_thread + oneDNN + the dlopen'd
UR adapters, plus a sweep of the backend binaries' own direct deps) and
the Intel GPU userspace driver (libze_intel_gpu + libigdrcl + IGC + gmm)
with its OpenCL ICD manifest, mirroring how package_vulkan_libs bundles
Mesa. run.sh points the Level Zero and OpenCL loaders at the bundled
driver, and install-base-deps.sh installs it in the SYCL build image.
Bundling the driver is safe across kernels because it talks to the host
i915/xe via the stable DRM UAPI (unlike NVIDIA's kernel-locked
userspace).
Validated on Arch (glibc 2.43, i915): the backend loads and runs on an
Iris Xe with no host Intel packages installed.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
* fix(sycl): install a driver that exists, and let the user choose their own
The driver install added earlier in this branch asked apt for
intel-level-zero-gpu, which is not a package in Ubuntu 24.04. apt fails
outright on an unknown name, so neither driver was installed, nothing was there
to copy, and the images carried no driver at all.
It now comes from Intel's own repository, which has 25.18 for this Ubuntu
release, against 23.43 from late 2023 in the Ubuntu archive. The archive driver
does not know any card released since, so a machine with a recent Intel GPU
would end up carrying a driver that cannot drive it. Anything that goes wrong
during that install fails the build on purpose: an unreachable repository is a
passing problem that a retry fixes, while quietly carrying a different driver,
or none, is a difference nobody would notice until a user reports an idle GPU.
run.sh used to overwrite whatever driver the user had chosen. Level Zero uses
only the driver it is given, so on a machine with a card too new for the
carried driver, the GPU would go unused with no way back. Both that setting and
the OpenCL one are now left alone when already set, and the docs say how to
point a backend at the machine's own driver.
The OpenCL setting also used to be applied whenever the backend held a driver
list, even when the driver it named had not been copied, which leaves OpenCL
with nothing instead of falling back to the machine's own driver. It now
requires the copied driver to be present, and the packaging leaves out the list
entry of any driver it did not copy. The oneAPI images list a processor-only
OpenCL library, which was being carried with nothing behind it.
Two more corrections in the packaging. The scan for libraries a program is
linked against only looked at files named llama-cpp-*, so turboquant and bonsai,
which are also built for Intel GPUs, were left with the incomplete set of
libraries this branch set out to fix; it now looks at every program in the
directory. And a build that should carry a driver but ends up without one now
says so, which is what a stale prebuilt base image looks like: such a backend
still runs on a machine that has its own driver, so nothing fails and the only
other symptom is a user reporting an idle GPU.
Backends now also ask the driver to report how much graphics memory is free,
without which llama.cpp reads zero on an integrated GPU, since such a chip
shares the system memory instead of having its own. turboquant and bonsai get
the same run.sh handling as llama.cpp.
The driver is only carried by the builds that start through run.sh, because
run.sh is what points Level Zero and OpenCL at it. The Python backends for
Intel GPUs start differently and would never load it, so they keep using the
machine's own driver rather than carrying several hundred megabytes they cannot
use.
Checked in a container on Ubuntu 24.04: the install brings driver 25.18 with
the files where the packaging expects them, an unreachable repository fails the
build, and the copied set resolves on its own once the machine's Intel packages
are moved away.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
* fix(ci): rebuild every Linux backend when the GPU packaging script changes
scripts/build/package-gpu-libs.sh decides which GPU libraries end up inside an
image. The filter that builds the backend matrix listed it as an input of the
Python images only, so changing it rebuilt no Go and no C++ backend, even
though those run it from their own package.sh. A packaging fix aimed at the
Intel llama.cpp backend could merge and reach no image, which is the same
failure this rule was written to prevent.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
* fix(sycl): carry only the driver Level Zero uses, not the OpenCL one
llama.cpp reaches an Intel GPU through Level Zero, which hands the driver
programs that are already compiled and so needs only the back end of the
graphics compiler. The OpenCL driver can be handed source code instead, so it
needs the compiler's front end as well, and that arrives with its own copy of
clang. Carrying it cost about 139 MB in every backend built for Intel GPUs, and
took the carried set from 123 MB to 261 MB.
Nothing here takes that path. No LocalAI code selects an OpenCL device, each
backend image holds one backend, and the documentation never described OpenCL
as a way to run models: the only mentions are a stale clblas row in the
BUILD_TYPE table, for a llama.cpp backend that no longer exists and that no
build matrix entry uses, and the sycl-ls troubleshooting hint. Before this
branch the packaging carried the OpenCL loader and adapter but no driver, so
the path could not work in a released image either. There is nobody to keep
working.
The driver list that OpenCL reads is no longer carried, and run.sh no longer
sets OCL_ICD_VENDORS, so OpenCL inside a container keeps using whatever the
image provides rather than being pointed at a directory with no driver in it.
Checked in a container against the real 25.18 driver: the carried set is 123 MB
with nothing unresolved, and Level Zero still reports the GPU with the
machine's own Intel packages moved out of the way. Neither the Level Zero
driver nor the compiler back end names the front end or clang among the
libraries it opens by name, so the leaner set is complete for this path.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
---------
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
* feat(chat): edit saved conversation messages
Add inline edit, save, and cancel controls for stored user and assistant messages without triggering inference. Preserve structured message attachments and cancel edits when streaming starts.
Assisted-by: Codex:gpt-5
* test(chat): preserve seeded conversation on reload
The saved-message edit test reloads the page to verify persistence, but its init script was replacing localStorage with the original fixture on every navigation. Seed only an empty store so reloads exercise the data written by the application.
Assisted-by: Codex:gpt-5 [Codex]
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Direct repository-root users to the supported make docs target and document the equivalent direct Hugo invocation from docs/.
Fixes#10062
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The page reads as a list of features with nothing joining them, so two
things did not land.
The engines section never said these are the backends LocalAI loads. The
runtime section describes a core that pulls each engine in on demand, and
the engines section describes engines written from scratch, and nothing on
the page connected the two sentences. Readers were taking parakeet.cpp and
the rest for unrelated side projects by the same people. The lede now says
whose backends they are before it says anything else.
APEX was used as a known term on first appearance, in a section that opened
onto a benchmark table. Nothing said what it is or why it follows the
engines. It now opens by placing itself in the stack: the engine decides how
fast a model runs, the weights decide whether it runs at all, and APEX is
the second of those. Then the numbers.
Also drops "Most backends wrap somebody else's engine. These do not", which
is the machine-written antithesis shape, and fixes a list that broke its own
parallel halfway through.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m]
The star, fork, contributor and release counts were typed into the templates
by hand, so they only moved when somebody remembered. They had already
drifted: stars read 48,042 against 48,067, forks 4,314 against 4,320, and
contributors 224 against 225.
They move to website/data/stats.yaml, which .github/ci/refresh-site-counters.sh
rewrites from the GitHub API, run weekly by a new workflow. The contributors
and releases endpoints never report a total, so the script asks for one item
per page and reads the count out of the Link header. It refuses to write a
zero or a non-number, which is what a rate-limited or failed call looks like,
and the workflow commits only when a number actually moved. The Discord count
has no API behind it, so the script reads the existing value back and carries
it through.
The engine count was wrong in a second way. The hero said 18, the section
heading said "Eighteen engines", the timeline said "Nineteen engines of our
own", and the /engines/ page derived 19 from the data file. All of them now
derive from that same file, so they cannot disagree again.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m]
Add Q4_K_M and Q8_0 llama.cpp entries for the newly released Qwopus3.6-27B Fusion reasoning and coding merge, with MTP enabled.
Assisted-by: Codex:gpt-5 [Hugging Face API]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The timeline set six 15rem columns in a flex row with overflow-x:auto, which
needs 90rem and so scrolled sideways on any normal laptop. It is a wrapping
grid now, and the rule that carries the dots moves from the container onto
each item so a wrapped row still gets a line above it. Column gap is zero and
the items carry their own right padding, so the rule stays continuous.
Integrations move from a card grid to a reel. Any single integration is a weak
signal and the whole moving line is the strong one, so the count is doing the
argument. It pauses on hover and on keyboard focus, since the names are links.
The list grows from 8 to 26: Open WebUI, Dify, LibreChat, RAGFlow, Continue,
big-AGI, Nextcloud, Frigate, promptfoo, Mods, TypingMind, baibot, k8sgpt-
operator and others. Each was admitted only after opening that project's own
repository or docs and reading the line that names LocalAI. The ones that
failed that test are listed in the data file so nobody re-adds them.
The blog cards were hand-written, which is how one of them came to advertise
"Porting vLLM to C++", a post that does not exist, and how all three linked to
the blog index instead of an article. They range over the posts now.
The section intro used the "a changelog tells you what moved, these posts show
you what it does" shape, which is the standard machine-written antithesis. It
states what the posts contain instead, including the perplexity regression
that APEX costs, because publishing the price is the actual claim.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m]
The band led with three sentences of hedging and printed a commit count
next to each employer, so a one-commit entry beside a large name read as
weakness rather than as the modest, true claim it was. It now opens on the
contributor count, sets the employers as a sentence instead of a pill wall,
and keeps the caveat to one line. The counts stay in ecosystem.yaml, since
they are the provenance for the list and anyone re-checking it needs them.
Two "coverage" cards were not about this project. The modelslab.com piece
reviews Frikallo/parakeet.cpp, an unrelated project of the same name, and
the snailtext.app benchmark measures Parakeet through ONNX Runtime without
mentioning LocalAI at all. Both are removed, along with the contributor
card that duplicated the band's opening line.
Press was four posts from one vendor, which read as the whole of the
coverage rather than one enthusiastic outlet. SUSE collapses to a single
series entry, and Pulumi, Semaphore and Spectro Cloud join it. Each was
opened and checked against the project before being added. K8sGPT and
LlamaIndex join the integrations; both document LocalAI as a backend.
The quotes move above the lists so the section opens on its strongest
line, which is somebody else's. The hero gains a GitHub call to action,
the APEX collection link was returning 404 and is corrected, and the
footer no longer describes the site as a design mock.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m]
Follow-up to 42541dd4f, which routed lint to arc-runner-set. Both of its
jobs failed there in one second (run 30637392862): the runner image has
git, curl, unzip, tar, ldd and python3, but not make, and build-scripts
additionally needs gcc because the packaging-script tests compile a
throwaway binary and inspect it with ldd.
golangci-lint needs make twice over: `make protogen-go` (which also wants
curl + unzip to fetch protoc) and `make lint` itself. So both jobs go back
to ubuntu-latest.
gh-pages.yml stays on arc-runner-set and is unaffected: it uses no make and
no C toolchain, and setup-go / actions-hugo fetch their own toolchains.
The preflight steps stay. They cost about a second on the hosted pool, and
they are what turned this into a one-second named failure instead of an
opaque one midway through a build. When the runner image gains make + gcc,
re-routing is one runs-on line per job. Any such re-route must stay
push-only: lint also runs on pull_request, and fork PRs execute untrusted
code that must not reach a persistent self-hosted runner.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
The GitHub-hosted runner pool is shared per ACCOUNT, not per repo, so a
burst in one repo starves every other. On 2026-07-31 it reached zero
scheduled jobs for 35 consecutive minutes with 39 jobs queued, while
arc-runner-set completed 12 jobs without interruption across the same
window. Actions was healthy globally (other public repos were scheduling
normally), so this is an account-level throttle we cannot fix from inside
the workflows, only route around.
Site publishing and lint are small, run on nearly every commit, and gain
nothing from waiting behind a saturated hosted queue, so both move to
arc-runner-set, the label already proven in generate_intel_image.yaml.
lint.yml is routed for PUSH ONLY, and this is the important part: that
workflow also triggers on pull_request, and a fork PR executes untrusted
contributor code. Running that on a persistent self-hosted runner would be
a real compromise vector, so anything that is not a push to mudler/LocalAI
stays on the ephemeral hosted pool. gh-pages.yml needs no such clause: it
triggers only on push-to-master and workflow_dispatch, so it never runs
pull-request code. Both carry a repository guard so forks, which have no
such runner label, fall back to hosted instead of queueing forever.
Neither workflow uses sudo or apt, and both fetch their own toolchains via
setup-go / actions-hugo. A self-hosted image can still be leaner than the
hosted one, so each lint job opens with a preflight that names the missing
tool (curl/unzip/make for protoc and lint; gcc/ldd/python3 for the
packaging-script tests) rather than failing opaquely mid-build. Reverting
is one runs-on expression per job.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Signed backend verification performs separate registry requests for manifests and referrers. Reuse LocalAI's version-aware User-Agent there so the full install flow is attributable to LocalAI.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Adds the "who turns up around this project" section, split into three lists
because the evidence behind each one is a different strength and collapsing
them into a single logo wall would overclaim.
Contributors 21 companies whose engineers have commits here. Evidence is
the commit history plus the employer on that person's public
GitHub profile, so it is a claim about the person. Commit
counts are shown next to each name, including the ones that
are a single patch, because hiding that would be the whole
problem.
Integrations six projects that reference LocalAI in their own repository
or documentation, which anyone can verify without asking us.
Press four SUSE Communities articles about running LocalAI.
Names are set in type rather than fetched as logos. A logo reads as
endorsement, and a one-line typo fix from somebody who happens to work at a
large company does not support that, quite apart from what their trademark
policy says about it.
ADOPTERS.md is the mechanism for the stronger claim. An organisation that
wants to be listed as a user opens a pull request adding itself, which is both
the evidence and the permission, and is publicly auditable afterwards. The
file says plainly what the website does and does not claim, so the next person
to ask "can we add some big names" has the answer in the repository.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] [WebSearch]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(website): split the site, move docs to /docs, add a landing page
The Hugo docs site has always been localai.io itself, which left nowhere to
explain what LocalAI is or show what the team builds. This adds a separate
marketing site at the root and moves the documentation under /docs/.
Docs:
The existing site keeps its content tree and its Relearn theme, and now
builds with baseURL <root>/docs/. Its _index.md, which held a hand written
landing page, becomes a real documentation home.
Every previously published URL keeps working. GitHub Pages has no server
side rewrites, so .github/ci/gen-redirects.sh walks the built docs output
and leaves a meta refresh plus a canonical link at each old root path. It
covers bare .html files too, which is what keeps /gallery.html alive, and
it never overwrites a path the marketing site already owns.
Website:
A second Hugo site under website/ with its own layouts and no external
theme, so the marketing side does not have to fight Relearn's home rooted
menu and asset pipeline. CI builds both and merges them into one Pages
artifact.
The design is derived from the project logo rather than invented: the navy
of the triangle, the cyan of the llama, the purple of the speed bars. Those
offset bars became the motion signature. The background renders a real
depth-anything.cpp depth map as contour lines and switches to a
locate-anything.cpp style detection overlay over the engines section.
Also included: an /engines/ index driven entirely by data/engines.yaml, a
/blog/ section with five posts written from the release notes and the
engine benchmark suites, install.sh and a Kubernetes manifest since the
site advertises both, and a rule in .agents/ that release preparation now
includes a blog post and demo clips.
Every figure on the site is derived from the repository or the GitHub API,
not from memory. Correcting them against their sources found one error in
README.md: voxtral-tts.c is text to speech, not speech to text.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] [Agent]
* feat(website): add a star history chart, rewrite the history post in first person
The history post read like a changelog written by a committee. It is now in
Ettore's voice, first person, with the admissions left in.
The numbers paragraph in particular read like a directory listing. It now says
what the figures mean rather than which file they came from.
Adds an interactive star history chart, built from the GitHub stargazers API
rather than embedded from a third party, so the page makes no external request
and cannot break when someone else's service is down. The four releases the
post is organised around are marked on the curve, and the labels stack into
rows because three of them land within two months of each other.
The API stops paginating at 40,000 items, so the curve is measured up to
December 2025 and the segment from there to today's total is drawn dashed,
labelled as an estimate in the caption and in the tooltip. It is a straight
line between two known points, and the chart says so rather than implying it
is data.
Also drops "marketing site" from the README heading and everywhere else it
appeared, and calls it the main site instead.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
PR #11041 rewrote the testllama31inputResult1 fixture and un-escaped the
backslashes inside its Go raw-string literal, turning `[^"\\]` into
`[^"\]` and `["\\/bfnrt]` into `["\/bfnrt]`. The fixture is compared
line-by-line against the grammar built from PRIMITIVE_RULES in
bnf_rules.go, which is unchanged and still emits the doubled form, so
"generates a valid grammar from JSON schema" fails on every platform.
Restore the four fixture lines to their pre-#11041 form. The cyclic $ref
and depth specs added by that PR are untouched.
The regression reached master because only the DCO check ever reported
on #11041; its test runs were cancelled during the CI purge.
Assisted-by: Claude Code:claude-opus-5[1m] [Bash] [Edit]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Add the Qwen3.5-MoE POCKET-35B and Gemma 4 POCKET-26B releases with Q4, Q2, and compact IQ1 variants where available. Verify every artifact hash against its Hugging Face linked etag.
Assisted-by: Codex:gpt-5 [web]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* refactor(ui): add layout/text primitives and per-page CSS blocks
The React UI already shipped a design system (tokens, form grids, data
tables, stat cards, callouts) that the pages largely bypassed: ~2,000
`style={{ ... }}` literals across src/pages and src/components. Each one is
a spacing or colour decision made locally, so no two pages share a rhythm,
which is the main reason the app reads as unfinished rather than as one
product.
Two additions, both to App.css:
- A small semantic primitive layer: .stack / .hstack for vertical and
horizontal rhythm, .text-note / .text-sub / .text-meta / .text-mono for
the text roles the pages kept re-deriving, .tone-* + .icon-chip for
semantically tinted icons, plus scale-locked spacing and size steps.
Deliberately short and semantic, not a utility framework: the size and
spacing classes exist mainly so that an OFF-scale value stays an inline
style and therefore stays visible.
- Named blocks for the shapes fifteen pages actually have (.p2p-diagram,
.usage-tile, .tr-code, .mw-badge, .set-rail, ...), so those shapes are
defined once instead of per call site.
Two findings worth recording. The type scale is xs 0.6875 / sm 0.8125 /
base 0.875, and 0.75rem was in use roughly 100 times without being on it
(along with 0.7, 0.85, 1.1 and 0.625rem); all now snap to the nearest step.
And there were thirteen distinct table column widths across the app where
three or four would do; they are pulled into .col-w-* so the ladder is
visible in one place, ready to normalise separately.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
* refactor(ui): move repeated inline styles onto shared classes
Six passes over src/pages and src/components, each matching a whole
`style={{ ... }}` attribute exactly so the swap is provably equivalent:
- the identical "waiting for the first response" wrapper, repeated
verbatim in 21 files
- text roles (size + colour combinations) onto .text-note / .text-meta /
.text-sub / .text-mono
- semantic colours, type-scale steps, .panel-title, .list-row
- spacing and weight steps, .stack / .hstack rows
- table column widths, small pills, chart legend swatches
- the remaining shapes appearing three or more times
One class of bug is worth calling out, because it is what a careless
style-to-class conversion produces and it is invisible to every check we
run. Adding `className="x"` to an element that already had a className
leaves TWO className attributes; JSX keeps the last and silently drops the
first, so `<i className={icon} className="text-xs" />` loses its icon while
passing eslint, `vite build` and the full Playwright suite. 112 of these
were introduced and repaired here. The gate added in a later commit fails
on them.
No visual change is intended beyond snapping off-scale font sizes onto the
type scale. Verified after every pass: eslint 0 errors, vite build passes,
Playwright page-render-smoke + navigation 22/22.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
* refactor(ui): convert fifteen pages onto named classes, add inline-style gate
Full per-page conversions, each one reading the page, naming the shapes it
actually has, and leaving inline only what is computed at runtime:
P2P 205->0 Traces 50->0 ConfigFieldRenderer 23->0 NodeDetail 29->1
ModelEditor 25->1 NodeInstallPicker 29->1 FineTune 175->2
Nodes 33->2 ImportModel 35->3 Talk 43->4 AgentJobDetails 27->4
Usage 97->6 Settings 24->6 Middleware 54->8 Backends 87->45
Every remainder is genuinely dynamic: a data-driven badge colour, a
`width: ${pct}%`, a tooltip's coordinates.
Naming the shapes made reuse fall out on its own. Nodes reuses the P2P
setup shapes (both present the same "no workers yet, here is how to add
one" flow) and Model Editor reuses the Settings section rail, which was
byte-identical. Two shared *style objects* also turned out to be classes
wearing a costume and were deleted: `monoCell` in Usage and `hintStyle` in
ImportModel.
Some fixes fell out of the conversion. The evaluation toggle on the
fine-tuning page was a hand-rolled div that rendered as a clipped circle;
it is the existing Toggle component now. That page's empty state pointed at
a "New Job" button that was scrolled off the top of the page, and now
carries its own call to action. And `.input--file` is added at the system
level rather than as a local hack, because ImageGen and VideoGen truncate
their file inputs the same way today.
scripts/inline-style-gate.mjs is the ratchet that keeps this from
regressing. It does not forbid inline styles; it fails when the total goes
UP (same discipline as the coverage baseline) and when an element carries
two className attributes. eslint would catch the latter via
react/jsx-props-no-duplicate-props, but that needs eslint-plugin-react,
which this project does not depend on, so the check lives next to the tool
that causes the problem.
npm run lint:inline-styles # check against baseline
npm run lint:inline-styles:report # per-file counts, worst first
npm run lint:inline-styles:write # refresh after converting
Net: 2,061 -> 611 inline styles. eslint 0 errors, vite build passes,
Playwright page-render-smoke + navigation 22/22, gate green on both checks.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
refactor(backends): extract shared package-system-libs.sh from package.sh
The arch-detect-and-copy-system-libs block (Darwin rpath / x86_64 / aarch64
loader + libc/libstdc++/libgcc_s/libm/libgomp/libdl/librt/libpthread) was
inlined verbatim in 31 backend package.sh scripts. Extract it into a single
sourced scripts/build/package-system-libs.sh, the CPU-side counterpart to
scripts/build/package-gpu-libs.sh and its sourcing contract.
Consolidating the copies fixes three drift classes that had crept in:
- libgcc_s.so.1 and libstdc++.so.6 were listed twice in 9 backends
(acestep-cpp, crispasr, moss-tts-cpp, omnivoice-cpp, piper,
qwen3-tts-cpp, silero-vad, stablediffusion-ggml, whisper); the shared
script copies each once.
- libgomp.so.1 was omitted from opus. OpenMP consumers dlopen it rather
than link it, so the missing copy only failed at runtime; the shared
script always includes it.
- the Darwin @loader_path/lib rpath was applied only in piper and
silero-vad; both now pass their packaged binary to the shared script,
preserving that behavior. Every other backend passes an empty binary
path so no rpath is added, preserving its current behavior.
Each backend's pre/post packaging steps (binary copy, run.sh, ldd closure
walks, ggml variant bundling, espeak/OpenBLAS extras, the ds4 validate step)
are preserved verbatim; only the inline if/elif/else arch block is replaced
by a single source line.
Signed-off-by: supermario_leo <leo.stack@outlook.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
* fix(backends): fall back to copying links when the filesystem rejects symlinks (#10890)
Backend installation extracts the OCI image tar via containerd's
archive.Apply, which calls os.Symlink directly. On filesystems that do
not support symlinks (notably CIFS/SMB mounts, commonly used to back the
/backends volume) the syscall fails with "operation not supported" and
the whole install aborts, leaving an empty backend directory. The CUDA
llama.cpp image trips this on the libcublas.so -> libcublas.so.12.x
symlink.
When archive.Apply fails with a link-unsupported error, reset the
staging directory and re-extract with a pure-Go walker that still
attempts real symlinks/hardlinks first and degrades to copying the link
target's contents in place when the filesystem rejects them.
mutate.Extract already flattened the layers, so the tar carries no
whiteouts to interpret. Link copies are deferred to a second pass so
forward references resolve.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]
* fix(oci): check deferred Close in copyFilePreservingMode (errcheck)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]
* fix(oci): reject path-traversal tar entries in the link-copy fallback
safeJoin sanitized "../.." entries by clamping them under root instead of
rejecting them, so a malicious entry was silently redirected rather than
refused. Join without the leading-slash trick and reject any entry whose
cleaned path resolves outside root; absolute link targets are still mapped
under root (image-root relative) rather than escaping.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]
* fix(oci): silence gosec on the validated link-copy file ops
Use hdr.FileInfo().Mode() instead of converting the int64 tar mode to
os.FileMode (removes two G115 overflow findings), and annotate the tar
extraction file operations with justified #nosec comments: every path is
validated by safeJoin against the extraction root before use (G304/G305).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]
* fix(oci): build extraction image from downloaded layers
Avoid appending downloaded layers to the original remote-backed image, which duplicates the layer stack and reopens the source during extraction. Building from an empty image preserves the flattened whiteout semantics while keeping extraction local.
Assisted-by: Codex:gpt-5
* fix(oci): materialize chained links in dependency order
Retry deferred link copies until their targets exist so soname chains work on filesystems without symlink support. Document that copied links can increase backend storage usage on CIFS and SMB mounts.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(api): add /v1/detokenize endpoint
Closes#1649.
Mirror of the existing /v1/tokenize path, requested by @benniekiss in
the issue thread for "complete API workflow" use cases that need to
turn token IDs back into text without local processing.
- Add Detokenize gRPC RPC with DetokenizeRequest{tokens} /
DetokenizeResponse{content} messages.
- Implement in the llama.cpp backend using common_token_to_piece, the
same primitive TokenizeString already uses internally.
- Other backends inherit the default Unimplemented from base.Base, in
line with how Detect, Rerank, etc. are gated per-backend.
- Wire up the Go gRPC interface, server, client, and in-process embed
wrapper alongside their TokenizeString counterparts.
- Add the schema types, ModelDetokenize wrapper, HTTP handler, route
registration, RouteFeatureRegistry entry (gated by FeatureTokenize so
no new feature flag is needed), and the discovery map entry under
ai_functions.
- Regenerated swagger reflects the new endpoint and types.
- Update authentication.md to list /v1/detokenize alongside /v1/tokenize.
Assisted-by: Claude:claude-opus-4-7
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
* test(e2e): add mock backend tests for /v1/detokenize
Add Detokenize to the mock gRPC backend and wire up two e2e tests in
the MockBackend suite: one that posts known token IDs and asserts a
non-empty content response, and a round-trip that tokenizes first then
detokenizes the returned IDs.
Addresses reviewer feedback on #9620.
Assisted-by: Claude:claude-sonnet-4-6
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
* fix(kokoros): implement detokenize in the Rust backend service
The Detokenize RPC added in this PR grows the tonic-generated Backend
trait. Unlike the other languages there is nothing to inherit a default
from — Rust trait impls must list every method — so
backend/rust/kokoros failed to compile:
error[E0046]: not all trait items implemented, missing: `detokenize`
--> src/service.rs:72:1
72 | impl Backend for KokorosService {
Go backends pick up the Unimplemented default from base.Base, and the
generated C++/Python servicer bases default to UNIMPLEMENTED, which is
why the Rust backend was the only one that broke. kokoros is the sole
Rust crate in the tree, so this is the full extent of the fallout.
Return Status::unimplemented("Not supported"), matching how this same
file already gates tokenize_string and ~20 other unsupported RPCs.
Fixes the tests-kokoros and backend-jobs-singlearch-4 (-cpu-kokoros)
failures on the previous head.
Assisted-by: Claude:claude-opus-5 cargo
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
---------
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
On 2026-07-30, 12 of the 23 queued runs of this workflow were commits like "add
1 new model to gallery" or a docs fix, each rebuilding all 18 container images.
That was roughly 216 queued jobs producing byte-identical output, in a queue
holding 1071 jobs with an oldest entry two days old.
Verified against the shipped Dockerfile before assuming it: the final stage
copies only entrypoint.sh, healthcheck.sh and the local-ai binary, there is no
go:embed of gallery/ or docs/, and the gallery is fetched at runtime from
github:mudler/LocalAI/gallery/index.yaml@master. A gallery-only commit produces
an identical image, and the gallery change reaches users through GitHub whether
or not an image is rebuilt, so nothing is delayed by skipping.
Add a `changes` job that decides once whether the push can affect an image; the
other 11 jobs take `needs: changes` and an `if:` on its output.
A job gate rather than paths-ignore on the trigger, for two reasons that both
fail silently if got wrong:
- paths-ignore on `push` also applies to tag pushes, and a tag created on an
existing commit carries an empty commits list. That would skip the release
image build with no failure anywhere. The gate short-circuits to build for
refs/tags/*, and for a base commit that is missing, zero or unresolvable --
the same run-everything posture the backend matrix filter takes for a
truncated diff.
- the merge jobs use `if: ${{ !cancelled() && ... }}`, and !cancelled() is
true when a dependency is skipped, so they need the gate named explicitly
or they would try to merge manifest lists for images never built.
Checked the decision logic against real commits from the queue: the two
gallery/docs commits resolve to build=false, the two code commits to build=true,
and all three fallback paths (tag, zero base, unresolvable base) to build=true.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Document all stateful container paths, explain upgrade behavior and UnRAID mappings, and correct the obsolete troubleshooting mount target.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
docs: clarify reverse proxy bulk job guidance
Mention ingress controllers as another place to configure equivalent upstream response timeouts, and include an example private LocalAI URL for trusted bulk jobs.
Assisted-by: Hephaestus:openai/gpt-5.5 [opencode]
Signed-off-by: Owen Adirah <owenadira@gmail.com>
Stage the negative GPU-layer sentinels expected by the upstream argument parser, then restore LocalAI resolved values unless a passthrough flag explicitly overrides them. This avoids the parser assertion that terminated the backend for any generic option.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Version-pin bumps dominate PR volume: 48 update/* PRs in the week to
2026-07-30, from 16 pins, each a two-line diff. bump_deps.yaml runs a 28-entry
matrix daily and opens one PR per moved pin; CRISPASR and the gallery checksum
produced one every day, ik-llama-cpp six in seven days. Almost all of them edit
nothing but a single backend/*/<name>/Makefile.
Neither image-pr.yml nor build-test.yaml can observe such a change. `make build`
is `go build ./cmd/local-ai`, GoReleaser builds that plus ./cmd/launcher, and
the core image's final stage ships only entrypoint.sh, healthcheck.sh and the
binary. The per-backend trees are copied into the builder but nothing in them
reaches the output. That is 7 + 3 jobs per bump PR that cannot fail for a reason
the diff caused, roughly 410 jobs a week.
Add backend/{cpp,go,python}/** to the paths-ignore of those two workflows. The
inputs that do reach the binary are deliberately outside those prefixes and so
still trigger a full run: backend/backend.proto (protogen-go), go.mod/go.sum
(the go mod tidy before-hook), and backend/Dockerfile.* .
Not applied to the workflows that genuinely read that tree:
test.yml TEST_PATHS names ./backend/go/{cloud-proxy,local-store,
valkey-store}/...
lint.yml .golangci.yml carries backend/-scoped rules
tests-e2e.yml the e2e suite drives real backends over gRPC
backend_pr.yml its whole job is rebuilding the changed backend
Simulated against the change shapes that occur in this repo. Pin bumps and
python requirement bumps skip; backend.proto, go.mod, a core Go edit, a
backend/Dockerfile edit and any mixed diff all still run, since paths-ignore
skips only when every changed file matches.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
.agents/ci-caching.md stated that `cache-to: type=registry,mode=max` "exports
the cache mount data into the registry cache, so subsequent builds restore it".
BuildKit does not do that. A `--mount=type=cache` lives in the builder's local
state and is not part of a registry cache export, and every CI job gets a fresh
runner with a fresh builder, so /root/.ccache starts empty on every build.
The compile script already prints `ccache -s` after a `ccache -z`, so the
evidence was sitting in the logs:
job 89766266951 llama-cpp cublas-13 6369s 0 / 889 hits (and 0 / 1778)
job 89766267281 llama-cpp hipblas 8160s 0 / 537 hits
job 90210828110 llama-cpp cublas-12 5673s 0 / 813 hits
The first two are the control. Their commit, 90355cd44, changed exactly one
file: backend/go/magpie-tts-cpp/Makefile, nowhere near llama.cpp. The engine
source was byte-identical to the previous build, which is the case this section
claims ccache serves, and the hit rate was still 0.00%. A restored-but-stale
cache would show partial hits; 0-of-N is an empty cache.
So llama-cpp, ik-llama-cpp, turboquant, bonsai, ds4 and privacy-filter pay the
ccache wrapper overhead and get nothing back, and multi-hour C++ builds
recompile identical translation units every time.
Record this rather than silently extending it. Wiring the same mount into
Dockerfile.golang, which covers 215 of the 434 matrix entries, measured 18%
faster locally on a rebuild after a source edit with a 71.5% hit rate, but only
because that test reused a single builder across both builds. In CI it would be
a no-op. The note spells out what would actually work (ccache remote_storage or
sccache with a real backend, or round-tripping the cache dir through
actions/cache) and what each costs.
Also correct the composite-actions list, which still named test.yml as a
free-disk-space consumer after #11219 removed it.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Dockerfile.golang builds 215 of the 434 matrix entries: every ggml/C++ engine
wrapped in Go. Each of those Makefiles clones an upstream repo at a pinned SHA
and compiles it once per SIMD variant (depth-anything-cpp builds four: avx,
avx2, avx512, fallback), and those variant targets depend only on the clone.
They cannot observe a change anywhere else in the LocalAI tree.
The compile sat below `COPY . /LocalAI`, so any edit anywhere invalidated it and
recompiled C++ that had not changed. Move it above that COPY, behind a copy of
only the backend's own directory.
This lands the compile in the part of the image the registry cache already
restores. Measured on two real CI builds of this Dockerfile (jobs 90551029008
and 90551028904, both fresh runners): 13 of 17 layers CACHED from
quay.io/go-skynet/ci-cache. The uncached tail is exactly `COPY . /LocalAI`, the
git-config RUN and the build RUN. Putting the engine above the COPY moves it
from the uncached tail into the cached region.
Local measurement, depth-anything-cpp CPU, rebuild after editing a Go file
outside the backend:
master 78s, 216 C++ objects compiled
this change 25s, 0 C++ objects compiled (67% faster)
Note what this deliberately is not. An earlier attempt wired a
--mount=type=cache ccache into the same RUN. BuildKit does not export cache
mounts to a registry cache, so that measured well locally and is a no-op in CI
(see the ccache section of .agents/ci-caching.md). This change relies only on
ordinary layer caching, which the 13-of-17 figure above shows already works
here.
The layer copies the backend's whole directory rather than just the Makefile:
the CMake targets also need CMakeLists.txt and the file list differs per
backend. The cost is that editing a backend's own Go sources invalidates its
engine layer. The expensive cases are unaffected, since a shared-build-input or
backend.proto change, the weekly full-matrix cron and a tag push all rebuild
every backend while touching none of their directories.
Scoped to one backend for now: only depth-anything-cpp gains the `engine`
target. The other 27 fall through the `make -n engine` guard and build exactly
as before, verified against local-store and silero-vad. Rolling the target out
to the remaining 12 backends that define VARIANT_TARGETS is mechanical once this
is confirmed against the registry cache on master.
One caveat on merge: inserting layers shifts the cache keys, so the first build
of each entry after this lands is a full miss. It pays for itself on the second.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Measured over the week to 2026-07-30, 97% of CI wall-clock is queueing and 3%
is execution: a median 5-hour queue against a 4-20 minute median job. With the
queue saturated, throughput is concurrency divided by service time, so cutting
execution time raises the drain rate directly. Two steps stood out as paying
nothing for what they cost.
test.yml: drop the free-disk-space step (~3.1min per run, ~22 h/week). That
action exists to make room for docker buildx layers and this job runs no buildx
step. It was also sized for a `make test` that downloaded multi-GB GGUF/whisper
fixtures and built llama-cpp/whisper/stablediffusion-ggml; the test-suite reorg
moved all of that into tests/e2e-backends and tests/e2e-aio, as the Makefile
test target already records. Its tool-cache:true wipe was additionally deleting
/opt/hostedtoolcache, forcing setup-go and setup-node to re-download toolchains
that ship preinstalled on the runner.
build-test.yaml: build only the host target on pull_request. The three-platform
cross-compile (linux/amd64, linux/arm64, darwin/arm64) is the bulk of that job's
~6.6min median, ~47 h/week, and nothing consumes a PR's binaries. goreleaser's
--single-target still runs every before-hook (protogen-go, react-ui, go mod
tidy), so the "is the release build broken" signal is unchanged. master pushes
and tags keep building all three.
Also record why the Linux Go workflows pass cache: false to actions/setup-go,
since it reads as an oversight and is not. Set up Go has a median of 11 seconds
on those runners, so there is nothing to win, and the repo already sits at
GitHub's 10 GB Actions cache ceiling with 31 entries, where each setup-go entry
is 222-375 MB on Linux and up to 1.4 GB on macOS. Re-enabling it would evict
something that is earning its space.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(vllm): apply Options[] engine flags before engine init (#11130)
CLI-style flags in a model's `options:` array (`--quantization:gptq_marlin`,
`--enable-prefix-caching`, `--kv-cache-dtype:fp8_e5m2`) were discarded: the
backend only ever read `tool_parser`/`reasoning_parser` out of Options[], and
did so *after* `AsyncLLMEngine.from_engine_args()`, where nothing it set could
still reach the engine.
Map `--` prefixed options onto the AsyncEngineArgs dataclass before the engine
is constructed. Names are normalized the way vLLM's CLI spells them
(`--enable-prefix-caching` -> `enable_prefix_caching`), values are coerced to
the target field's type (bare flag -> True for booleans), and unknown or
uncoercible flags warn and are skipped instead of failing the load, since
Options[] is a bag shared with backend-level settings. Field types come from
the annotation's base so `Literal["auto", "float16"]` (vLLM's dtype) is not
mistaken for a float.
Precedence is typed proto fields -> `options:` -> `engine_args:`. The
production engine_args defaults seeded in hooks_vllm.go therefore skip any key
the user already set as an option, otherwise the later engine_args pass would
silently override it. Parser lookups now accept both spellings, so
`--reasoning-parser:qwen3` selects LocalAI's parser as well.
The helper's tests are stdlib-only and run in the lint workflow's
dependency-light job via `make test-python-helpers`.
Assisted-by: Claude:claude-opus-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
* backend(audio-cpp): add the native build scaffold
Links 0xShug0/audio.cpp engine_runtime through its public framework headers
and serves Health/Status. Model loading and the audio RPCs follow.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): keep the build-tree rpath at $ORIGIN
Upstream sets CMAKE_BUILD_WITH_INSTALL_RPATH in its own directory scope, so
CMake was appending its build-tree library dir to our target and baking an
absolute build-host path into the shipped binary. Set BUILD_WITH_INSTALL_RPATH
on the target so a package that forgets to bundle libggml*.so fails on the
build machine too, instead of only on a user's box.
Also document why EXCLUDE_FROM_ALL must stay on the add_subdirectory call,
correct the claim that Ubuntu ships no gRPC CMake config, stop the pin comment
from repeating the assignment token that bump_deps.sh rewrites, and make
test-engine fail rather than pass when no test is registered.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): parse namespaced model options
Splits option entries on the first colon so path values survive, and routes
load./session. prefixes to the upstream load and session option maps.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): reject out-of-range numeric model options
std::atoi is undefined once the digits exceed long and in practice wraps, so
device:2147483648 was accepted and handed the ggml backend selector a device
index of -2147483648 from a function whose error text promises a non-negative
integer. Parse with strtol and reject on ERANGE, on a value above INT_MAX, and
on any unconsumed trailing input. The error strings are unchanged.
Name the whole entry in the unknown-key error too: an entry like ':value' has
an empty key and left the user nothing to grep for in their YAML.
Tests look keys up through a helper instead of map::at, so a prefix off-by-one
fails one named check rather than aborting the binary and skipping the rest of
the suite, and cover the overflow, negative and non-numeric paths.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): route LocalAI RPCs onto audio.cpp tasks
Task-major resolution over the family's advertised capability set, with the
voice-reference and instructions signals selecting cloning and voice design,
and a streaming-to-offline fallback for server-streaming transcription only.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): use upstream's 'spk' task name and pin the preference order
The SpeakerRecognition short name was 'spkrec', which audio.cpp neither prints
nor parses; a name copied out of audio.cpp was rejected and a pinned 'spkrec'
would not survive the engine boundary. Emit 'spk', keep 'spkrec' as an
input-only alias, and correct the known-tasks lists.
Three assertions were vacuous because their fixtures advertised a single task,
so reversing a preference order or dropping the RPC name and the attempted
pairs from the capability error all passed. Give them fixtures that can tell
the orderings apart.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): convert sample, time and PCM units
Integer nanosecond conversion so 44.1 kHz stays exact, float seconds for the
VAD and diarization messages, and saturating s16le encode so an overshooting
sample cannot wrap to the opposite sign.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): harden seconds_to_samples against NaN and overflow
seconds_to_samples is the one entry point fed by untrusted-shaped input: a
float-seconds timestamp off the wire, or a boundary from a model that diverged.
Its guard covered only the low side, so NaN and out-of-range values fell through
to an undefined double-to-int64 cast and came back as INT64_MIN. A hugely
negative sample index used later as an offset or a length is a wild pointer
rather than merely a wrong timestamp. Reject NaN with the !(x > 0) form and
saturate before the cast.
Also round instead of truncating there. These functions exist to cross the float
seconds boundary the VAD and diarize messages use, and truncation lost a sample
about half the time on the samples-to-seconds-and-back round trip, starting at
n=1.
Pin the decode scale at INT16_MIN, pin nanosecond truncation on a nonzero
fraction, and record why the clamp argument order in f32_to_s16le is
load-bearing for NaN.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): map NaN PCM samples to silence explicitly
f32_to_s16le relied on std::min argument order to keep a NaN sample away from
std::lround, whose result is unspecified for NaN. That was too subtle to rest on
a comment, and the comment was itself wrong: it warned against a spelling that
the outer std::max already catches, while three real spellings leak, including
std::clamp, which is the idiomatic C++17 way to write the same clamp and so the
likeliest future edit.
Divert NaN before the clamp and encode it as 0. A NaN sample rendered as a
full-scale click is worse audio than a dropped one, and this unit converts audio
that may have originated off the wire.
Pin it with an exact-value check rather than a range check, since all three
outcomes the plausible spellings produce are finite and inside full scale, plus
an invalid-operation check that fails unless the NaN is diverted before any
ordered comparison. That second check is what catches modernizing the clamp and
dropping the guard together.
Also bound the seconds round-trip comment, which claimed unconditionally what
holds only below roughly 2^23 samples, and document NaN, saturation and that
bound in the header.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): assemble transcripts from runtime spans
The top-level transcript text is TaskResult.text_output verbatim. audio.cpp
carries text nowhere else: speech_segments, speaker_turns and word_timestamps
hold spans and labels only, so deriving the text from them empties the
transcript for any producer that omits word timing, VibeVoice diarized ASR
included. Fixtures cover every observed producer shape.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): keep a nested speaker turn's own label
A segment sourced from speaker_turns re-derived its speaker by greatest
overlap. A turn's overlap with its own span is the largest possible, so a turn
nested inside another speaker's turn could only tie with the container, and the
tie went to whichever came first. sortformer_diar binarizes each speaker
independently and sorts by start sample, so the container always comes first
and the interjecting speaker was silently erased from DiarizeSegment.speaker.
choose_segment_spans now carries the label out with the span.
Also pins the nearest-segment fallback against measuring from either endpoint
or from segment position, which a trailing-only stray word could not do, and
exercises the empty-word guard in join_words. Two fixtures that pin a rule but
do not mirror any pinned family are relabelled defensive.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serialize runs with a wedge-aware guard
audio.cpp sessions are not reentrant and a wedged CUDA call cannot be
cancelled, so a plain mutex would pile every worker thread behind a stuck GPU.
Callers waiting past the configured bound, or arriving while the holder has
already overrun it, fail fast instead.
A caller that queues behind a healthy run deliberately does not stamp the
clock: only the thread that takes the lock does. Stamping on arrival would
restart the wedge clock on every request and hide a stuck run from everyone
behind it, which is the pile-up this guard exists to prevent.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serialize inference through an InferenceLane
One audio.cpp model is loaded per backend process and its sessions are not
reentrant, so concurrent gRPC handlers have to take turns. Serialization alone
is not enough: a wedged GPU call cannot be cancelled from userspace, so an
unbounded queue behind one stuck run would swallow every gRPC worker thread
until the process is useless.
InferenceLane gives handlers a lane with room for one runner. LaneEntry occupies
it for a scope and gives it back on every exit, including an exception, and is
the only way to take the lane at all: occupy/vacate are private with LaneEntry
as the sole friend, so a caller cannot acquire without holding something that
releases. LaneEntry is immovable on purpose, because a moved-from entry would
have to stop releasing while the lane still recorded it as occupied.
A caller either waits indefinitely or brings a millisecond budget. A bounded
caller that cannot get in fails instead of waiting on, and a bounded caller
whose budget is already shorter than the age of the run in the lane fails
immediately, which is what stops a queue forming behind a wedged run. The two
failures carry different text: one names the wait it exhausted, the other states
the measured age of the run without claiming to know why it is long, since a
short budget meeting a legitimately long run lands there too.
The run's age is stamped only after acquisition. A waiter that published itself
as holder would restart the measurement and hide a genuinely stuck holder from
every caller behind it.
Budget negotiation and the overrun decision are pure functions taking their
inputs explicitly, so both are covered without threads or sleeping. The
per-model ceiling arrives as an int of milliseconds; a request may tighten it
and may never loosen it.
Replaces the previous run_guard unit, which was a derivative of an
Apache-2.0 file upstream and could not stay in an MIT tree. Written from a
behaviour contract with no reference to the removed code.
Tests: 65 checks, standard library only, single translation unit, clean under
-Wall -Wextra. Mutation tested at 23/23 killed; two of those mutants exposed
missing coverage and the tests were extended until they died. ThreadSanitizer
clean.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): make the B10 test able to fail, and document LaneEntry
Review of the previous commit found the B10 test could not fail for the reason
it was named. It aged the in-flight run to about 120 ms and then tried two
budgets, 30 ms and 60 ms, both under that age, so both callers took the
fail-fast path. "The two failure modes do not share one message" was comparing
two fail-fast messages that differ only in the budget they print, and the
timeout path was never reached. The second budget is now 400 ms, well over the
run's age, so that caller queues and times out, and a new check asserts which
path each caller took instead of inferring it from inequality. A mutant that
makes the fail-fast path emit the timeout message previously died only on B4 and
B8 checks; it now also dies on B10.
Comment-only changes elsewhere. LaneEntry now says it is not reentrant and does
not detect reentrancy: a second entry on a thread that already holds the lane
surfaces as LaneUnavailable with a positive budget, but parks silently in
unbounded mode, which matters because a handler may hold one across a whole
stream. The immovability note now names the shapes that work, an optional
emplaced in place or a unique_ptr, rather than saying to hold the entry
indirectly without saying how; all three documented forms were compiled before
being written down, which is how the note came to say that an optional of an
immovable type cannot itself be returned.
The header's explanation of why fail-fast exists is reworded. Two clauses traced
back to a specification written after reading the Apache-2.0 upstream header,
and while that was judged de minimis, this unit was rewritten precisely to carry
no upstream expression at all.
The margin table in the report was also wrong about which wall-clock margins are
load-sensitive: there are four, not one, and the tightest is the B3 arrival
check, which is now flagged at the call site. No margin value changed and none
moved across 65 runs.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): gate model loading on the audio.cpp family
Refuses any GGUF without an audiocpp.model_spec.family key and any non-GGUF
path without an explicit family option, so the model loader's greedy backend
probe cannot bind an unrelated llama.cpp GGUF to this backend (#9287).
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): load models and cache sessions per task
Loads one ILoadedVoiceModel and creates an IVoiceTaskSession lazily per
(task, mode), so the same model serves both the unary and streaming RPCs.
LoadModel derives the family from GGUF metadata or an explicit option and
fails with INVALID_ARGUMENT otherwise, so a failed load is a gRPC error the
backend probe can see.
audiocpp_backend::Task mirrors engine::runtime::VoiceTaskKind positionally,
and drift there is silent: every unit still compiles and every test still
passes while the backend runs a different task. Two mechanisms pin it. The
static_asserts in loaded_model.cpp catch an insertion or a reorder, and
-Werror=switch on that one file turns an appended upstream enumerator into a
build failure rather than a warning in a 600 file log.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): stop aborting the process on SIGTERM
The signal handler called grpc::Server::Shutdown directly. Shutdown takes an
absl::Mutex, which is not async-signal-safe: the handler can interrupt a thread
already holding that mutex, and abseil's deadlock detector responds by aborting.
Every SIGTERM therefore ended in exit 134 and a 'dying due to potential
deadlock' stack rather than a drained shutdown.
The handler now sets a lock-free atomic and returns. Server::Wait moves to a
helper thread so the main thread can poll that flag and call Shutdown itself,
outside any signal context. A condition variable would not have helped, because
notifying one from a handler is not async-signal-safe either.
SIGTERM and SIGINT both exit 0 with no stack trace, where both previously
exited 134.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): correct the status, lifetime and state contracts of LoadedModel
An environment fault during session creation was reported as UNIMPLEMENTED. A
missing libggml-cpu-*.so surfaced to the client as 'family silero_vad advertises
vad/offline but refused to create the session: Failed to initialize CPU
backend', which tells LocalAI the model cannot do this and must never be
retried, and sends an operator hunting a capability bug instead of a packaging
one. A throw from create_task_session is now a plain runtime_error, so it maps
to INTERNAL. Only a null return, where the family genuinely declined, stays a
CapabilityError.
The model.'s task: option was parsed and then dropped: it lived in a local that
died at the end of LoadModel and had no route to RequestShape::pinned_task.
LoadedModel now keeps it and exposes pinned_task().
The global model becomes a shared_ptr reached through snapshot(). An audio RPC
runs for seconds and cannot hold g_model_mu for its duration, so under a
unique_ptr a Free arriving mid-request would destroy the model underneath it.
Handlers now take a counted reference and whichever finishes last does the
teardown, outside the lock.
session_for documents the streaming state contract rather than resetting the
session itself. Resetting on a cache hit was tried first and is not possible:
silero_vad throws 'session prepare() must be called before Silero VAD reset()',
so it would turn an ordinary second fetch into a hard error. start_stream's base
implementation is already a reset, so a caller that runs prepare then
start_stream per stream gets a clean session; a probe against the bundled
silero_vad confirms an identical replay when it does and a carried-over stream
when it does not.
Also: an unknown backend: name is rejected before the model loads rather than
after; MainGPU is parsed instead of passed through std::atoi, which turned
'gpu1' into device 0 silently; and device carries a device_set flag, because 0
is both the default and a real device index, so MainGPU was overriding an
explicit device:0 that the neighbouring threads: handling promises will win.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serve the VAD and Diarize RPCs
Both emit float seconds, converted from the runtime's sample-index spans, and
both take a counted reference to the loaded model through snapshot() and hold it
for the whole call: a Free arriving mid-request drops only the global's
reference, so whichever request finishes last destroys the model instead of one
of them running on freed weights. An AddressSanitizer build reproduces exactly
that heap-use-after-free inside ggml_vec_dot_f32 when the handler keeps a raw
pointer instead, which is why the shape is what it is.
The inference lane is taken before session_for, not after. session_for reads and
writes an unsynchronised session cache and the offline run calls prepare(),
which mutates the session, so both belong inside the lane.
Diarize routes before it reads the input file, so a family that cannot diarize
at all says so rather than complaining about the audio first. Its per-segment
text stays empty because audio.cpp's SpeakerTurn carries a span and a speaker
label only, and nested or overlapping turns are passed through untouched: a
sortformer turn inside another speaker's turn is correct output for overlapped
speech, and LocalAI is overlap-tolerant downstream. Duration counts frames
rather than floats, so a stereo input does not report twice its length.
Verified end to end against upstream's bundled silero_vad, which needs no
download, using the bundled 16 kHz speech asset: a synthetic tone returns
nothing, correctly, because silero detects speech and a sine is not speech.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): enforce ModelIdentity on VAD and Diarize
audio-cpp was the only C++ backend without the model-identity guard, and no
later task in the plan added it. pkg/grpc/server.go enforces checkModelIdentity
on exactly these two RPCs, for the reason #10952 records: in distributed mode a
worker can recycle a stopped backend's gRPC port for another model's backend,
and the controller's liveness-only probe cannot tell a stale cached route from a
live one. Without this guard a stale route gets a different model's VAD or
diarization answer back with a 200.
The loaded identity lives on LoadedModel rather than in a separate global, which
is where this differs from llama-cpp. A handler holding the model through
snapshot() then necessarily judges against the identity that model was loaded
with, and a concurrent reload cannot swap one without the other. The refusal is
NOT_FOUND carrying the verbatim grpcerrors.ModelMismatchSentinel substring.
session_for and run_offline now take a const LaneEntry & proof-of-holding
parameter. The rule that both must run under the inference lane was prose, which
is exactly how the plan came to specify the inverted order; it is now a compile
error. Restoring the inverted order fails to build rather than racing on an
unsynchronised session map with a mutating prepare().
Diarize's speaker-hint comment claimed the dropped hints were "not a silent
failure". From the caller's side that is what they are, and backend.proto
documents num_speakers as forcing, so the comment now says plainly that the
forwarding is dead for sortformer and that the family which lands must either
honour num_speakers or refuse it. read_audio_file inspects the error_code from
exists(), so an unsearchable parent directory no longer reports as a missing
file. The VAD handler records the stimulus that actually works, since silero
correctly ignores synthetic tones and the next task would otherwise rediscover
that.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): make the lane and identity guards structural
Two hardenings ahead of the eleven handlers still to be written, both of which
get harder to retrofit later.
The lane proof-of-holding parameter was a const reference, which binds to a
temporary, so session_for(rpc, shape, model->acquire(0)) compiled. Each such
temporary dies at the end of its own full-expression, releasing the lane between
two calls that must share one: precisely the split the parameter exists to
prevent, and the form a future author is most likely to reach for because it
reads as tidy. A non-const reference requires an lvalue, so the temporary form
now fails to compile while the named-local handlers build unchanged. The header
comment no longer implies the check is total either: it proves a lane was taken,
not that it is this model's lane.
The identity check was two lines each handler had to remember, with nothing
failing if a new one forgot them and no C++ equivalent of
model_identity_modalities_test.go to notice. snapshot() becomes
snapshot_unchecked(), whose only legitimate caller is Status, since HealthMessage
carries no ModelIdentity. Handlers go through snapshot_for(), which takes the
counted reference, refuses when nothing is loaded, and runs the identity check
before anything can route. Every handler already has to call something to obtain
the model, so the guarded call is now the shortest path and skipping it means
deliberately typing snapshot_unchecked. A convention that has to be remembered
can rot; this cannot.
Verified: the temporary-argument and inverted-order forms each fail to compile
with the expected diagnostic, the real handlers build, and bypassing the guard in
Diarize alone turns the identity test red on that RPC while VAD stays green.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serve the AudioTranscription RPC
Adds result_map, the engine-to-proto boundary, and wires the offline
transcription RPC.
The handler branches on the ROUTED task: for Asr the request's prompt is
whisper-style decoding context and becomes a request option, for Alignment
the same field IS the transcript to align and becomes the text input.
Routing has already decided which.
The result text is TaskResult.text_output verbatim and is never derived
from the segments. audio.cpp carries transcript text in text_output and
nowhere else, so deriving it returns an empty transcript for every
producer that reports segments without word timing. transcript_assembly
already enforces that; this commit's job is not to undo it at the proto
boundary, and result_map_ctest pins it there.
read_audio_file now takes the sample rate the caller needs. Both file-fed
speech handlers ask for 16 kHz mono, for two reasons: silero_vad and
sortformer_diar refuse anything else outright, which turned an ordinary
44.1 kHz upload into INTERNAL, and nemotron_asr emits word timestamps in
its own 16 kHz feature domain whatever the input was, so only a 16 kHz
buffer makes the emitted nanoseconds right. Zero keeps the file's native
rate and channels, which is what source separation will need.
LoadedModel::check_can_serve answers a capability refusal before the lane
is taken and before the input file is read. Routing is a pure read of the
immutable capabilities, so a model that cannot serve an RPC no longer
waits out somebody else's run to say so. VAD and Diarize use it too.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): stop linking sentencepiece's vendored protobuf
engine_runtime links sentencepiece, whose default SPM_PROTOBUF_PROVIDER
builds the protobuf-lite 3.14.0 sources it vendors. The generated
backend.pb.cc is built against the toolchain's protobuf 3.21.12. Both
ended up in the binary: 476 google::protobuf:: symbols came from the
archive, 278 of them also defined by libprotobuf.so, and the archive won,
because once ld pulls a member in for sentencepiece's own code every
reference binds to the definitions that member carries.
The visible symptom is one function.
ParseContext::ParseMessage(MessageLite*, const char*) is what a generated
_InternalParse calls for a submessage field and for nothing else, so flat
messages parsed and nested ones did not: a TranscriptResult carrying
segments serialized to correct bytes that the same process could not read
back, and TranscriptLiveRequest, a oneof of submessages, could not have
been parsed at all. Underneath that, 3.21 generated code was running 3.14
arena, ArenaStringPtr and ExtensionSet code.
-Wl,--exclude-libs does not fix it. It makes those symbols LOCAL in
.dynsym and the parse still fails, because the binding was decided at
static link time and no visibility flag revisits it.
Setting SPM_PROTOBUF_PROVIDER to "package" before add_subdirectory points
sentencepiece at the protobuf the generated code was already built
against. Zero google::protobuf:: definitions remain in the executable
afterwards, every nested message round trips, and citrinet_asr, which
parses a SentencePiece ModelProto at load time and would break first if
this were wrong, still tokenizes and transcribes correctly.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): fix the segment text a transcription response is built from
Segment text is not decoration. core/http/endpoints/openai/transcription.go
routes response_format text, srt, vtt and lrc through
schema.TranscriptionResponse, which builds the entire body out of
Segments[].Text and never reads the top-level text. So for those four
formats the segment text IS the response.
nemotron_asr emits one word_timestamp per SentencePiece token, and the
word boundary is carried as a LEADING SPACE on the piece ("So", "me",
" call"). join_words inserted a space unconditionally, so
response_format=text returned "So me call me na ture ," while the
correct sentence sat unread in the top-level field. The separator is now
chosen from the words themselves: whole words are space-joined, subword
pieces are concatenated, and one leading space anywhere selects the
latter. Concatenating the real nemotron pieces reproduces text_output
exactly, verified end to end.
This does not touch the top-level text, which is still text_output
verbatim. The rule that forbids deriving the transcript from the segments
is about the direction segments -> text; segment text has no source other
than its words.
Two smaller corrections in the same area:
timestamp_granularities ["word"] set only "word_timestamps", a key no
family in the pinned upstream reads. It now sets "return_timestamps",
which qwen3_asr does read and which both runs its forced aligner and
shortens its chunk window, so asking for word granularity no longer
silently returns nothing.
The request-option comment claimed more than it delivered. prompt,
translate and temperature are read by no ASR family, and are forwarded
only so a family adopting them works unchanged; the comment now says so
per key, and gives TranscriptRequest.diarize the same explicit treatment
threads already had.
Also: the shipping target now carries -Wall -Wextra -Wpedantic, which it
never did, so "the build is clean" starts meaning something; and
fill_transcript_result no longer swallows a null response pointer, since
answering OK with an empty transcript is the one failure mode this unit
exists to prevent.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serve the AudioTransform RPC
Covers voice conversion, singing voice conversion, speech to speech and source
separation, the four tasks LocalAI's AudioTransform can represent.
AudioTransformResult carries one dst while htdemucs and mel_band_roformer
produce several named stems from a single run, so inference runs ONCE, every
stem is written as a sibling file <dst-stem>.<name>.<ext>, and params[stem]
selects which one dst receives, defaulting to vocals and falling back to the
first output. An unknown stem name is INVALID_ARGUMENT listing the real stem
names rather than a silent substitution, and the selection happens before the
first write so a refused request leaves no files behind. params[stem] is
consumed here and is not forwarded into the engine's request options.
The stem decision lives in stem_selection, which is stdlib only and therefore
tested by backend/cpp/run-unit-tests.sh. It also validates the names, because
they come from the model (htdemucs reads them from the GGUF's config.sources)
and each becomes a component of a path this backend writes: a name carrying a
path separator would escape the caller's output directory, and two stems
sharing a name would silently overwrite one another.
Both files are read at their native rate and channel count. Separation forces
it, since demucs and roformer refuse any rate but 44.1 kHz and lose the stereo
image that separates a centred vocal from a wide mix. The conversion families
all resample internally (seed_vc, vevo2, miocodec, chatterbox were each
checked), so passing the file through unchanged is also strictly better than
band limiting it to 16 kHz first.
Verified end to end against htdemucs f16 on a 44.1 kHz stereo mix: four stems
plus dst, dst byte identical to the selected stem, params[stem] selecting a
different one, an unknown stem refused with no files written, and mono input
preserved as mono output. Also against miocodec for the single output path,
where params[stem] is refused rather than ignored.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): refuse an impossible stem early, and stop blaming the caller for a failed write
Four fixes from the first review of the AudioTransform RPC.
check_can_serve now returns the resolved route, so params[stem] on a route that
is not source separation is refused from the route instead of after a full
inference: 11 ms rather than the 4.5 s a miocodec conversion costs, and far
worse on seed_vc or vevo2. The post-run refusal stays as the backstop for a
separation-routed family that returns no stems anyway. The typo'd-stem-name
case still needs the run, since no framework header publishes the stem names
before one.
Stem names carrying control bytes are refused. GGUF strings are length prefixed
and demucs reads its sources from JSON, so an embedded NUL survives to here:
two names differing only after the NUL are distinct std::strings, so the
duplicate check passes them, and then path::c_str() truncates both and they
open the same file. That is exactly the silent overwrite the duplicate check
exists to prevent, with the .wav lost as well.
A failed write is now INTERNAL rather than INVALID_ARGUMENT. The destination is
LocalAI's own generated-content directory, not anything the caller named, so a
full disk or a permission fault there is a server fault and is worth retrying,
which is the opposite of what INVALID_ARGUMENT tells a client. An empty output
path stays INVALID_ARGUMENT.
Two comment corrections and one clarification: the separators' required rate is
their checkpoint's declared samplerate rather than a hardcoded 44100, seed_vc
resamples with soxr and falls back to sinc-hann, and the "no files left behind"
guarantee covers a refused request, not a write that fails partway through the
loop.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(audio-transform): stop folding every upload to 16 kHz mono, and name the separation stems
Two defects that made source separation unusable through LocalAI's own API,
even though the backend served it correctly over gRPC.
/audio/transform normalized every upload to 16 kHz mono s16 through
utils.AudioToWav, with no way past it. htdemucs and mel_band_roformer refuse
any rate but their checkpoint's own and separate a centred vocal from a wide
mix using the stereo image, so every separation request through the HTTP API
died with "HTDemucs prepare() sample rate mismatch: expected 44100, got 16000"
while the same call over gRPC worked. The fold is not wrong, it is
backend-specific: LocalVQE's echo cancellation genuinely wants 16 kHz mono and
needs the reference in the same shape. So it becomes a declaration,
BackendCapability.AudioTransformInputMono16k, set for localvqe and for nothing
else. A backend that declares nothing gets its upload unchanged, which means no
backend has to opt in to work. utils.AudioToWavPreservingShape is the
non-folding conversion: a 16-bit PCM WAV passes through byte for byte at any
rate and channel count, anything else is transcoded to WAV with its rate and
channel layout kept.
The other defect is that the run-once stem design bought nothing. A separation
backend writes every stem beside dst from one inference, but AudioTransformResult
carried only dst, so the other three were files no caller could find and a
caller wanting all four had to run four separations. AudioTransformResult grows
a repeated AudioTransformStem, the backend fills it, core/backend validates that
each path really is inside the generated-content directory it handed over, and
the endpoint publishes them as an X-Audio-Stems JSON header beside the existing
X-Audio-Input-Url. JSON because a stem name is the model's own string and could
contain any separator a hand-rolled format would use.
Verified end to end through the HTTP endpoint with htdemucs f16 on a 44.1 kHz
stereo file: 200 with a 44.1 kHz stereo body, all four stems named and fetchable
through /generated-audio/, body byte identical to the selected stem, and
params[stem]=drums returning a different one. The same upload sent to a model
whose backend is localvqe still reaches the backend as 16 kHz mono, confirmed
both by the engine's own rate refusal and by the persisted input file.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(audio-transform): reject extensible WAV from the passthrough, escape stem URLs, convert stems with dst
Four fixes from the second review, plus one bug they made visible.
isPCM16Wav tested only the bit depth, and go-audio's IsValidFile never looks at
the format tag, so a 16-bit WAVE_FORMAT_EXTENSIBLE (0xFFFE) upload was passed
through untouched where the old fold would have transcoded it. audio.cpp's WAV
reader accepts 16-bit only when the tag is 1, so such a file died with
"unsupported WAV encoding". Extensible is what many DAWs and Windows tools
write and music files are this endpoint's new headline input, so it is a
first-contact failure rather than a corner. The check now requires tag 1, with a
spec that fails against the old implementation.
Stem URLs are percent-escaped. A stem name is the model's own string and legally
contains a space, a '#', a '?' or a '%'; an unescaped '#' truncates the URL
before the request is even sent. The name field keeps the raw name.
sample_rate and response_format are applied to the stems as well as to dst.
Applying beat documenting: dst IS one of those stems, so leaving them alone
broke the "dst duplicates the selected stem" invariant the whole design rests
on, and both conversions are no-ops when unset. A stem whose conversion fails is
dropped from the header rather than advertised in the wrong shape.
Verifying that turned up why it had never been noticed: the two fields were
never bound at all. The request arrives as multipart/form-data and echo's binder
falls back to the FIELD NAME without a form tag, matching only
case-insensitively, so "SampleRate" never matched "sample_rate" and "Format"
never matched "response_format". Both were documented in the endpoint table and
silently ignored. Two form tags fix it, and with them the conversion is
observable end to end.
Docs: audio-transform.md now documents what LocalAI does to an upload before the
backend sees it, which backend gets the 16 kHz mono fold and why, params[stem],
and the X-Audio-Stems header with a worked example.
Also records the known limitation that the fold lookup is on the bare backend
name, so pinned variants (vulkan-localvqe) do not match, and points at
IsLlamaCppBackend as the suffix-tolerant precedent.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serve the TTS and SoundGeneration RPCs
TTSRequest.voice is treated as a speaker reference clip when it names an
existing regular file, which makes routing prefer VoiceCloning, and as a named
preset otherwise, in which case it travels as VoiceReference::cached_voice_id.
Both the clip and SoundGenerationRequest.src are read at the file's own rate and
channel count: upstream's own CLI and server do exactly that, every consuming
family resamples internally and mostly with a better resampler than ours, and
ace_step and stable_audio resample their input per channel, so a downmix here
would delete the stereo image they are built to consume.
The request builders live in their own unit rather than in grpc-server.cpp's
anonymous namespace so they can be tested; grpc-server.cpp has a main() and
cannot be linked into a test binary. The option keys are the whole point of
these functions, so each one was grepped against the pinned upstream and the
accounting is written down beside it. instructions maps to "instruct", which is
what upstream's own server maps the OpenAI field to and what qwen3_tts and
omnivoice read, and to "caption" for irodori_tts; the style tag is spelled
"instruct" too, because "instructions" is looked up nowhere. duration maps to
"duration_seconds", read by all three generation families, with the proto's own
name kept only as a forward-tolerant alias. Keys that no family reads say so.
Both handlers answer a capability refusal before taking the lane and before any
file read, so a model that cannot synthesise does not queue behind somebody
else's run to be told no.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): stop emitting an empty style language, and name the missing clip
StyleCondition::language was set whenever has_language() was true, with no
!empty() guard, while the language option twelve lines below had one.
core/backend/tts.go sets Language unconditionally, so has_language() is true on
every request LocalAI sends and carries "" when the caller named none. An
engaged-but-empty style language is worse than an absent one: supertonic reads
text_input->language behind its own !empty() guard and then overrides it from
style->language with no guard at all, so "" replaced its "en" default and its
tokenizer threw "invalid Supertonic language: ". Every /v1/audio/speech request
that set instructions and no language would have been an INTERNAL against a
supertonic model. A plain request never saw it, because the style condition only
exists when instructions are non-empty, which is why the chatterbox end to end
run did not catch it.
TTS also stops discarding the Route that check_can_serve already returns. A
family routed to voice cloning without a reference clip used to be refused from
inside its own prepare(), which meant an INTERNAL naming neither the RPC nor the
field to set; chatterbox advertises clon and no tts, so that was every
preset-only request to it. It is now an INVALID_ARGUMENT naming
TTSRequest.voice, answered in about 4 ms, and it cannot misfire because
has_voice_reference is what selected cloning in the first place. Reading
CapabilitySet::supports_speaker_reference to generalise this stays a follow-up.
The src read carries a written caveat rather than a family blocklist, because
ace_step's editing routes legitimately need src: setting src on a stable_audio
model corrupts the heap and aborts the process in the pinned upstream, and the
only thing keeping that off the network is that
schema.ElevenLabsSoundGenerationRequest has no field for it. Nobody reading that
Go schema would know why, so the reason is recorded where the field is read.
build_tts_shape is extracted so TTSStream cannot describe the same request
differently, and it arrived untested: two mutations of it survived until a
test_tts_shape case was added.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serve the TTSStream and AudioTranscriptionStream RPCs
TTSStream leads with a streaming WAV header carrying 0xFFFFFFFF sizes, matching
the convention backend/go/vibevoice-cpp established, so an HTTP client can start
playback before the full PCM exists. Its chunks are read from
StreamEvent::named_audio_outputs and not audio_output: supertonic, omnivoice and
voxcpm2 all put their streamed audio there and leave audio_output empty until
the very end, so reading the obvious field yields a stream with no audio in it.
The finish_stream result is the family's own merged whole rather than a tail, so
it is emitted only when nothing was streamed.
Streaming transcription sends incremental deltas and degrades to a single delta
plus the final result on families that offer no streaming ASR, which is the same
message sequence with fewer deltas. The four streaming ASR families disagree on
what partial_text means: nemotron_asr, vibevoice_asr and higgs_audio_stt report
incremental fragments while voxtral_realtime reports the whole hypothesis and
reports it twice, so the reconciliation lives in one tested unit rather than in
the handler. nemotron_asr reports only through the stream event sink, and only
from inside finalize, so the audio driver installs one and clears it again
before returning: the session is cached and a sink left holding the caller's
frame is a use after free waiting for the next stream.
begin_stream is now the only implementation of the streaming state obligation,
prepare then start_stream. Streaming sessions are cached, and what clears the
previous stream is start_stream's reset; a family override that dropped it would
break every call site with no compile error, so there is one call site.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): keep streaming deltas on UTF-8 boundaries, refuse dtypes that abort
TranscriptStreamResponse.delta is a proto3 string, whose wire format requires
valid UTF-8. voxtral_realtime reports its hypothesis as a concatenation of raw
token BYTES (tokenizer_text.cpp:171-183), so the cumulative difference between
two consecutive reports is eventually a lone continuation byte, and the C++
runtime serializes that with only a logged warning while the Go runtime refuses
to unmarshal it: the client loses the remaining deltas AND the final_result.
Measured on a trace of a non-ASCII sentence, 11 of 31 messages failed to
unmarshal and every accented character was lost. TranscriptDeltaTracker now
holds back an incomplete trailing sequence and merges it into the next fragment;
reconcile flushes it, which it always can because the final text is complete.
The same trace now unmarshals in full with zero failures.
A streaming buffer whose float count is not a whole number of frames is refused
rather than truncated. The integer division dropped the tail floats from the fed
audio and therefore from the transcript, with no diagnostic; vibevoice_asr
refuses the same thing from the other side of the call.
A supertonic GGUF whose weights are not f32 is refused at load. It reaches
ggml_concat with mismatched operand types and ggml_abort takes the whole backend
process down on the first request, so nothing downstream can report it: the
model loads, then every request kills the process. Attributed rather than
assumed, the unary TTS path aborts identically, and upstream records that
package as untested. The refusal names the orig package and says what to run
before deleting the guard.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): stop a repeated lead byte from orphaning the next delta
The first UTF-8 fix closed the cumulative half only. Rule 2 discards a fragment
the known text already starts with, and when that fragment is the LEAD BYTE of a
new character it looks exactly like a repeat of an older character beginning
with the same byte. It was discarded rather than held, its continuation bytes
then arrived alone and began the next delta, and utf8_complete_prefix_length
only ever inspected the trailing sequence, so a delta invalid at the FRONT went
out whole. Through a real Go proto.Unmarshal the review's four-character repro
gave 3 deltas, 2 unmarshal failures and a lost transcript.
Reachable from the incremental families, not only from voxtral: nemotron_asr's
decoder cuts at a byte offset and vibevoice_asr's common_prefix_size compares
bytes, so both split characters. Measured over 30,000 randomized incremental
traces, 53.28% of Japanese traces and 9.52% of French ones carried at least one
delta the Go runtime refuses.
Two changes. Rule 2 no longer judges a fragment that ends mid-character, so the
lead byte is held instead of swallowed and the character survives intact; the
cost is a few duplicated bytes in a shrinking cumulative report, which no pinned
family produces. release() additionally drops leading orphan continuation bytes,
so no delta can begin mid-character whatever the rules above it decide. Losing a
byte keeps the stream alive; emitting one ends the RPC and takes the
final_result with it.
Post-fix all 60,000 traces produce zero unmarshal failures, and the cumulative
streams plus both pure-ASCII incremental streams are byte-identical to the
previous commit, so nothing changed for the families already working.
The weight-dtype allow list moves to family_gate, where it is stdlib-only and
pinned by a test rather than only by a comment. Two comment citations corrected.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): read only an exact repeat as a repeat, not any prefix
Rule 2 discarded any partial the known text merely started with. For a cumulative
family that is a duplicate; for an incremental family it is an ordinary short
fragment that happens to coincide with the start of the transcript, and it was
dropped, silently corrupting the text. Pure ASCII, no multi-byte character
anywhere: the fragments "pure ", "ascii ", "trans", "c", "ri", "p", "t" left the
client holding "pure ascii transcrit". Over 5,000 randomized traces per
transcript, 9.50% of pure-ASCII and 29.12% of French traces ended with the client
holding something other than final_result.text, with a 200 and no diagnostic.
Both incremental families emit fragments that small routinely, since nemotron_asr
cuts at a byte offset and vibevoice_asr at a common prefix.
Narrowing rule 2 to an exact repeat drives that to zero on all six transcripts
and changes no cumulative stream at all: 30,000 randomized cumulative traces are
byte-identical to the previous commit.
What rule 2 guarded was established from upstream rather than from its own
comment. The only duplicate any pinned family produces is voxtral_realtime's,
where process_available_stream_chunks feeds each event to the sink from inside
its loop and returns the last of the batch, so that event arrives twice with
byte-equal text. A duplicate is an exact repeat, so equality still covers it. The
case given up is a cumulative report that SHRINKS, which no pinned family can
produce: voxtral decodes a token vector that is only push_back'ed and cleared by
reset(), so within a stream it can only grow.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): serve the AudioTranscriptionLive RPC
The one bidirectional stream this backend serves. The client sends a
TranscriptLiveConfig, then TranscriptLiveAudio frames; the server acknowledges
with ready, emits deltas as the audio arrives, and sends final_result once the
read side closes. There is no offline fallback: live transcription has to
consume audio incrementally, so a family with no streaming ASR is refused
rather than served a batch run, which is what this RPC's Streaming-only
mode_candidates list already says.
The driver is a new sibling of run_streaming_audio, run_streaming_live, because
the audio does not exist yet: instead of slicing a buffer it pulls frames from
the caller until the read side closes. It installs the same ScopedStreamSink in
the same order, which is not optional, since nemotron_asr returns a bare event
from process_audio_chunk and reports every partial through the sink from inside
finalize(). It buffers the wire's frames up to the family's own preferred window
rather than feeding whatever size the client's audio callback produced, and it
does not call finish_stream at all when no audio arrived, because nemotron_asr
throws "finalize requires streamed audio" and an empty transcript is the
truthful answer to transcribing nothing.
Three things the handler had to get right and one it cannot:
- The audio contract. A live request carries no samples, but nemotron_asr's
streaming prepare() throws without an audio contract, and
build_preparation_request derives it from TaskRequest::audio_input, so that
field is an EMPTY buffer holding only the rate and the channel count.
- 16 kHz or a refusal. The families express their spans in their own 16 kHz
feature domain whatever the input was, and live frames cannot be resampled
on the way in the way a file can, so an 8 kHz session would return
timestamps 2x off with a 200. core/backend hardcodes 16000 anyway.
- A mid-stream Config is refused. backend.proto calls it a decoder reset, but
deltas already on the wire cannot be retracted, so a reset would leave the
final text contradicting the transcript the client assembled. Ignoring the
message would hand a client that believes it reset the decoder a transcript
that silently continues the audio it thought it discarded.
- The stale-route identity check cannot run here: TranscriptLiveRequest
carries no ModelIdentity in either arm of its oneof, so snapshot_for does
not instantiate for it. snapshot_unchecked's comment now names that as a
second legitimate class of caller and says the fix is a proto change.
eou and eob stay false. They exist for cache-aware models that emit
end-of-utterance and end-of-backchannel tokens; audio.cpp's StreamEvent has no
equivalent signal, and a client uses eou to decide the speaker yielded the turn,
so a guess inferred from silence cuts people off mid-sentence.
The lane is held for the whole stream, which is as long as the user keeps
talking: the streaming session is stateful and cached, so a concurrent run would
interleave two callers' audio and corrupt both transcripts.
Verified against nemotron_asr over a real connection with a 14 s WAV in
512-sample frames: ready first, 59 incremental deltas with no repeated prefix,
concat(deltas) equal to final_result.text, word timestamps in nanoseconds, eou
and eob false. citrinet_asr answers UNIMPLEMENTED naming the family and listing
asr/offline. A config followed by a close returns an empty final_result rather
than hanging, and a first message that is not a config is INVALID_ARGUMENT. Two
concurrent streams both return the complete transcript.
Two cleanups on lines Task 12 touched, folded in. The DtypeAllowList terminator
is now asserted at compile time: the reported out-of-bounds read did not exist,
the single entry does terminate, but the loops have no other bound and any edit
that widened an entry would walk off the end. And the dtype guard now
short-circuits on "is there a table entry" through a new predicate rather than
on the emptiness of the description string, which would have skipped the check
on an entry with an empty allow list, i.e. on precisely the entry that refuses
every dtype.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): bound the lane a live stream can hold
AudioTranscriptionLive holds the model's inference lane for the whole stream,
which is correct (the streaming session is stateful and a concurrent run would
interleave two callers' audio) and newly dangerous. Every other RPC holds the
lane across compute, or across a write to a slow reader, and both of those
terminate on their own. A live stream instead blocks in a client-driven read,
and a peer that goes silent WITHOUT closing the stream never terminates
anything: the lane stays taken and every other request against that model queues
behind a client that stopped speaking.
live_watchdog is a one-shot idle timer that ends the stream when no frame has
arrived inside a window. It is standard library only, so it is unit tested
without an engine. gRPC's synchronous Read has no timeout and cannot be given
one, so the only way to unblock it is ServerContext::TryCancel, which decides
the wire status itself: the client sees CANCELLED rather than the
DEADLINE_EXCEEDED the handler returns, the reason is logged, and the lane coming
back is the point. When it fires the read loop throws rather than reporting
end-of-input, so the driver does not go on to finalize a decode nobody is
waiting for.
It is armed only after the lane is taken and disarmed as soon as the read side
closes, and both ends matter. Arming earlier would cover acquire(), which
legitimately blocks while another live stream runs, so a queued caller would be
cancelled for waiting its turn. Disarming later would cover our own decode,
where a window overrun is not a peer going quiet and cancelling would throw away
the transcript the client is waiting for.
The window is the new live_idle_timeout_ms option, 30 s by default, 0 meaning no
limit. core/http/endpoints/openai/realtime.go drives a 300 ms ticker and feeds
every tick that produced new audio while a turn is open, so 30 s of silence is a
hundred ticks that delivered nothing. It is also longer than any pause a speaker
takes mid-utterance, which is the case that must never be cut off, and
backend.proto lets one stream span many utterances, so a client that pauses
longer between them raises the option rather than discovering it.
Two smaller corrections in the same handler:
- check_can_serve now runs BEFORE the sample rate check.
pkg/grpc/grpcerrors/errors.go degrades to the file path on UNIMPLEMENTED and
on nothing else, so a live-incapable model asked at a wrong rate was
answering INVALID_ARGUMENT and costing the caller its fallback.
- a negative sample rate is refused instead of silently becoming 16000. Zero
still means 16000, which is what the proto documents; -1 is malformed rather
than absent and gets the same refusal every other bad rate gets.
And one thing recorded rather than changed, at the handler: "live" here means
incremental INPUT, not low latency, and with the pinned families it does not yet
mean incremental OUTPUT either. nemotron_asr's process_audio_chunk only appends
to its buffer, so its whole decode and every delta happen inside finalize(),
after the client closes its send side. The policy-window buffering is inert for
that family and matters only for vibevoice_asr and higgs_audio_stt.
Verified on the wire with live_idle_timeout_ms:3000. A silent client acked at
371 ms and was cancelled at 3.371 s; a second live stream opened one second
later received its ack 2.37 s in, i.e. at the instant the first was cancelled,
and then transcribed successfully on the same cached session. Without the
watchdog it would still be waiting. Re-ran the live transcription (ready first,
59 incremental deltas, concat equal to the final text, word timestamps in
nanoseconds, eou and eob false), the citrinet refusal at both a right and a
wrong rate (UNIMPLEMENTED either way now), and Task 12's AudioTranscriptionStream
on nemotron_asr, which is unchanged.
Mutation testing the watchdog found a weakness in its own test: the destructor
test slept past the window inside the watched scope, so a destructor that
DETACHED the thread instead of joining it passed unnoticed. The test now uses a
window longer than the scope, which kills that mutant, and says why.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): refuse the unsupported RPCs with a reason
AudioEncode, AudioDecode, AudioTransformStream, AudioToAudioStream and
VoiceEmbed have no counterpart in audio.cpp's VoiceTaskKind. Each now returns
UNIMPLEMENTED naming the loaded family, what that family does support, and the
upstream limitation, instead of the generated base class's bare status. The
reasons live in a table in capability_routing.cpp so they are data rather than
literals copied into five handlers, and so a test can assert every one of them.
The five claims this was planned against were re-read at the pinned upstream
e800d435d130dc776baf6f3e6129bb62b1495c89, and one did not hold. "audio.cpp
streams tts and asr only" is false: silero_vad advertises vad with
RunMode::Streaming. The refusal stands on the narrower claim that survives, that
no family advertises streaming for any task AudioTransform routes to, and a test
asserts the refuted wording does not come back.
VoiceEmbed is the one refusal whose request carries a ModelIdentity, so it runs
the #10952 check before answering: a stale route must get NOT_FOUND and the
router's sentinel, not "audio.cpp cannot embed speakers" about a model that is
not loaded here. It cannot use snapshot_for, whose no-model branch would tell
the caller to load a model when no model can help, so it takes the reference
through snapshot_unchecked and checks identity itself. That function's comment
now names three classes of caller instead of two.
The two bidirectional surfaces refuse without reading their stream, verified
with a client that writes a config and eight frames first and gets the status
rather than hanging.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): correct the vevo2 clause, and assert the absences
Review found a false clause in the AudioToAudioStream refusal. It said s2s is
"offline voice conversion ... which converts one clip into another speaker's
voice", which is true of miocodec and false of vevo2: vevo2's s2s route is
`editing` and only `editing` (default_route_for_task and route_matches_task in
src/models/vevo2/session.cpp), documented as "Edit source speech into new target
text while using the target voice" and requiring --target-text, so it rewrites
what was said. vevo2's voice conversion is its separate vc task. It now reads
"offline clip-to-clip processing against a target voice, declared only by
miocodec (voice conversion) and vevo2 (speech editing)", and a test asserts the
miscast cannot come back. The conclusion is unchanged: neither family converses.
That defect was undetectable on the wire, since vevo2 does not load here, which
is the argument for upstream_absence_ctest.cpp. It links engine_runtime purely
to interrogate make_default_registry() and asserts the five premises the refusal
reasons rest on: no codec task kind, no family advertising spk, no streaming for
sep/vc/svc/s2s, miocodec advertising exactly vc and s2s, and s2s advertised by
exactly miocodec and vevo2. The last two are exact sets, so an addition fails
here rather than leaving a message stale. A positive control proves the registry
is populated and the query works before any absence is believed, and every
assertion has a reproduced negative control. This turns an AUDIO_CPP_VERSION
bump from "remember to re-read five prose paragraphs" into a test failure.
unsupported_surface now switches over UnsupportedRpc with no default label, so
-Wswitch reports a sixth enumerator added without a row at build time; the
runtime bounds guard it replaces is deleted.
The AudioTransformStream reason had a true premise and an overreaching
conclusion: an offline sep family could be buffered into a stream, as other
LocalAI backends do. It now says this backend declines to offer a buffered
offline call in disguise, rather than implying impossibility.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): make the missing-switch-case diagnostic fatal
unsupported_surface() switches UnsupportedRpc onto the table row that explains
it, with no default label, so -Wswitch reports an enumerator nobody handled. As
a warning that is not enough: adding a sixth enumerator and building the shipping
target gives exit 0, a binary and one warning, and the trailing
`return surfaces[0];` then answers the new RPC with AudioEncode's codec reason.
That is a confident, specific and false statement about audio.cpp on the wire, on
the one code path whose entire job is to be truthful about what this backend
cannot do, and it is worse than the runtime fallback it replaced, which at least
named itself as a bug in this file.
capability_routing.cpp therefore joins loaded_model.cpp on the existing
-Werror=switch pin, whose comment already made this argument for the engine enum.
The comment now covers both files. The pin stays per-file rather than
project-wide because upstream's own ace_step/vae_decoder.cpp has unhandled
-Wswitch cases of its own.
Verified: a sixth enumerator now fails `make grpc-server` with exit 2 and no
binary; appending a 14th VoiceTaskKind upstream still fails loaded_model.cpp, so
the two pins fire independently; both reverted clean.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): package the backend image
Bundles the dependency closure for the from-scratch image, the dlopened ggml
CPU-variant shared objects that ldd cannot see, and upstream's bundled
silero_vad and marblenet_vad assets so VAD works with no download.
The bundled loader sits in the package ROOT rather than at lib/ld.so. run.sh
execs it, which makes /proc/self/exe name the loader, and this backend has two
consumers of that path: ggml discovers the libggml-cpu-*.so by listing
dirname(/proc/self/exe), and resolve_model_path expands bundled:<name> under the
same directory. Rooting the loader makes the binary, the ggml objects and
assets/ share the one directory all three resolution mechanisms agree on.
llama-cpp's lib/ld.so layout would need assets/ moved into lib/ as well.
The image builds against apt gRPC and protobuf, like Dockerfile.ds4 and unlike
Dockerfile.privacy-filter. The from-source gRPC that install-base-deps.sh and
the base-grpc-* images supply vendors protobuf 26, which pulls abseil into
message_lite.h; with SPM_PROTOBUF_PROVIDER=package that collides with
sentencepiece's vendored mini-abseil and every absl::internal reference becomes
ambiguous. Noble's protobuf 3.21.12 predates the abseil dependency and is the
pair every earlier verification of this backend ran against.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): exempt the driver libraries from the packaging gate
package.sh already left libcuda.so* and libnvidia-* to the host when copying,
because the driver has to match the kernel module on whatever host runs the
image, but the validation gate had no matching exemption. With BUILD_TYPE=cublas
ggml is static and links CUDA::cuda_driver, so grpc-server carries DT_NEEDED
libcuda.so.1 and the gate would have rejected the very absence the copy loop
created, failing every cublas build in CI. One regex now feeds both.
Building a control for that found a second defect: ld.so --list refuses to trace
an object with an unresolvable dependency at all, exiting 127 without emitting a
per-library line, so the "=> not found" rule was dead code and no exemption could
have applied to it. The gate now traces with LD_TRACE_LOADED_OBJECTS and
LD_LIBRARY_PATH, which reports the missing name and exits 0, and which is also
what run.sh does at run time.
Adds a layout assertion so a future move of the loader into lib/ fails the build
instead of shipping a package that resolves bundled: models into lib/assets and
finds no ggml CPU backend, and records for Task 16 that the Darwin script must
not be a straight copy of privacy-filter-darwin.sh, which never calls package.sh
and would silently drop assets/.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): register the backend with CI and the gallery
Adds the five Linux matrix entries (cpu amd64/arm64 sharing a tag-suffix so the
manifest merge fires, cuda 12, cuda 13, vulkan), the path-filter case that keeps
later PRs touching backend/cpp/audio-cpp/ from getting zero CI jobs, the
bump-bot entry pointing at the AUDIO_CPP_VERSION pin in the backend Makefile,
the gallery meta plus its -development variant and the image entries for every
variant, and the Makefile docker-build wiring.
The matrix entries carry base-image only, with no builder-base-image, unlike
the llama-cpp and privacy-filter blocks they sit next to. The prebuilt
quay.io/go-skynet/ci-cache:base-grpc-* images ship a from-source gRPC whose
protobuf v26 depends on abseil, and this backend's sentencepiece is built with
SPM_PROTOBUF_PROVIDER=package, so it sees real abseil's absl::lts_20240116::
internal alongside its own vendored plain absl::internal and every
absl::internal:: reference becomes ambiguous. Building against base-grpc-amd64
fails at sentencepiece-static.dir/error.cc.o with "reference to 'internal' is
ambiguous". Dockerfile.audio-cpp installs apt's gRPC/protobuf 3.21.12 itself,
which is also the pair every unit and end-to-end run of this backend has been
verified against, and the CUDA toolkit therefore has to come from base-image.
No Darwin matrix entry and no metal gallery entries: the Metal build needs
scripts/build/audio-cpp-darwin.sh, a backends/audio-cpp-darwin make target and
a routing step in backend_build_darwin.yml, none of which exist yet, so an
entry added now would be routed to build-darwin-go-backend and look for
backend/go/audio-cpp/. The inferBackendPathDarwin case and the
DARWIN_BESPOKE_BUILDERS membership are in place, inert, so that adding the
entry later is a one-line change that cannot be claimed by the generic Go path.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): pin the CUDA architectures, drop the vulkan variant
Upstream sets CUDA_ARCHITECTURES to `native` on the engine_runtime target
whenever CMAKE_CUDA_ARCHITECTURES is unset at root scope, and docs/build/
linux.md says so outright. ggml's own default does not rescue it: it
list(APPEND)s in the ggml subdirectory scope, which never reaches the root
scope where the engine_runtime property is decided. No CI runner has a GPU for
`native` to enumerate, so both cublas entries would have gone red on the very
commit that first turns a CUDA build on.
Pin the list in backend/cpp/audio-cpp/Makefile, selected by CUDA_MAJOR_VERSION,
which Dockerfile.audio-cpp now forwards from the CI build-arg it was previously
discarding. The values are copied from ggml's own version guards rather than
invented, so engine_runtime and ggml compile for the same set: CUDA 12 keeps the
Maxwell/Pascal/Volta virtual archs and stops at 120a-real, CUDA 13 drops them
and adds 121a-real. The `a` suffix is used rather than `f` because the latter
needs CMake 3.31.8 and Ubuntu Noble ships 3.28.3. Verified by driving CMake
3.28.3's own CUDA architecture validator over both lists, with 120f-virtual as
the rejected control.
Drop the vulkan matrix entry, its two gallery entries, the vulkan capability
key on both metas and the Vulkan tag. Every other vulkan backend gets its Mesa
ICD drivers from .docker/install-base-deps.sh, which package-gpu-libs.sh then
bundles; Dockerfile.audio-cpp calls neither and installs only libvulkan-dev and
glslc, so the image would ship a Vulkan loader that finds no GPU. No CI job runs
a vulkan image against real hardware, so that would have passed green and failed
in users' hands. BUILD_TYPE=vulkan stays supported for local builds.
Also note on the cublas entries that cuda-major-version now selects the
architecture list and that cuda-minor-version and the base-image tag encode the
same toolkit, and correct the stale entry counts on matrixEntryKey.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): build for Darwin Metal
Bespoke C++ Darwin path like ds4 and privacy-filter: an includeDarwin matrix
entry, a backends/audio-cpp-darwin make target, a gated workflow step, and the
metal image entries plus metal/metal-darwin-arm64 capability keys in the
backend gallery.
The build script deliberately does NOT reassemble the package the way
privacy-filter-darwin.sh does. It runs the backend's own `make package` and
copies the result, so the Darwin package keeps the root-level layout the Linux
one has: grpc-server, run.sh, the ggml objects and assets/ in one directory,
with lib/ for the dylib closure. Hand-assembling would drop assets/, and
assets/ is what makes the bundled: model paths resolve with nothing downloaded.
The dylib walk is a full transitive closure rather than the single level ds4
and llama-cpp do, because Homebrew's grpc++ pulls libgrpc, abseil, upb, cares
and OpenSSL that grpc-server does not link itself, and a level-1 walk ships a
package that only works on a machine that already has Homebrew grpc.
Two fixes folded in, both in the backend Makefile:
- an EMPTY CUDA_MAJOR_VERSION fell through to the CUDA 12 architecture list,
which contains 120a-real and so needs nvcc >= 12.8. A local
BUILD_TYPE=cublas build on a 12.0-12.7 host failed to compile where
upstream's documented default (native) worked. EMPTY now maps to native,
12 and 13 keep their lists, and any other non-empty value is an error on
cublas builds. CI always passes a major, so CI is unaffected.
- the Darwin branch now points CMake at Homebrew's keg-only libomp. AppleClang
ships no OpenMP runtime and nothing is symlinked into /opt/homebrew, so
FindOpenMP finds neither the library nor the header, and audio.cpp calls
find_package(OpenMP REQUIRED) whenever ENGINE_ENABLE_OPENMP is on. Without
the hint the macOS build would have died at configure time. If the keg is
absent the build disables OpenMP instead of failing.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): make the Darwin fallbacks loud and the rpath walk complete
Review follow-up on the Darwin Metal build.
The OpenMP fallback was silent. If brew --prefix libomp ever comes back empty,
CI produced a green Metal package with 108 #pragma omp directives across ~30
files compiled out, and clang says nothing about an ignored omp pragma without
-Wsource-uses-openmp, so the only trace was one absent flag inside a set -x
cmake line. That regression would have been blamed on Metal. It now warns.
The @rpath arm of the dylib walk had no live candidate when it was written, on
the reasoning that a Metal build links ggml statically. The OpenMP fix in the
same commit made libomp.dylib one, and whether Homebrew records it as an
absolute opt path or as @rpath/libomp.dylib is not observable from Linux. The
walk now expands @rpath, @loader_path and @executable_path against the object's
own LC_RPATH entries, and only fails when nothing on disk answers, printing the
rpath list with the error so a failure on a machine nobody can attach to
explains itself.
Also: ADDITIONAL_LIBS now go through the closure rather than a bare cp, so they
are deduplicated and their own dependencies bundled; build/darwin/lib is
created explicitly instead of relying on package.sh pre-creating it; the libomp
probe uses nested ifneq rather than $(and ...), which needs GNU make 3.81 and
would otherwise expand empty and take the OFF branch on an older make; and
-DOpenMP_ROOT is quoted like its CUDA sibling.
Verified with a Linux harness that runs the script verbatim against a stubbed
otool: a level-2 transitive dep, an @rpath dep reachable only through LC_RPATH,
and an ADDITIONAL_LIBS dep are all bundled, a dependency cycle terminates,
system libraries are skipped, the packaged tree has assets/ at the root beside
grpc-server with the dylibs in lib/, and both failure paths exit non-zero.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): make bundled: reachable from a model YAML
resolve_model_path() tested the bundled: prefix on `candidate`, which prefers
ModelFile and falls back to Model. LocalAI fills ModelFile by joining ModelPath
onto the configured model string (pkg/model/loader.go, LoadModelWithFile), and
only sets it from a managed artifact otherwise, so a model YAML saying
`model: bundled:silero_vad` arrives as ModelFile "/models/bundled:silero_vad"
and Model "bundled:silero_vad". The prefix therefore never matched through the
normal load path: it matched only for a hand-written LoadModel call that left
ModelFile empty, which is exactly how task 15 verified it, and every model YAML
using the form failed with "model path does not exist:
/models/bundled:silero_vad".
Both fields are now checked, Model first, so the zero-download VAD path the
package ships assets for is reachable the way it is documented. A caller that
puts the form in ModelFile still works, so task 15's verification stands.
Compiled clean; the runtime check could not run on this host, whose system
libprotobuf/libre2 have gone missing (the pre-existing grpc-server binary no
longer resolves its libraries either), so it wants a container run.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): advertise the backend and document its options
Registers audio-cpp as preference-only in /backends/known: the family lives in
GGUF metadata that an importer cannot read from a remote repo, and one repo
hosts thirty families, so there is no honest auto-detect signal. Modality is a
single string and the import form chips on a fixed key set, so it registers as
tts with the other modalities named in the description rather than under an
invented key the UI would bucket as "other".
Adds a features page covering the option namespacing, the routing table per
endpoint, the RPCs this backend declines and why, the bundled VAD path, the
separation stem behaviour, and the family gotchas (supertonic needs the orig
package; chatterbox advertises cloning and no plain tts; nemotron_asr defers
its whole decode to finalize so live transcription emits nothing until the
client half-closes, unlike higgs_audio_stt and voxtral_realtime). Every option
name and family capability in it was read off the pinned upstream checkout.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(audio-cpp): test resolve_model_path, and correct the family names
The bundled: fix in 842443cd7 shipped without a test, which is how the bug got
there: task 15 verified the form with a hand-written LoadModel that left
ModelFile empty, and that is the one shape the server never produces. Four cases
in streaming_driver_ctest, which already links loaded_model.cpp, pin the
PRODUCTION shapes instead. The first fails against the pre-fix source (returns
the joined /models/bundled:silero_vad); the other three are the branches the
bundled: lookup now runs in front of and must fall through for.
Three family names in the docs were the source directory rather than the
registered family, on pages whose whole argument is that these names cannot be
guessed: demucs is htdemucs (demucs/loader.cpp:22), roformer is
mel_band_roformer (roformer/assets.h:15), and moss is TWO families,
moss_tts_local and moss_tts_nano. The hyphenated ASR names are underscored to
match, here and in the compatibility table.
The supertonic dtype note claimed more than the evidence carries. The f16 abort
is a local observation, identical through TTS and TTSStream; upstream's
docs/gguf.md leaves the 16-bit column untested and records q8_0 as "No
(unsupported weight dtype)", which says unusable rather than fatal. Both are
still refused, because the allow list is what the family can run. Corrected in
family_gate.h, family_gate.cpp and the docs together, since the docs inherited
the wording from the code.
The importers tripwire says in the file that it is a tripwire: it exercises no
audio-cpp behaviour, and the registration assertion lives in backend_test.go.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* gallery: add audio.cpp models covering every served RPC
One representative model per RPC group of the audio-cpp backend, plus the two
bundled VAD models, which need no download at all because the assets ship
inside the backend package.
Every hash was computed with sha256sum on the downloaded file. Quantizations
come from upstream's tested-status table in docs/gguf.md rather than a default
of q8_0: supertonic ships the orig package (its q8_0 is recorded as an
unsupported weight dtype and its f16 aborts in ggml_concat), and nemotron_asr
and htdemucs ship f16 because their q8_0 builds are recorded with drift while
16-bit is a clean pass.
Diarization and separation use the diarization and audio_transform usecases,
not transcript: /v1/audio/diarization and /audio/transform filter the default
model on FLAG_DIARIZATION and FLAG_AUDIO_TRANSFORM respectively, so a
transcript flag would have hidden both models from their own endpoints. The
forced aligner sets parameters.language, which the transcription endpoint uses
as the fallback when no language form field is sent, because the family
requires both a transcript and a language.
All ten entries were run twice: once against the raw gRPC server, and once
installed with local-ai models install and called through the HTTP endpoint.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* gallery: correct the audio.cpp entries' licenses
Swept all ten entries against the real upstream named in audio.cpp's
tools/model_manager.py rather than against the audio.cpp repo's own license.
Three were wrong:
supertonic apache-2.0 -> openrail weights come from
mlx-community/supertonic-3-mlx, and
both it and Supertone/supertonic are
openrail
citrinet apache-2.0 -> other pulled from NGC
nvidia/nemo/stt_en_citrinet_256,
governed by the NGC Terms of Use
sortformer other -> cc-by-nc-4.0 nvidia/diar_sortformer_4spk-v1 is
CC BY-NC 4.0, and the gallery already
uses that exact string, so there is no
reason to obscure a non-commercial bar
The license field is one word, so citrinet and sortformer also gained a
sentence saying why they are restricted. The other seven were confirmed
correct against their sources.
Also drops an unverified claim from the nemotron description. It said the
model drives the realtime transcription session; that endpoint actually calls
TranscribeStream, and the live RPC reaches LocalAI only through
realtime_semantic_vad.go. Neither path was exercised here, so the description
now states only the two calls that were.
MarbleNet gains the NeMo upstream under urls: for parity with silero.
No sha256, quantization, usecase or model choice changed.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(audio-transform): bound sample_rate, keep same-named uploads apart
Four defects the whole-branch review found on the Go side, plus two comment
corrections.
sample_rate is a disk-exhaustion hazard. The branch added the `form:` tag that
makes the field bind for the first time, so the resample path went from dead to
live, and utils.AudioResample interpolates the int straight into ffmpeg's -ar
with no bound. Measured with ffmpeg 7: -ar 999999999 on a 0.01 s clip writes
20 MB and exits 0, which scales linearly to the reported 3.9 GB for one second,
into a GeneratedContentDir nothing sweeps, and convertStems repeats it once per
separation stem. Clamped to 8000..192000 in the handler, before the temp dir and
before the model is touched, and rejected with a 400 outside it.
The low end was reported as "a 0-byte file". It is not: -ar 1 writes a 78-byte
header with no audio behind it, whose declared data size still claims 70 bytes,
so go-audio parses it as a 35 SECOND file and a size check does not see it. The
guard therefore compares the declared data chunk against the bytes actually on
disk, and AudioResample now fails rather than returning a WAV carrying nothing.
Both parts of a transform request land in one temp dir, and the raw copy was
named only after the client's basename, so `-F audio=@mic/clip.wav
-F reference=@loopback/clip.wav` wrote "raw-clip.wav" twice. Since
AudioToWavPreservingShape hardlinks an already-PCM16 WAV rather than copying it,
the reference part's os.Create truncated the inode audio.wav pointed at: mic and
reference came out identical, which makes an echo canceller null everything and
return near-silence with a 200. The raw copy now carries the form field name.
audio-cpp had no BackendCapabilities entry, so VoiceCloningForModel returned nil
before it ever consulted the model's tts.voice_cloning override and every
`voice: "profile:<id>"` request was refused with a 400, on a backend that ships
audio-cpp-chatterbox whose family serves cloning and not plain TTS. Registered
with its RPCs, usecases and the reference-audio contract, and deliberately
without the 16 kHz mono fold, which its separation families cannot survive.
GetBackendCapability was exact-match only, so every pinned gallery variant read
as an unknown backend: vulkan-localvqe lost the 16 kHz mono fold that used to be
unconditional and started failing inside LocalVQE, and the usecase gate does not
stand in for it because BuildFilteredFirstAvailableDefaultModel returns early
once the client names a model. Lookup now falls back to the meta name by
stripping the gallery's hardware prefix and release-channel suffix, exact match
first so nothing can be shadowed. Same class as #10945.
Also corrected: the AudioTransformRequest comment claimed echo's binder falls
back to the field name, which it does not in either direction (bindData binds
ONLY tagged fields and `continue`s otherwise; `model` arrives from
setModelNameFromRequest's c.FormValue). And the stable_audio `src` heap
corruption caveat now lives on ElevenLabsSoundGenerationRequest, where the Go
developer who would add the field can see it, instead of only in C++.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(audio-cpp): refuse a task pin the RPC cannot serve, and stop empty frames holding the lane
The model's `task:` option is copied into the request shape by all nine
handlers, which is correct, but resolve_route then replaced the RPC's candidate
list with the pin WHOLESALE and never asked whether the pin was something that
RPC routes to. One pin therefore bled across all nine surfaces, and because the
family still supported the pinned task the result was a wrong 200 rather than an
error. Reproduced live: nemotron with task:asr made Vad return 200 with zero
segments after a full ASR decode, so 14 seconds of speech was reported as
silence, and Diarize did the same; silero_vad with task:vad made
AudioTranscription return 200 with empty text and four segments whose spans were
VAD segments, which combined with response_format in {text,srt,vtt,lrc} building
the body solely from Segments[].Text yields a well formed SRT of four timed
EMPTY cues. It also contradicted the documented contract, that a family which
cannot serve a request is refused rather than rerouted.
A pin is now checked against the RPC's admissible task set before it is adopted,
and the refusal names both the pin and the RPC. The set is derived from
task_candidates with every shape flag set rather than restated, so a task added
to an RPC's candidates cannot become inadmissible by omission. Every legitimate
pin survives, and the test asserts all fifteen of them alongside the eight
crossings that must not.
The live watchdog was defeated by empty frames. idle.touch() ran on ANY message,
before the has_audio and pcm.empty() filters, so a peer writing unset-oneof or
zero-length frames faster than the window held the lane indefinitely while
feeding the decoder nothing. There is one lane per model and one model per
process, so that is a single client denying the whole backend, which is what the
watchdog exists to prevent, and the thrown text already said "no audio frame
arrived". The touch moved below the filters, which are now a named predicate so
the distinction is testable rather than a call order nobody can see.
Three comments corrected against measurement rather than reasoning:
- CMakeLists claimed zero google::protobuf:: definitions remain in the
executable. nm -C --defined-only reports 2515, and that is expected: they are
generated code, sentencepiece::ModelProto's own _InternalParse among them. The
claim that holds, and the one the ABI fix is actually about, is that no
vendored protobuf RUNTIME is linked and ParseContext::ParseMessage is
UNDEFINED in the executable, resolving to libprotobuf.so.
- refuse_cloning_without_a_clip's "cannot misfire" paragraph had its reasoning
backwards. Routing picks VoiceCloning as the FALLBACK when there is no clip,
which is the case being caught; chatterbox, which ships in the gallery,
advertises clon and no tts at all, so every voice-less request lands there.
- audio_units read "2.1 min at 96 kHz" for index 11289602, which is 1.96 min.
2.1 min is 96 kHz's OWN first failure at 12288002. Both were remeasured and
the note is now a per-rate table.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* build(audio-cpp): exclude the upstream checkout from the C++ gate, harden the darwin walk
run-unit-tests.sh pruned */llama.cpp/* but not */audio.cpp/*. It is safe today
only by luck: upstream's 44 tests all put "test" at the FRONT of the filename
(17 test-*.cpp, 27 test_*.cpp, zero *_test.cpp), so the glob misses every one of
them, and nothing enforces that. This gate runs on every PR for every backend
and compiles each match as a standalone translation unit with nothing but
nlohmann/json on the include path, so the day upstream adds or renames one test
the gate goes red repo-wide on an Apache-2.0 file nobody here wrote.
audio-cpp-darwin.sh now logs the raw otool -L output and the parsed LC_RPATH
list unconditionally, before the walk. Both awk filters in that script assume a
column layout nobody working on this can observe, since it runs only on the CI
Mac, and a green first Darwin run proves nothing about the assumption: an awk
that silently matched nothing yields an empty dependency list, which reads
exactly like "no non-system dependencies" and packages happily. Both filters
otherwise feed process substitutions, so their input never reached the log.
It also lists every symlink in the package and fails on one that cannot resolve
inside the image. A dangling link does not fail anything else here, because
every assertion tests with -e, which follows links; it fails at dlopen on a
user's Mac. Links are NOT banned outright, which the review suggested but which
would break the libggml.dylib -> libggml.0.dylib chain the `cp -a` above exists
to preserve. What is banned is a link that resolves on the build host and will
not resolve in the image: a broken one, or an absolute one pointing outside the
package.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* style(audio-cpp): drop em dashes from the audio-cpp capability entry
Follow-up to a84b3c4b9, no behaviour change.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(config): key the voice-cloning model rule on the resolved backend
Making GetBackendCapability strip the gallery hardware prefix and release
channel fixed pinned variants of /audio/transform, but VoiceCloningForModel
kept keying its per-backend switch on the caller's spelling. A pinned name
therefore resolved the capability by stripping and then missed every case in
the switch, falling through to the permissive default: cuda12-vibevoice-cpp
advertised voice cloning for the realtime 0.5B model, metal-coqui for
tacotron2, cuda12-crispasr for a pure ASR model, cpu-qwen3-tts-cpp for
CustomVoice. Each of those is a model that cannot clone, so /v1/audio/speech
accepted a profile: voice it had to fail on inside the backend rather than
rejecting it with a 400, and the UI advertised the capability too.
resolveBackendCapability now returns the key the entry was found under, and
callers that branch on backend identity use that key instead of the name they
were handed. The exact-match-first order is unchanged, so a backend genuinely
registered under a variant-looking name still keys on its own name.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* gallery(audio-cpp): declare audio_transform on the chatterbox entry
Chatterbox advertises VoiceCloning AND VoiceConversion
(src/models/chatterbox), and the entry's own description already said so, but
known_usecases listed only tts. /audio/transform selects its default model by
FLAG_AUDIO_TRANSFORM, so voice conversion was reachable only by naming the
model explicitly and was invisible to every usecase-driven surface. It is the
one audio.cpp task with a shipped gallery model and no way to find it.
Verified against the real model rather than inferred from the capability list:
AudioTransform with chatterbox-q8_0, speech as audio_path and a speaker clip
as reference_path, returns a 5.08 s 24 kHz mono WAV at -25.5 dB mean and zero
stems, which is the single-output shape voice conversion should have.
The description now says which endpoint reaches that half and warns that
installing this next to a source-separation model gives /audio/transform two
candidates, so the model should be named rather than defaulted.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* gallery(audio-cpp): add voice-design and singing-voice-conversion entries
Two of the three audio.cpp task kinds that had no gallery model now have one.
Both were driven end to end against the real weights through the backend
before being written, not inferred from the capability tables.
audio-cpp-irodori-voicedesign covers vdes. TTS carrying `instructions` routes
to the vdes task, so the voice is described in words rather than supplied as a
clip. Verified: "a calm elderly woman speaking slowly with a warm, gentle
tone" over an 8.76 s 48 kHz mono render at -16.8 dB mean, and a closed-loop
citrinet pass recovers the sentence with the accent drift expected from a
Japanese-first model read by an English recogniser.
audio-cpp-seedvc-singing covers svc, and pins task:svc because nothing else
can reach it. seed_vc advertises svc and ordinary voice conversion, no request
signal means "this input is singing", and auto-routing resolves the tie to
voice conversion every time. Verified with the pin: 5.04 s 44.1 kHz output
whose closed-loop citrinet transcription is exact.
s2s deliberately has no entry, and the reason is not effort. miocodec is the
only upstream family whose speech-to-speech route needs no text, and it
returned audio with correct duration and level but no recoverable speech in
four independent attempts: the stale build, v2 q8_0, v2 orig (the variant
upstream records as a clean Pass), both tasks, and matched 44.1 kHz inputs on
both sides. vevo2's route refuses with "Vevo2 text/prosody route requires
text_input or target_text", and session.cpp:897 fills target_text only from
request.text_input, which AudioTransform has no field to carry. The same
vevo2 weights convert voice correctly through the default route with an exact
ASR round trip, so the model and the plumbing are both healthy; it is the s2s
route specifically that this RPC cannot express.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* backend(audio-cpp): carry transform text through params, add the s2s entry
AudioTransform is audio-in / audio-out and its proto message has no text
field, but not every task it routes to is audio-only. vevo2's speech-to-speech
route is a text and prosody route: session.cpp:897 fills refs.target_text from
request.text_input and nowhere else, and the run refuses without one with
"Vevo2 text/prosody route requires text_input or target_text". The params map
is the only channel this RPC has that reaches the engine, so the text travels
through it and apply_transform_text_input unpacks it after the params have
been copied into task.options.
Before this, s2s was not awkward to reach through /audio/transform, it was
unreachable, and it was the last audio.cpp task kind with a real model and no
way to get to it.
target_text is canonical and text is its alias, the order vevo2's own option
table declares them in, so a request setting both gets the canonical one
rather than whichever the map happened to store first. An empty value falls
through to the next candidate instead of ending the search. language rides
along only when a text was found: on its own it conditions nothing, and
manufacturing a text_input for it would route a plain separation request
carrying a language hint through the text path. The keys are left in
task.options rather than erased, because vevo2's loader advertises target_text
as a request option and a family reading it there keeps working.
Nine tests, all confirmed failing on behaviour against a stub that returned
false before the implementation was written. Verified end to end afterwards:
vevo2-q8_0 with task:s2s and params[text] returns a 5.12 s 24 kHz output whose
closed-loop citrinet transcription is exact, and htdemucs separation with no
text param still returns its four stems, with and without params[stem].
audio-cpp-vevo2-speech-to-speech ships that route. Every audio.cpp task kind
with a loadable family now has a gallery entry; spk remains the only gap and
has no family upstream at all.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* docs(audio-cpp): document params[text] and the pinned transform tasks
The text channel and the two task pins are both invisible from the endpoint
contract alone: nothing in the AudioTransform form tells a reader that a
speech-to-speech model needs the line it is resynthesising, and nothing says
that asking for singing voice conversion without task:svc silently gets plain
voice conversion instead. Both are the kind of thing a user only discovers
from a refusal or, worse, from output that looks right and is not.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(utils): annotate the two G304 sites this branch introduced
gosec flags os.Open on a variable path, and both new call sites in ffmpeg.go
are its alerts on this PR. Neither is reachable by an outside caller: isPCM16Wav
opens the exact path it is about to hand ffmpeg as input, which in the upload
path is a server-created temp file named from path.Base of the client name so
no traversal survives, and wavAudioBytes opens AudioResample's own dst, a name
this package derives from src and has just had ffmpeg write.
Annotated in the repo's existing style rather than restructured, with the
reason spelled out, because a bare suppression is worth nothing to the next
reader. The three other G304 sites in this file, in passthroughWAV and
isTargetWav, predate the branch and are left untouched.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(audio-cpp): build arm64 with gcc-14 for the armv9.2 SME variants
The arm64 CPU image failed to build:
cc1: error: invalid feature modifier 'sme' in
'-march=armv9.2-a+dotprod+fp16+sve+i8mm+sve2+sme'
ggml's CPU_ALL_VARIANTS table includes armv9.2 variants compiled with +sme, and
Ubuntu Noble's default gcc-13 rejects that feature modifier. Every entry in the
table has to compile even though a host only ever dlopens the one its own CPU
supports, so a single unbuildable variant fails the whole image. gcc-14 accepts
it, which is exactly the fix llama-cpp already carries in
.docker/llama-cpp-compile.sh; this is the same problem reached by a different
Dockerfile.
Applied to every arm64 BUILD_TYPE rather than to the CPU one alone, and that
differs from llama-cpp on purpose. llama-cpp needs it only for its pure-CPU
image because its GPU builds run llama-cpp-fallback, which builds no variant
table. This backend's Makefile turns ENGINE_ENABLE_CPU_ALL_VARIANTS on for
every non-Darwin build, GPU included, so an arm64 GPU image would hit the
identical error. The matrix has no arm64 GPU entry today, which is precisely
why gating on an empty BUILD_TYPE would leave the trap armed for whoever adds
the first one.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(model): deterministic, file-type-filtered backend auto-detect (#9287)
When a model config declares no explicit `backend:`, Load() fell into a
trial loop built by ranging the external-backends Go map (random order)
with no filtering, returning the first backend whose gRPC LoadModel
succeeded. An unrelated installed backend - e.g. the "opus" audio codec -
could therefore win a GGUF/LLM model load, so a model that should run on
llama.cpp wrongly tried to use opus.
Extract the candidate selection into a pure, testable function
SelectAutoLoadBackends that:
- sorts the candidate list deterministically (no more map-order
nondeterminism), and
- for a `.gguf` model, filters to LLM-capable backends (via
core/config.BackendCapabilities) and puts llama-cpp first, so an
incompatible audio/codec/image backend can never win the trial loop.
If filtering would leave zero candidates, the full sorted set is returned
unchanged, so a previously-loadable model is never made unloadable.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(model): break core/config <-> pkg/model import cycle in backend auto-detect
The #9287 auto-detect change made pkg/model/autoload.go import core/config
for the backend capability table. core/config already imports pkg/model
(runtime_settings_registry.go uses model.DefaultWatchdogInterval), so this
closed a core/config -> pkg/model -> core/config import cycle and broke the
build and golangci-lint.
Invert the dependency so the lower-level pkg/model no longer imports the
higher-level core/config. pkg/model exposes RegisterLLMCapableBackendFunc and
uses the registered predicate; core/config (which owns the capability table)
registers it from an init(). The deterministic, GGUF-type-filtered selection
behaviour is unchanged. When the predicate is unwired the GGUF filter is
skipped, preserving the existing zero-candidate fallback.
The unit test now injects a fake capability predicate so SelectAutoLoadBackends
is exercised independently of the core/config table.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:opus-4.8 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
* fix(grammars): reject cyclic $ref in JSON-schema grammar to prevent stack-overflow crash
JSONSchemaConverter.visit resolved $ref entries by recursively calling
itself with no cycle detection. A client-supplied grammar_json_functions
schema whose $defs contains a self- or mutually-referential $ref (e.g.
{"A": {"$ref": "#/$defs/A"}}) made visit recurse until the goroutine
stack was exhausted, producing a fatal "stack overflow" that kills the
whole process rather than failing the single request. The schema is
converted synchronously in the /v1/chat/completions handler before any
backend call, so this is an unauthenticated remote crash. Fixes#11020.
Track the $ref targets currently on the recursion stack and error out
when one is re-entered, while popping after each descent so sibling
(non-cyclic) reuse of the same $ref is still allowed.
Signed-off-by: Tai An <antai12232931@outlook.com>
* fix(grammars): add a bounded recursion depth and cover llama31 $ref cycles
Addresses the review on #11041. The stack-set approach catches cyclic
$ref chains, but a deeply nested yet acyclic client schema (thousands of
nested arrays/objects) can still recurse through visit until the
goroutine stack is exhausted, which is the same unauthenticated remote
crash surface as #11020.
- Add a bounded depth counter to JSONSchemaConverter.visit (incremented
with a defer-based cleanup, capped at maxSchemaDepth = 256, far above
any realistic schema) so an over-deep schema fails the request with an
ordinary error instead of crashing the process.
- Apply the same cyclic-$ref guard and depth bound to
LLama31SchemaConverter.visit, the other production grammar entry point
named in #11020, which previously had no cycle detection at all.
- Regression tests: a deeply nested acyclic schema is rejected while a
moderately nested one still builds, plus direct/indirect $ref cycle
and depth tests for the llama31 converter.
Signed-off-by: Tai An <antai12232931@outlook.com>
* test(grammars): make llama31 cycle fixtures valid function-call shapes
The two new llama31 $ref-cycle specs asserted on "cyclic $ref" but the
converter requires each top-level oneOf alternative to carry its
function-name property before descending, so both fixtures failed
earlier with "no function name found in the schema" and never reached
the cycle guard.
Give each fixture a valid llama31 shape: construct the converter with
NewLLama31SchemaConverter("function"), put "function": {"const": "test"}
on the top-level alternative, and hang the cyclic $ref under an
arguments property, so all 29 grammar specs pass and the assertions
genuinely observe the cyclic $ref error.
Signed-off-by: Tai An <antai12232931@outlook.com>
---------
Signed-off-by: Tai An <antai12232931@outlook.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
backend_pr.yml and test-extra.yml already filter themselves, so a gallery-only
or docs-only PR costs them about one job each. The image and Go workflows had no
filter of any kind, so a one-line gallery/index.yaml edit queued 20 jobs: 7
container image builds, 3 GoReleaser/darwin launcher builds, 3 unit test jobs, 2
golangci-lint, 1 e2e, 1 yamllint, plus the 3 that correctly stop after their
detect step. A docs-only PR queued the same.
This matters more than the job count suggests. Measured over the week to
2026-07-30, 97% of CI wall-clock is queueing and 3% is execution: a median
5-hour queue against a 4-20 minute median job. Cutting job count is the only
lever that shortens feedback time. The volume is there to cut, too: 13
gallery-only PRs merged that week with 10 open at once, and 78 of the 137 PRs
opened were bot-generated.
Add paths-ignore for gallery/**, docs/**, examples/** and **/*.md to the
pull_request trigger of image-pr.yml, build-test.yaml and tests-e2e.yml, and
add gallery/** to lint.yml, which already excluded the rest. That drops 13 of
the 20 jobs. None of the four can observe such a diff: gallery metadata is
parsed at runtime and never copied into an image, docs and markdown never enter
one at all, GoReleaser and the launcher take no such input, the e2e suite drives
backends over gRPC directly, and golangci-lint runs new-from-merge-base so a
diff with no touched Go lines is a no-op. The build-test exclusion also frees
macOS capacity, which is the scarcest runner class.
The two checks that do validate the gallery are deliberately left alone.
test.yml still runs core/gallery/variants_lint_test.go, which reads the real
gallery/index.yaml and asserts the index invariants, and yaml-check.yml still
lints the syntax.
paths-ignore skips a run only when every changed file matches, so a PR touching
the gallery and Go code still runs everything. master carries no branch
protection and no rulesets, so a skipped workflow reports no status and nothing
waits on it; .agents/ci-caching.md records that constraint for whenever required
status checks are introduced.
image.yml on master push is left unfiltered on purpose: skipping it would stop
the master and latest tags being republished for a gallery commit, which is a
publishing decision rather than a cost one.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* gallery: add Nanbeige4.2 3B GGUF variants
Add Q4_K_M and Q8_0 builds of the compact Nanbeige4.2 agentic and reasoning model, grouped as install-time variants.
Assisted-by: Codex:gpt-5
* gallery: simplify Nanbeige4.2 model name
Apply the maintainer-requested canonical model name while retaining the quantization variants under the entry.\n\nAssisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add the official Q4_K_M build and seven APEX quality and size variants for llama.cpp.
Assisted-by: Codex:gpt-5 [Hugging Face API]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
GenerateImage hardcoded TilingParamsSetEnabled(vaep, false), so tiled VAE
decoding was unreachable from a model config even though all four upstream
setters were already bound in main.go.
Sampling runs in latent space, but the final VAE decode expands to full
resolution and needs one large compute buffer. At 1024x1024 that buffer
exceeds 8GB, which fails on two kinds of device: cards without the VRAM
for a full-frame decode, and drivers that cap a single allocation
regardless of how much memory is free. Mesa RADV reports a 4GiB
maxMemoryAllocationSize, so a Radeon 8060S with 74GiB of device-local
heap still cannot serve that decode:
[INFO ] sampling completed, taking 251.82s
[INFO ] decoding 1 latents
ggml_vulkan: Requested buffer size exceeds device buffer size limit:
ErrorOutOfDeviceMemory
[ERROR] vae: failed to allocate the compute buffer
[ERROR] decode_first_stage failed for latent 1
Every sampling step completes and then the run is discarded at the last
stage, so the whole generation is wasted.
Add three options, parsed in Load and applied per generation:
vae_tiling:true enable tiled decoding (bare flag also works)
vae_tile_size:512 tile size, or 512x384 for a rectangle
vae_tile_overlap:0.25 overlap between tiles
Tiling stays off unless requested, so existing models are unaffected. Tile
size and overlap only reach the library when the operator set them, which
keeps upstream's defaults rather than pushing a zero, and an unparseable
value is treated as absent for the same reason.
Truthy spellings match what load_model already accepts for its own bool
options, and the bare-flag form matches diffusion_model, so no new
convention is introduced.
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
Add the CAJAL GGUF gallery template and gallery index entry for local llama-cpp installs.
Assisted-by: Codex:gpt-5
Signed-off-by: Ching Kao <0980124jim@gmail.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
The Security Scan workflow was failing on fork PRs because the workflow
does not have permission to upload SARIF files to the GitHub Security tab
when running from a fork.
This change adds '!github.repository.fork' checks to all steps
to prevent the workflow from running on fork repositories.
This fix should be applied to the main repository so that
all forks inherit the correct configuration.
Fixes#10322, #10318, #10320, #10321
Co-authored-by: ghshhf <ghshhf@users.noreply.github.com>
fix(chatterbox): pin cublas12 torch/transformers and setuptools so the backend loads
The cuda12-chatterbox gallery backend fails to load on a fresh install
because several deps in requirements-cublas12.txt are unpinned:
- torch/torchaudio: unlike requirements-cublas13.txt and
requirements-cpu.txt, this file has no --extra-index-url, so pip pulls
a wheel whose CUDA runtime (cu130) is newer than the host driver
supports ("NVIDIA driver on your system is too old"). Add the cu124
index and pin torch/torchaudio 2.6.0+cu124.
- transformers: resolves to 5.x, which dropped LlamaConfig.rope_theta
that chatterbox-tts 0.3.1's T3 config still reads. Cap to <5.
- setuptools: 81+ dropped pkg_resources, which perth imports under a
bare try/except and silently sets PerthImplicitWatermarker=None,
making ChatterboxTTS.__init__ raise 'NoneType' object is not callable.
Cap to <81 in requirements.txt.
Fixes#11070
Signed-off-by: Tai An <antai12232931@anaiguo.com>
Co-authored-by: Tai An <antai12232931@anaiguo.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
response.create accepts a metadata map and ResponseCreateParams has carried
the field all along, but triggerResponse never copied it onto the Response it
emits, so both terminals went out with metadata omitted.
That field is the only thing tying a terminal event back to the
response.create that asked for it. Our own doc comment on ResponseCreateEvent
says so — "the metadata field is a good way to disambiguate multiple
simultaneous Responses" — and it is what makes an out-of-band response
(conversation: "none") usable at all: a client running one alongside the
spoken conversation has no way to tell its own answer from the conversation's,
so it waits for a reply it already received and gave away.
Found from the client side: a headless text turn injected into a live session
was answered correctly in about a second, and the caller still blocked until
its own two-minute timeout because it could not recognise the answer.
Carry the map on liveResponse so all three terminals (in_progress, cancelled,
completed) report it, and leave it omitted when response.create sent none.
Assisted-by: Claude:claude-opus-5 gofmt
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat: add Valkey Search vector store backend
Add a new built-in Go gRPC store backend 'valkey-store' that implements the
four Stores RPCs (Set/Get/Delete/Find) against the Valkey Search module (FT.*)
using the pure-Go github.com/valkey-io/valkey-go client. It is selected via the
existing per-request 'backend' field on /stores, so there is no proto or HTTP
API change, and it mirrors the in-memory local-store while adding persistence
across restarts and opt-in HNSW.
Each vector is a Valkey HASH keyed by hex(little-endian float32); the index is
created lazily on first Set (FLAT+COSINE by default), cosine similarity is
derived as 1-distance, and namespaces get a collision-resistant token. Includes
unit tests (valkey-go mock) and env-gated integration tests against
valkey/valkey-bundle, plus build/matrix/gallery wiring and docs.
Assisted-by: Kiro:claude-opus-4.8 golangci-lint
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* Address review feedback: recover persisted index dimension, harden Find
- Load now recovers the persisted vector DIM from FT.INFO (not just index
existence), so a post-restart Set/Find validates against the real DIM
instead of silently re-learning a wrong one and dropping mismatched
vectors from the index. This also restores Find's dimension check after
a restart.
- StoresFind treats a dropped/missing index as an empty store (empty
result, no error) and clears the stale indexCreated flag, matching
local-store's empty-store behaviour.
- StoresSet reuses checkDims for its per-key length check so the four RPCs
share one dimension-guard implementation.
- Add unit tests for FT.INFO dimension recovery, loadIndexState, and the
dropped-index Find path.
Assisted-by: Kiro:claude-opus-4.8
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* Address review feedback: TLS ServerName/CA, Find nil-check, config fail-fast
Addresses external review comments on the valkey-store backend:
- StoresFind now rejects a nil/empty query Key before dereferencing it,
so a malformed gRPC request can no longer panic the backend.
- TLS: derive ServerName (SNI) from the VALKEY_ADDR host so certificate
verification works for IP-addressed endpoints, and add VALKEY_TLS_CA_CERT
(custom CA bundle) and VALKEY_TLS_SKIP_VERIFY (testing-only) knobs.
- Config integer parsing now fails fast on a malformed value (e.g.
VALKEY_HNSW_M=1x6) instead of silently defaulting, matching the
fail-fast behaviour of the index-algo/distance-metric validation.
- Add VALKEY_DB (SELECT n) support for logical-DB isolation.
- Cap the human-readable part of a namespace token at 64 chars so a very
long model name cannot produce an unbounded key prefix / index name
(the appended short hash keeps distinct namespaces collision-free).
- Document the KNN-query injection-safety invariant (fields are constants)
and why StoresGet uses a single aggregate DoMulti deadline for reads.
- Unit tests for the Find nil/empty-key guard, fail-fast HNSW parsing,
and VALKEY_DB parsing/validation; docs + .env updated for the new vars.
Assisted-by: Kiro:claude-opus-4.8 golangci-lint
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* Address review feedback: configure valkey-store via model config
richiejp asked that the valkey-store backend take its configuration from
a model config rather than process-wide VALKEY_* environment variables,
so multiple stores can each have their own Valkey config within one
LocalAI process. This removes every env access from the backend and
routes config through the model-config seam every other backend uses.
- config.go: loadConfig(opts *pb.ModelOptions) now parses the model
config `options:` list (key:value strings, split on the first ':')
instead of os.Getenv. Option keys mirror the old VALKEY_* names without
the prefix (addr, index_algo, distance_metric, ...). Defaults, fail-fast
validation and the mandatory client name are unchanged.
- store.go: Load threads opts into loadConfig; TLS comments/errors renamed
off the VALKEY_* names.
- core/backend/stores.go: StoreBackend and NewVectorStore take a
*config.ModelConfigLoader, resolve the per-store ModelConfig by store
name, and pass its Options (and Backend when unset) to the backend via
WithLoadGRPCLoadModelOpts. No config -> default backend + built-in
defaults, preserving the zero-config experience.
- Endpoints/routes/application: thread the config loader to StoreBackend.
- Unit + integration tests: configure via options; the integration test
passes addr through the model-config path (VALKEY_ADDR is now only the
test harness locating the server).
- docs + .env: document the model-config options, drop the env var table.
Assisted-by: Kiro:claude-opus-4.8
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* Remove valkey-store informational comment from .env The backend is configured via model config, not env vars — the comment was unnecessary noise in .env. The configuration is already documented in docs/content/features/stores.md.
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* feat(valkey-store): gate Load on NamespacePrefix to refuse autoload probing Mirror local-store's pattern: reject model names without store.NamespacePrefix so the model loader's greedy autoload probe cannot bind an arbitrary model name to the vector store backend (the #9287 failure mode). Also adds unit tests for the gate covering: prefixed namespace, prefix alone, unprefixed model name, empty model, and nil opts.
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* feat(valkey-store): add username_env/password_env credential indirection Add support for resolving Valkey credentials from environment variables named in the model config, mirroring cloud-proxy's api_key_env pattern. This keeps secrets out of model YAML files and lets distinct store configs each reference their own credentials. Options: username_env / password_env name the env var holding the value. The direct username / password options still work and take precedence when both are set (backward compatible). Includes 5 unit tests and updated stores.md documentation.
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
* fix: correct rebase artifacts in backend-matrix.yml and Makefile Fix two issues introduced by the conflict-resolution script during the rebase onto master: 1. .github/backend-matrix.yml: valkey-store entries were merged INTO the cloud-proxy entries (duplicate keys in same YAML map items) instead of being separate list items. This broke cloud-proxy Linux builds and the cloud-proxy darwin entry lost its build-type/lang. Fixed by making them standalone entries and restoring cloud-proxy exactly as on master. 2. Makefile: duplicated .NOTPARALLEL and docker-build-backends lines. Collapsed to single lines that are master's current content plus the valkey-store additions. Also adds the three optional pickups from #10801: - /valkey-store in .gitignore (the built binary) - valkey-store row in docs/content/reference/compatibility-table.md - valkey-store line in backend/README.md
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
---------
Signed-off-by: Daria Korenieva <daric2612@gmail.com>
Co-authored-by: Daria Korenieva <daric2612@gmail.com>
* docs(gpu): add gfx1151 / ROCm 7.x and fix ROCm section
- Fix typo: "deditated" → "dedicated", "ROCm6" → "ROCm"
- Add ROCm 7.x to requirements (alongside ROCm 6.x)
- Add Ubuntu 24.04 to tested OS list
- Add AMD Strix Halo / gfx1151 section with kernel params,
required env vars (HSA_OVERRIDE_GFX_VERSION, ROCBLAS_USE_HIPBLASLT),
and Docker Compose example
- Add gfx1151 to the list of compiled GPU targets
- Add ROCm version column to verified devices table
- Add gfx1151 / Radeon 8060S (ROCm 7.11.0) as verified device
* fix(docs/gpu): correct gfx1151 section — env vars, image tag, safety warning
- Add all 4 required env vars (HSA_OVERRIDE_GFX_VERSION, ROCBLAS_USE_HIPBLASLT,
HSA_XNACK=1, HSA_ENABLE_SDMA=0) with descriptions in a table
- Fix Docker Compose example to use the ROCm 7.x image tag (-gpu-hipblas-rocm7),
not the ROCm 6.x image
- Add explicit warning: GGML_CUDA_ENABLE_UNIFIED_MEMORY must NOT be set
(even =0 activates hipMallocManaged due to getenv != nullptr check)
- Add --force-recreate note (docker restart does not update container env)
- Add tested hardware note (Geekom A9 Mega / Ryzen AI MAX+ 395)
* docs(gpu): single ROCm image — drop -rocm7 tag suffix
Per maintainer feedback on PR #9229: there is only one ROCm/hipblas
main image, and it ships with ROCm 7.x by default — no separate
-rocm7 tag.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Document the reverse-proxy settings needed for long-running and multimodal requests, and distinguish edge-generated 504 responses from the optional LocalAI busy watchdog.
Assisted-by: Codex:gpt-5 [Codex]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(3d): add Generate3D RPC, FLAG_3D capability, and /v1/3d/generations endpoint
Adds the plumbing for image-conditioned 3D asset generation (binary
glTF / GLB output), modeled on the video generation path:
- backend.proto: Generate3D RPC + Generate3DRequest (staged image src,
glb dst, seed/step/cfg_scale/texture_steps, quality and background
enums, params map for backend-specific extras)
- pkg/grpc: thread Generate3D through client, server, embed, base and
the backend interfaces; connection-evicting and distributed-node
wrappers (in-flight tracking + file staging) included
- core/config: FLAG_3D usecase (guessed only for the trellis2cpp
backend), '3d' canonical usecase string mapped to the Generate3D
method, and a '3d' output modality
- REST: POST /v1/3d/generations (+ unversioned alias) returning
OpenAIResponse with a /generated-3d URL or b64_json; conditioning
image accepted as URL, base64, or data URI; quality/background
validated at the edge; .glb served as model/gltf-binary
- auth: '3d' route feature (default ON); /api/instructions entry
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(trellis2cpp): add the trellis2.cpp image-to-3D backend
Wraps localai-org/trellis2cpp (C++/GGML port of Microsoft TRELLIS.2,
pbr-textures branch) as a Go+purego backend, following the
stablediffusion-ggml pattern:
- backend/go/trellis2cpp: purego bindings to the flat C ABI (v9,
asserted at startup), eager pipeline load with model-set validation
(refuses non-trellis GGUFs; degrades coarse/geometry-only/textured
exactly like the upstream demo), Generate3D via t2_generate +
t2_bake_glb writing a binary glTF to dst. Weight-free unit tests
cover resolution/validation/param mapping — CI never downloads the
multi-GB GGUF set or runs inference.
- CPU SIMD variants build into per-variant directories (the shared
libggml sonames collide across variants, unlike sd-ggml's flat
renamed-.so scheme); run.sh picks one via /proc/cpuinfo.
- CI wiring: backend-matrix entries (cpu, cuda12/13, vulkan
amd64+arm64, l4t, l4t-cuda13, darwin metal), index.yaml meta +
latest/master image entries, bump_deps tracking of the pbr-textures
branch, changed-backends.js mapping, top-level Makefile targets.
- Importer: auto-detects trellis GGUF repos/URIs (registered before
llama-cpp so the .gguf match isn't stolen) and expands any trellis
URI to the full 10-file component set spanning the three LocalAI-io
HF repos.
- Gallery: trellis2-4b (full PBR + 1024 cascade) and
trellis2-4b-geometry (512 untextured) with verified sha256s.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(ui): 3D generation page with native GLB viewer and IndexedDB history
Adds a Studio tab + /app/3d page for the new image-to-3D endpoint:
- GlbViewer ports the trellis2cpp demo's dependency-free WebGL2
renderer (quaternion trackball, metallic-roughness PBR, ACES,
hidden-line wireframe with a bounded index budget) and pairs it with
a minimal GLB parser for the two forms t2_bake_glb emits — dense
vertex-PBR (linear COLOR_0 + _METALLIC_ROUGHNESS, uploaded as
normalized integers) and the opt-in UV-atlas textured form. Parsing
happens before any GL so stats and errors render without WebGL2.
- use3DHistory stores past generations (params, input thumbnail, and
the GLB blob itself) in IndexedDB with keep-newest-20 eviction —
GLBs are multi-MB binaries localStorage can't hold — and the page
offers a download button for the active GLB.
- Wiring: CAP_3D capability constant (FLAG_3D — the exact string
/api/models/capabilities serves), threeDApi, router entries, Studio
tab, vite dev proxy, en locale keys.
- e2e: render-smoke entry plus a focused spec that feeds a real
one-triangle vertex-PBR GLB through the parser/viewer and exercises
IndexedDB persistence, selection, deletion, and API errors.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(3d): address API correctness and UX issues
Keep 3D generation on the LocalAI-specific /3d/generations route and ensure authentication and permissions cover it.
Propagate distributed transfer failures, publish a portable ARM64 backend image, honor importer overrides, and align discovery, upload validation, and touch controls.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(3d): add previewable print remeshing
Add a single-detail CGAL Alpha Wrap workflow for existing Trellis GLBs, including PBR reprojection, API documentation, tracing, and an in-browser preview before download.
Allow the remesh route to enforce its 512 MiB upload cap independently of the smaller global default so generated high-resolution meshes can be processed.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* build(trellis2cpp): centralize remesh dependency pins
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(kokoros): implement Generate3D stub for new proto RPC
The Generate3D RPC added to backend.proto for the trellis2cpp backend
made tonic's generated Backend trait require generate3_d, breaking the
kokoros-grpc build. Return unimplemented like the other unsupported
modalities.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
backend/backend.proto is consumed by every language, so its SHARED_BUILD_INPUTS
rule could only ever be always/always: 417 Linux plus 56 Darwin builds. It fires
on ~1.3% of commits (10 of 767 over six months), which made it the single
largest CI cost driver in the repo.
On 2026-07-29 the queue reached 2178 jobs against 8 concurrent runners. Four
runs totalling 935 of those jobs were triggered by nothing but a proto edit. The
largest, 378 jobs on master, came from PR #11158, whose entire proto diff was
six lines adding `bool cache_prompt = 8;` to one message. No backend that does
not read that field behaves any differently for it.
Make the rule content-aware. changed-backends.js resolves backend.proto at the
base revision (the contents-API pattern already used for backend-matrix.yml) and
hands both texts to protoChangeIsAdditive(), which compares them structurally so
a comment reflow, reindent or field reorder does not read as a change. An
additive-only edit (new field with an unused number, new message, new enum
value, new RPC) suppresses the rule and rebuilds nothing; a removed, renumbered,
retyped or renamed field, a dropped RPC or a changed option still rebuilds
everything, as does an unresolvable base revision.
Every other matched rule is untouched, so a PR that edits the proto and
scripts/build/ is still a full rebuild, and the weekly full-matrix cron remains
the backstop for stale wheels.
Verified against all ten proto commits of the preceding six months: the nine
with a resolvable parent all classify as additive, and controls covering a
retyped-and-renumbered field, a deleted RPC, identical revisions and a
reindent-plus-comment-reflow all classify correctly.
Assisted-by: Claude:opus-5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
If your endpoint should be tracked for usage (token counts, request counts), add the `usageMiddleware` to its middleware chain. See `core/http/middleware/usage.go` and how it's applied in `routes/openai.go`.
## Control-plane database health metrics
In distributed mode the frontend registers three OpenTelemetry gauges over the
wired in `core/application/distributed.go`). They reach `/metrics` through the
same Prometheus exporter as the rest of the API metrics.
| Metric | Meaning | Page when |
|--------|---------|-----------|
| `localai_control_plane_oldest_xmin_age` | Transactions elapsed since the oldest snapshot any backend still holds | above a few million, and rising |
| `localai_control_plane_longest_transaction_seconds` | Age of the longest open transaction | above 3600 |
| `localai_control_plane_dead_tuple_ratio` | Dead tuples per live tuple, labelled by `table`, on `backend_nodes`, `node_models` and `gallery_operations` | sustained above ~10 on a small table |
A sustained high `localai_control_plane_oldest_xmin_age` is the one to page on.
While it grows, autovacuum can reclaim nothing anywhere in the database no
matter how often it runs, so the dead tuple ratio keeps climbing and a six-row
registry table can reach hundreds of megabytes. Tuning autovacuum does not help.
The fix is to find the transaction holding the horizon open and clear it:
An orphaned prepared transaction is cleared with `ROLLBACK PREPARED '<gid>'`,
and a stale slot with `pg_drop_replication_slot('<slot_name>')`. Check both
before concluding that a bloated table has some other cause.
Sampling is scrape-driven behind a 30 second cache, so scrape frequency does not
translate into database load. Failed and timed-out samples cost the same interval
as successful ones, so a database that is already struggling is not retried on
every scrape. A failed sample reports the last good values rather than failing the
scrape, because these gauges matter most when the database is struggling. Before
the first successful sample the gauges are absent rather than zero, since a zero
xmin age would read as a healthy horizon: alert on `absent()` too if you need to
distinguish "healthy" from "never sampled".
## Advertising surfaces — where to register a new capability
Beyond routing and auth, LocalAI publishes its capability surface in **four independent places**. When you add an endpoint — especially one introducing a net-new capability like a new media type or a new auth-gated feature — you must update every relevant surface. These aren't optional: missing them means the endpoint works but is invisible to clients, admins, and the UI.
@@ -304,7 +356,9 @@ React pages that want to filter the ModelSelector by capability import this symb
A new capability deserves its own page under `docs/content/features/`, plus cross-links from related features and an entry in `docs/content/whats-new.md`. See the pattern used by `face-recognition.md` / `object-detection.md`.
A new capability deserves its own page under `docs/content/features/`, plus cross-links from related features. See the pattern used by `face-recognition.md` / `object-detection.md`.
Announcing it is the release's job, not this page's: the capability gets covered in the release blog post under `website/content/blog/`. See [preparing-a-release.md](preparing-a-release.md). `docs/content/whats-new.md` is only a pointer at the blog and GitHub Releases, so there is nothing to add there.
## Path protection rules
@@ -334,7 +388,7 @@ When adding a new endpoint:
- [ ] Swagger block on the handler: `@Summary`, `@Tags`, `@Param`, `@Success`, `@Router`
- [ ] If new capability area (new swagger tag): entry in `instructionDefs` in `core/http/endpoints/localai/api_instructions.go` + test count bumped in `api_instructions_test.go`
- [ ] If new `FLAG_*` usecase flag: matching `CAP_*` symbol exported from `core/http/react-ui/src/utils/capabilities.js`
- [ ]`docs/content/features/<feature>.md` created; cross-links from related feature pages; entry in `docs/content/whats-new.md`
- [ ]`docs/content/features/<feature>.md` created; cross-links from related feature pages; capability covered in the release blog post (see [preparing-a-release.md](preparing-a-release.md))
**Quality**
- [ ] Error responses use `schema.ErrorResponse` format (or `echo.NewHTTPError` with a mapped gRPC status — see the `mapBackendError` helper in `core/http/endpoints/localai/images.go`)
- **Don't weaken the gate:** never hand-lower a baseline or widen a tolerance to turn a red gate green. The ratchet only moves up.
- If a change drops coverage, **add tests** (sort `coverage-summary.json` by line% ascending to find untested code) rather than editing the baseline. When coverage legitimately rises, commit the regenerated baseline (`make test-coverage-baseline` / `test-ui-coverage-baseline`).
- The Go gate is **strict — no tolerance**; `covermode=atomic` keeps it deterministic. The UI gate keeps a small tolerance only because its e2e coverage isn't.
## Distributed-mode test suites
Two suites cover distributed mode (frontend replicas, worker nodes, PostgreSQL), split by a Ginkgo label. Neither starts a message broker: a distributed deployment needs PostgreSQL and the frontends' own HTTP listener, and the `nats-io` modules are not in the build.
-`make test-e2e-distributed` runs `Distributed && !VLLMMultinode && !Cluster` over `./tests/e2e/distributed` recursively. Services are wired directly into the test binary. 231 specs in about 2m20s (measured 133.3s for the main suite plus the 8 argument-validation specs in the subpackage).
-`make test-e2e-cluster` runs `Cluster` and spawns real `local-ai` child processes through the `tests/e2e/distributed/cluster` helper package. It **builds `./local-ai` itself** (the `e2e-binary` target), so you do not run `make build` first and cannot forget to. 26 specs, 933.8s of specs in a 15m37s target run (four measurements; the suite grew from 6 specs and roughly 509s, then 21 specs and 802.5s, then 24 specs and 897 to 907s, so budget on the newest number).
Both jobs live in `.github/workflows/tests-e2e-distributed.yml`, with `timeout-minutes: 45` each. They trigger on pull requests *and* on every push to `master`; the `paths-ignore` filter (see [.agents/ci-caching.md](ci-caching.md)) sits on the pull-request trigger only, so a master push always runs both. They are advisory only because `master` carries no branch protection, which is a repository setting and not a YAML key: `continue-on-error: true` would flip the run's *conclusion* to success and hide the failure, so it is not used.
- **The container is suite-scoped, not spec-scoped.** `SetupInfra` used to start a PostgreSQL (~10s) and a broker (~3.5s) per spec. Across the 213 specs behind it that was roughly **48 minutes of pure container startup per run**, which is why this suite was never in CI. (213 rather than the ~240 above: the larger number is everything the label filter selects, the smaller one is just the specs that call `SetupInfra`.) The broker container is gone entirely, and the PostgreSQL one now starts once in `BeforeSuite` with each spec getting its own database via `CREATE DATABASE` (~67ms), which is what the `dbName` argument was always describing. Adding a spec needs no change: call `SetupInfra("some-name")` as before, the name is a prefix and a counter keeps it unique.
- **Consequence for new specs:** a fan-out carrier is *shared* within a Ginkgo process, so a wildcard subscriber can observe another spec's traffic. Filter assertions on an identifier your spec owns (a node ID, a job ID) instead of counting everything on `jobs.*.progress`, and verify the spec with `--randomize-all`.
- **`BeforeSuite`, not `SynchronizedBeforeSuite`.** Under `ginkgo -p` each process then gets its own container, keeping each process's carrier isolated. A single shared database across parallel processes would let specs on different processes see each other's messages on the same subject.
- **The label split.** The 8 argument-validation specs under `tests/e2e/distributed/cluster/` carry `Label("Distributed")` only, on purpose: they need no binary and no PostgreSQL, so they belong in the fast job. That is why `test-e2e-distributed` keeps `-r` (it must reach the subpackage) and `test-e2e-cluster` deliberately does **not** (the subpackage is out of its scope).
- **The binary is built, and a stale one is refused.** `test-e2e-cluster` used to take `./local-ai` as given and only check that a file by that name existed. An edit that was never rebuilt therefore ran the entire suite against an older build while reporting on the working tree, which surfaced once as a spec failing on caller line numbers that matched no line in any source file. Two things close it. The target depends on `e2e-binary`, which is `go build -o local-ai ./cmd/local-ai` and not `make build` (that one also builds the React UI, which this suite never touches and CI stubs). And `localAIBinary` in the harness **fails**, on a developer box as well as under CI, when the binary is older than the newest non-test Go source in the tree, which covers a run started with `LOCALAI_E2E_BINARY` or by invoking ginkgo directly. `_test.go` files are excluded from that scan on purpose: they compile into the ginkgo suite and never into `local-ai`, so counting them would fire on every edit to the spec being run. Set `LOCALAI_E2E_BINARY` to opt out of the build; there is no opt-out of the staleness check, because a rebuild is the remedy.
- **`--fail-on-empty` is load-bearing on both targets.** Ginkgo exits 0 when a label filter selects nothing, so without it a refactor that renames or drops `Label("Cluster")` leaves the target reporting "Test Suite Passed" having started no cluster at all. `LOCALAI_E2E_REQUIRE_BINARIES` does not cover this case: it only fires inside a spec that is actually running.
- **The binary gate.** `localAIBinary()` and `mockBackendBinary()`**fail** rather than skip when `CI` is set, or when `LOCALAI_E2E_REQUIRE_BINARIES` is truthy; `LOCALAI_E2E_REQUIRE_BINARIES=0` (also `off`, `no`, `n`, `disabled`, and anything `strconv.ParseBool` reads as false) forces skipping even under CI. **Any value that parses as neither reads as ON**, not off: setting the variable to something meaningless means someone meant to turn the gate on, and reading it as false would quietly restore the silent skip the flag exists to remove. The whole polarity is deliberate, because in CI a skipped cluster spec is indistinguishable from a passing one: Ginkgo exits 0 on skips. Locally a missing binary still just skips, since `CI` is unset in an ordinary shell.
- **Flake budget: no retries at all.** `--flake-attempts` is *total attempts*, not retries (ginkgo v2.29.0 `internal/group.go` sets `maxAttempts = FlakeAttempts` and loops `attempt < maxAttempts`; the flag's own usage string reads "0 - failed tests are not retried"). `DISTRIBUTED_TEST_FLAKES` defaults to **1**, so each spec runs once and a failure is a failure, and `test-e2e-cluster` pins `--flake-attempts 1` outright rather than reading the variable. The repo-wide `TEST_FLAKES=5` means up to five attempts, so up to four retries. These suites exist to surface nondeterminism, and a retry converts exactly that signal into a green run. Raise it locally when bisecting something unrelated, not in the Makefile.
- **Coverage:** `tests/e2e/distributed` is excluded from the coverage roots (`COVERAGE_E2E_ROOTS = ./tests/e2e`, run non-recursively), and so is the `cluster` helper package beneath it. Neither suite moves the baseline, so production code that these suites are the only cover for reads as **uncovered**. Unit tests for such code belong under `./core/...` with `testutil.SetupTestDB()`.
- **The cluster job builds against a stubbed React UI.** `core/http/react-ui/dist` is gitignored and built by Node, so the workflow writes a one-line `index.html` there to satisfy the `//go:embed react-ui/dist/*` in `core/http/app.go` and skips a full Node and Vite install. That holds only while the suite drives the HTTP API and never the UI, which has its own e2e suite. A spec that ever asserts on a UI asset would pass locally, where a real `dist/` exists, and be served the stub in CI: if you write one, the stub step has to go and the real build come back.
- **The topology that matters is two frontends and two workers.** `cluster_busless_test.go` runs the whole programme on it, and it is the only shape in which the OWNER path and the RELAY path are live at the same time: with the two workers' tunnels landing on different replicas, every request either goes to the replica holding that worker's tunnel or is relayed through the one that does. A one-worker cluster cannot tell those apart, and cannot state that killing a replica re-homes only ITS worker. Which worker served a request is read back from `node_models` rather than assumed, and the two models are pinned to one worker each through the admin API (`PUT /api/nodes/:id/labels` plus `POST /api/nodes/scheduling`), so a selector that failed open reddens the spec instead of hiding in it.
- **The fan-out carrier is proven under real binaries by `cluster_fanout_test.go`.** Both specs run two frontends and no workers against one PostgreSQL, publish at frontend 0 and read at frontend 1, and cost 7.0s together. They drive a gallery operation on purpose: `GET /models/jobs/<id>` answers from `galleryop`'s in-memory statuses map, which on a peer replica is filled by the `gallery.*.progress` subscriber and by nothing else (`Hydrate` runs once at startup, long before these operations exist), so a peer that answers was told over the carrier. Every other broadcast family has a durable table behind it that a peer would converge through anyway, and a spec on one of those cannot separate "the broadcast arrived" from "the row was read". The specs make that checkable rather than argued: the `gallery_operations` row is written when the gallery worker DEQUEUES an operation, so an operation still queued has none, and both specs assert zero rows while the peer is already answering with the operation's own bytes. Holding the queue is what `cluster.Options.Galleries` is for: it points every frontend at one gated index and turns the startup estimate warmer off, so the operation the spec submits is parked inside a fetch only the spec can complete. The second spec adds the oversized path: an element name of 9000 bytes puts the encoded notification past PostgreSQL's 8000-byte cap (exclusive), so the broadcast becomes a `bus_messages` row plus a notification carrying its id, and the spec asserts the row exists, decodes it, and compares its element name byte for byte with what frontend 1 answers. A one-byte control operation in the same run is asserted to leave NO row, which is what keeps "a row exists" a statement about size rather than one that is true of every operation.
- **Do not shorten the cluster suite's waits.** Three of its six specs sit at ~167s each because they wait out a 60s staleness threshold plus a 15s health-check tick. That wait is what stops the assertions from passing before the system could have reacted, which was a real false green earlier on. If the job has to get faster, the levers are CI concurrency or making the thresholds configurable, not shorter waits.
@@ -122,18 +122,89 @@ The per-backend prefix match only sees files under a backend's own directory, so
| Changed path | Rebuilds |
|---|---|
| `backend/backend.proto` | everything (all languages compile or copy it) |
| `backend/backend.proto` | nothing if the edit is additive-only, otherwise everything (see below) |
| `backend/Dockerfile.<x>` | the Linux entries whose `dockerfile:` names it |
| `backend/python/common/` | Python, Linux + Darwin |
| `scripts/build/package-gpu-libs.sh` | Python, Linux only |
| `scripts/build/package-gpu-libs.sh` | every Linux entry (Python, Go and C++ all run it) |
| `scripts/build/<lang>-darwin.sh` | the Darwin entries that build target routes to |
| `.github/workflows/backend_build[_darwin].yml` | everything on that OS |
| anything else under `scripts/build/` (except `*_test.sh`) | everything — conservative default for unclassified packaging inputs |
Deliberately excluded: `backend/index.yaml` (gallery metadata, never enters an image), `.github/backend-matrix.yml` (adding a backend would rebuild all of them), `backend/Dockerfile.base-grpc-builder` (owned by `base-images.yml`), and the root `Makefile` (touched in ~11% of commits, and its backend-relevant edits arrive alongside the backend directory anyway). `make test-ci-scripts` pins all of this.
#### `backend/backend.proto` is content-filtered, not path-filtered
Every language consumes the proto, so a path rule for it can only ever say "rebuild all 473 images". It changes in ~1.3% of commits, and that was enough to make it the single largest CI cost driver in the repo: on 2026-07-29 four runs totalling 935 queued jobs traced to nothing but a proto edit, one of which (#11158) was a six-line diff adding `bool cache_prompt = 8;`.
An additive proto edit cannot change how a backend that never references the new symbol behaves, so `filterMatrix()` suppresses the rule for one. `changed-backends.js` fetches `backend/backend.proto` at the base revision (same contents-API pattern as `.github/backend-matrix.yml`) and hands both texts to `protoChangeIsAdditive()`, which compares them structurally rather than textually:
- **Additive, rebuilds nothing**: a new field with an unused number, a new message, a new enum value, a new RPC. Comment, whitespace and ordering changes also land here.
- **Breaking, rebuilds everything**: a removed, renumbered, retyped or renamed field, a dropped RPC, a changed `option` or `package`. So does an unresolvable base revision, matching the run-all posture used for a truncated diff.
Checked against every proto commit in the preceding six months, all nine resolvable ones classify as additive. Note the tradeoff this accepts: generated stubs do change for an additive edit, so image bytes would differ on a rebuild even though behavior does not. That is the same standard already applied when the filter declines to rebuild on unrelated `pkg/` changes, and the weekly cron remains the backstop.
The Sunday 06:00 UTC cron on `backend.yml` exists specifically because path filtering can leave Python backends frozen on stale wheels. `DEPS_REFRESH` (below) only fires when the build actually runs, so an untouched Python backend would never re-resolve its unpinned deps. The weekly cron is the safety net.
## Content-blind PRs skip the workflows that cannot see them
`backend_pr.yml` and `test-extra.yml` filter themselves (matrix generation and a `detect-changes` job), so a gallery-only or docs-only PR costs them about one job each. The Go and image workflows had no filter of any kind, so a one-line `gallery/index.yaml` edit queued 20 jobs, and a docs-only PR queued the same.
This is worth more than it looks. Measured over the week to 2026-07-30, **97% of CI wall-clock is queueing, 3% is execution** (median queue ~5h against a 4-20min median job). Cutting job count is therefore the only lever that shortens feedback time; making individual jobs faster moves 3%.
The volume is real: 13 gallery-only PRs merged that week with 10 open at once, and 78 of the 137 PRs opened were bot-generated.
`paths-ignore` on the PR trigger of `image-pr.yml` (7 jobs), `build-test.yaml` (3), `lint.yml` (2) and `tests-e2e.yml` (1) drops 13 of those 20, measured before `tests-e2e-distributed.yml` (2 jobs) landed. That workflow carries the same exclusion set for the same reason: its dependency graph is 99 packages, so an allowlist of paths would silently stop guarding the moment code moved, while a diff confined to the paths below provably cannot reach it. The excluded set:
| Path | Why no image or Go build can see it |
|---|---|
| `gallery/**` | Model-gallery metadata, parsed at runtime, never copied into an image |
| `docs/**`, `examples/**`, `**/*.md` | Never enter an image or a binary. `lint.yml` already excluded these before gallery was added |
### `backend/{cpp,go,python}/**` on `image-pr.yml` and `build-test.yaml` only
Version-pin bumps dominate PR volume: 48 `update/*` PRs in the week to 2026-07-30, from 16 pins, each a two-line diff. Most edit nothing but one `backend/*/<name>/Makefile`.
Neither of those two workflows can observe such a change. `make build` is `go build ./cmd/local-ai`, GoReleaser builds the same plus `./cmd/launcher`, and the core image's final stage ships only `entrypoint.sh`, `healthcheck.sh` and that binary. The per-backend trees are copied into the builder but nothing in them reaches the output.
What still triggers a full run, because none of it lives under those prefixes:
-`backend/backend.proto` — feeds `protogen-go`, so it does change the binary.
-`go.mod` / `go.sum` — the `go mod tidy` before-hook.
-`backend/Dockerfile.*` and anything else directly under `backend/`.
Deliberately **not** applied to:
| Workflow | Why it must keep seeing `backend/**` |
|---|---|
| `test.yml` | `TEST_PATHS` explicitly includes `./backend/go/cloud-proxy/...`, `./backend/go/local-store/...` and `./backend/go/valkey-store/...` |
| `lint.yml` | `.golangci.yml` carries `backend/`-scoped rules, so golangci-lint covers that tree |
| `tests-e2e.yml` | The e2e suite drives real backends over gRPC |
| `backend_pr.yml` | This is the workflow whose entire job is to rebuild the changed backend |
What still runs, and why it has to:
| Workflow | Why it keeps running |
|---|---|
| `test.yml` (`tests`) | `core/gallery/variants_lint_test.go` reads the real `gallery/index.yaml` and asserts the index invariants (no duplicate entry names, no build claimed by two parents). This is the only schema-level check the gallery has. |
| `yaml-check.yml` (`Yamllint`) | Lints `gallery/` for syntax. |
| `backend_pr.yml`, `test-extra.yml` | Already self-filtering; they stop after the detect step. |
Two properties this relies on:
-`paths-ignore` skips a run only when **every** changed file matches, so a PR touching the gallery *and* Go code still runs everything. That is what makes the exclusion safe rather than a hole.
-`master` carries no branch protection and no rulesets, so a skipped workflow reports no status and nothing waits on it. If required status checks are ever introduced, these five entries must be excluded from the required set or PRs will hang on "Expected — Waiting for status to be reported".
### `image.yml` on master push is gated too, by a job rather than a path filter
The same reasoning applies to master pushes, and the volume is larger there: on 2026-07-30, **12 of the 23 queued `image.yml` runs** were commits like "add 1 new model to gallery" or a docs fix, each rebuilding all 18 container images.
`image.yml` now has a `changes` job that decides once whether the push can affect any image; the other 11 jobs carry `needs: changes` plus an `if:` on its output. Verified against the shipped `Dockerfile`: the final stage copies only `entrypoint.sh`, `healthcheck.sh` and the `local-ai` binary, there is no `go:embed` of `gallery/` or `docs/`, and the gallery is fetched at runtime from `https://index.localai.io/models` (a caching mirror of `gallery/index.yaml` on master, with `github:mudler/LocalAI/gallery/index.yaml@master` as the fallback mirror). A gallery-only commit therefore produces byte-identical images, and the gallery change reaches users over the network immediately whether or not an image is rebuilt.
Two properties to preserve if you touch it:
- **It is a job gate, not `paths-ignore`.** `paths-ignore` on `push` also applies to tag pushes, and a tag created on an existing commit carries an empty commits list, which would silently skip the release image build. The gate short-circuits to "build" for `refs/tags/*`, and for any push whose base commit is missing, zero, or unresolvable.
- **The merge jobs must name the gate explicitly.** They use `if: ${{ !cancelled() && ... }}`, and `!cancelled()` is true when a dependency is *skipped*, so without the extra condition they would run and try to merge manifest lists for images that were never built.
## The `DEPS_REFRESH` cache-buster (Python backends)
Every Python backend goes through the shared `backend/Dockerfile.python`, which ends with:
@@ -169,15 +240,38 @@ RUN --mount=type=cache,target=/root/.ccache,id=<backend>-ccache-${TARGETARCH}-${
bash /usr/local/sbin/compile.sh
```
The compile script exports `CMAKE_C/CXX/CUDA_COMPILER_LAUNCHER=ccache` so CMake threads ccache through gcc/g++/nvcc. `cache-to: type=registry,mode=max` exports the cache mount data into the registry cache, so subsequent builds restore it.
The compile script exports `CMAKE_C/CXX/CUDA_COMPILER_LAUNCHER=ccache` so CMake threads ccache through gcc/g++/nvcc. Cache scope is per `(TARGETARCH, BUILD_TYPE)` so e.g. cublas-12 doesn't share with cublas-13 (their CUDA headers differ; cross-pollination would just be cache misses anyway).
On a `LLAMA_VERSION` bump, most translation units are byte-identical to the previous version's preprocessed source — ccache returns the previous `.o` and skips the real compile. Same for LocalAI source changes that don't actually touch llama.cpp's CMake inputs. Cache scope is per `(TARGETARCH, BUILD_TYPE)` so e.g. cublas-12 doesn't share with cublas-13 (their CUDA headers differ; cross-pollination would just be cache misses anyway).
### ⚠️ This ccache does nothing in CI today
This section previously claimed that `cache-to: type=registry,mode=max` "exports the cache mount data into the registry cache, so subsequent builds restore it". **That is not true.** BuildKit does not export the contents of a `--mount=type=cache` to a registry cache export. A cache mount lives in the builder's local state, and every CI job gets a fresh runner with a fresh builder, so `/root/.ccache` starts empty on every single build.
Measured on 2026-07-30 from the `ccache -s` output the compile script already prints (it runs `ccache -z` first, so the numbers are per-build):
| Job | Commit touched | Build time | ccache |
|---|---|---|---|
| 89766266951 (llama-cpp, cublas 13) | `backend/go/magpie-tts-cpp/Makefile` only | 6369s | **0 / 889 hits**, and 0 / 1778 |
The first two are the decisive control: commit `90355cd44` changed exactly one file, `backend/go/magpie-tts-cpp/Makefile`, nowhere near llama.cpp. The engine source was byte-identical to the previous build, which is precisely the case this section says ccache should serve, and the hit rate was still **0.00%**. A cache that was being restored but merely matching poorly would show partial hits; 0-of-N is the signature of an empty cache.
So the paragraph above about `LLAMA_VERSION` bumps reusing previous `.o` files describes an intended design that is not in effect. `Dockerfile.{llama-cpp,ik-llama-cpp,turboquant,bonsai,ds4,privacy-filter}` pay the ccache wrapper overhead and get nothing back. Multi-hour C++ rebuilds are recompiling identical translation units from scratch.
**Do not "fix" this by adding cache mounts to more Dockerfiles.** Wiring the same mount into `Dockerfile.golang` (215 of the 434 matrix entries) was measured locally at 18% faster on a rebuild after a source edit, with a 71.5% ccache hit rate — but only because the local test reused one builder across both builds. In CI it would be a no-op for exactly the reason above.
Making this actually work needs the cache to live outside the builder. The options, none of them free:
- **ccache `remote_storage`** (ccache ≥ 4.4, HTTP or Redis backend) or **sccache** with an S3/GCS/Redis backend. Genuinely works across runners; needs a cache service to point at. quay.io is a registry, not a blob store, so the existing infra does not cover it.
- **Round-trip the cache dir through `actions/cache` on the runner**: restore it, pass it in, and export it back out via a build stage output. No external infra, but clunky, and the repo already sits at GitHub's 10 GB cache ceiling while the llama-cpp ccache alone is capped at 5 GB.
Until one of those lands, treat C++ backend builds as always-cold and spend the effort on not running them instead (path filtering, see above).
## Composite actions
Two composite actions handle runner-side prep:
- **`.github/actions/free-disk-space/action.yml`** — wraps `jlumbroso/free-disk-space@main` plus an explicit apt purge of dotnet/android/ghc/mono/etc. Reclaims ~6–10 GB on `ubuntu-latest`. No-op on self-hosted runners. Used by `backend_build.yml`, `image_build.yml`, `test.yml`, `tests-aio.yml`, etc.
- **`.github/actions/free-disk-space/action.yml`** — wraps `jlumbroso/free-disk-space@main` plus an explicit apt purge of dotnet/android/ghc/mono/etc. Reclaims ~6–10 GB on `ubuntu-latest`. No-op on self-hosted runners. Used by `backend_build.yml`, `image_build.yml` and `base-images.yml` — the jobs that actually build images. Deliberately **not** used by `test.yml`, which runs no buildx step.
- **`.github/actions/setup-build-disk/action.yml`** — relocates Docker's data-root to `/mnt` on hosted X64 runners. GHA hosted `ubuntu-latest` ships ~75 GB of unused space at `/mnt`; combined with the free-disk-space cleanup this gives ~100 GB working space — enough for ROCm dev image + vLLM torch install + flash-attn intermediate layers. No-op on self-hosted and on non-X64 hosted runners. Used by `backend_build.yml`, `image_build.yml`, `base-images.yml`.
Both actions run before any docker buildx step.
@@ -218,10 +312,20 @@ Eviction is rarely needed in normal operation — `DEPS_REFRESH` handles weekly
## What the cache does **not** cover
- The `free-disk-space` and `setup-build-disk` composite actions run on every job — these reclaim runner-state, not Docker layers, so BuildKit caches don't apply.
- The `free-disk-space` and `setup-build-disk` composite actions run on every job — these reclaim runner-state, not Docker layers, so BuildKit caches don't apply.`test.yml` deliberately does **not** use `free-disk-space`: it runs no buildx step, and the multi-GB fixture downloads that once justified it left `make test` in the test-suite reorg.
- Intermediate artifacts of `Build (PR)` are not pushed anywhere — PRs only build for verification.
- Darwin builds (see below) — macOS runners have no Docker daemon, so the registry-backed BuildKit cache cannot apply.
### The Linux Go workflows set `cache: false` on purpose
`test.yml`, `lint.yml`, `tests-e2e.yml` and friends pass `cache: false` to `actions/setup-go@v5`, unlike the darwin jobs. This looks like an oversight and is not.
Measured over the week to 2026-07-30, the `Set up Go` step has a **median of 11 seconds** on these runners. There is essentially nothing to win: the module download is not where the time goes. The expensive steps are compilation and test execution (`Test (with coverage gate)` at ~18.6min, `Test Backend E2E` at ~14.5min), and Go's build cache would have to survive across runners to touch those.
Enabling it also has a real cost. GitHub caps Actions cache at **10 GB per repo and the repo already sits at that ceiling** (31 entries), so every `setup-go` entry written by a branch with a distinct `go.sum` (222-375 MB on Linux, up to 1.4 GB on macOS) evicts something else. See the darwin cache budget below.
Before re-enabling this, measure `Set up Go` again and confirm it has actually become slow. If room is needed in the 10 GB budget, the cheapest evictions are the `docker.io--tonistiigi--binfmt` entries (~30 MB each, trivially re-fetched).
## Darwin native caches
`backend_build_darwin.yml` runs natively on `macOS-14` GitHub-hosted runners — there is no Docker, no BuildKit, no cross-job registry cache. Instead, the reusable workflow uses `actions/cache@v4` for four native caches that mirror the spirit of the Linux cache (warm by default, weekly refresh for unpinned Python deps, PRs read-only).
@@ -255,6 +359,26 @@ GitHub Actions caches are limited to 10 GB per repo. Steady-state worst case: ~8
One residual self-hosted reference remains in `test-extra.yml` (`tests-vibevoice-cpp-grpc-transcription` uses `bigger-runner` for the 30s JFK-decode timeout headroom). That's a separate concern.
### Small always-on jobs routed to `arc-runner-set`
The hosted pool is shared across the whole *account*, not per repo, so a burst in one repo starves the others. On 2026-07-31 it went to **zero scheduled jobs for 35 consecutive minutes** with 39 jobs queued, while `arc-runner-set` completed 12 jobs without interruption over the same window. Actions was healthy globally at the time (other public repos were scheduling normally), so this is an account-level throttle, not an outage.
`gh-pages.yml` (`build` + `deploy`) is therefore routed to `arc-runner-set` when `github.repository == 'mudler/LocalAI'`. It needs no fork-safety clause because it only triggers on push-to-master and `workflow_dispatch`, so it never executes pull-request code. The repository guard keeps forks (which have no such runner label) from queueing forever. It fetches its own toolchains via `setup-go` / `actions-hugo` and uses no `sudo`/`apt`.
#### What the `arc-runner-set` image actually contains
Measured 2026-07-31 on run `30637392862` by a preflight step, not assumed:
That is why `lint.yml` is **not** on the self-hosted pool. Both of its jobs were routed there and both failed in one second: `golangci-lint` needs `make` (for `make protogen-go`, itself needing `curl`+`unzip` to fetch protoc, and for `make lint`), and `build-scripts` additionally needs a C toolchain because the packaging-script tests compile a throwaway binary and inspect it with `ldd`. Both jobs are back on `ubuntu-latest`.
The preflight steps were deliberately left in place. They cost about a second on the hosted pool and mean that whenever the runner image gains `make` + `gcc`, re-routing is one `runs-on:` line per job and any remaining gap reports itself by name rather than as an opaque mid-build failure.
Note for any future re-route: `lint.yml` also triggers on `pull_request`, and a fork PR runs untrusted contributor code. That must never reach a persistent self-hosted runner, so any re-route has to stay push-only, e.g. `${{ (github.event_name == 'push' && github.repository == 'mudler/LocalAI') && 'arc-runner-set' || 'ubuntu-latest' }}`.
## Touching the cache pipeline
When changing `image_build.yml`, `backend_build.yml`, any of the `backend/Dockerfile.*` files, `Dockerfile.base-grpc-builder`, `.docker/install-base-deps.sh`, `.docker/<backend>-compile.sh`, or `scripts/changed-backends.js`:
@@ -70,3 +70,37 @@ The project documentation is located in `docs/content`. When adding new features
- **Configuration**: If you modify configuration options, update the relevant sections in `docs/content/`.
- **Examples**: providing concrete examples (like YAML configuration blocks) is highly encouraged to help users get started quickly.
- **Shortcodes**: Use `{{% notice note %}}`, `{{% notice tip %}}`, or `{{% notice warning %}}` for callout boxes. Do **not** use `{{% alert %}}` — that shortcode does not exist in this project's Hugo theme and will break the docs build.
## React UI styling
The React UI ships a design system in `core/http/react-ui/src/App.css`: design
tokens, form grids, data tables, stat cards, callouts, plus a small semantic
@@ -94,4 +94,4 @@ The user chose KISS. Every mutating tool has a safety rule (`prompts/10_safety.m
## Distributed mode
The in-memory MCP server runs only on the head node (where the chat handler runs). `inproc.Client` wraps services that are already distributed-aware (`GalleryService` coordinates with workers; `ListNodes` reads the NATS-populated registry). No NATS routing of MCP tools — the admin surface lives on the head, period.
The in-memory MCP server runs only on the head node (where the chat handler runs). `inproc.Client` wraps services that are already distributed-aware (`GalleryService` coordinates with workers; `ListNodes` reads the PostgreSQL-backed node registry). MCP tools are not routed anywhere: the admin surface lives on the head, period.
A release is not finished when the tag is pushed. The GitHub release, the blog post and the demo clips ship together, because the changelog says what moved and the post and the clips are what make anyone care.
## What a release must include
1.**Labels on the merged PRs.** GitHub generates the raw notes from PR labels, so label first, generate second. Wrong labels mean a miscategorised changelog that has to be edited by hand.
2.**`RELEASE_NOTES_vX.Y.Z.md`** at the repository root, in the house style: what changed, why it matters, PR numbers so people can read the diffs.
3.**A blog post under `website/content/blog/`.** One post per release, front matter with `title`, `date`, `author`, `category: "Release"`, `tags`, `summary` and `extracss: ["blog.css"]`. Cover the two or three changes that alter what a user does day to day, not the whole changelog, and link the PR numbers. See `website/content/blog/what-landed-in-localai-4-8.md` for the shape.
4.**Demo clips for the notable features.** Anything visible (a new backend, a UI change, a new endpoint, a measured speedup) gets a short screen recording. Put the file in `website/static/media/`, reference it from the blog post, and reuse it on the marketing pages where it fits.
A release without a post and without clips is incomplete, in the same way a user-facing code change without a docs update is incomplete.
## Clip conventions
- MP4, H.264, no audio track unless the feature is about audio. Keep them short (10 to 30 seconds) and loopable.
- Record the real thing. A clip from the engine's own benchmark suite or a real session, never a mockup.
- Where the change is a speedup, record both sides on the same machine on the same input, so the comparison is honest.
- Name the file after the feature, not the release (`vllm-race.mp4`, not `v4-8-demo.mp4`), so it stays reusable once the release is old.
- The marketing site plays clips with `muted loop playsinline preload="none"` and a `data-lazy` attribute, which the site's IntersectionObserver uses to play and pause them on scroll. Follow that pattern for anything you add.
## Order of work
Label the PRs, generate and edit the release notes, cut the draft release, record the clips while the branch is still fresh in your head, then write the post against the notes and the clips. Publishing the release and merging the post should happen on the same day.
The `creating-localai-releases` skill drives steps 1 to 3 and captures the React UI screenshots that go into the notes.
Beyond the parser names above, `Options[]` carries `--` prefixed engine flags (`--enable-prefix-caching`, `--kv-cache-dtype:fp8_e5m2`). `apply_options_to_engine_args` in `backend/python/common/vllm_utils.py` maps them onto `AsyncEngineArgs` fields, and it must run **before**`AsyncLLMEngine.from_engine_args()` - applying them afterwards is a silent no-op, which is exactly what issue #11130 was.
Things to keep straight when touching this:
- Precedence is typed proto fields → `options:` → `engine_args:`. `applyEngineArgDefaults` in `core/config/hooks_vllm.go` therefore skips seeding a production default whose key the user already set as an option, otherwise the later `engine_args:` pass would silently override them.
- Only `--` prefixed entries are engine flags; `tool_parser:`/`reasoning_parser:` and friends keep their meaning. Parser lookups accept both spellings via `normalize_option_key`.
- Unknown or uncoercible flags warn and are skipped, unlike `engine_args:` which is strict - `Options[]` is a shared bag and knows entries this mapping doesn't.
- Field types come from the annotation's *base* (`Literal["auto","float16"]` is not a float). The helper's tests are stdlib-only: `make test-python-helpers`.
Auto-defaults for known model families live in `core/config/parser_defaults.json` and are applied:
- at gallery import time by `core/gallery/importers/vllm.go`
- at model load time by the `vllm` / `vllm-omni` backend hook in `core/config/hooks_vllm.go`
stale-issue-message:'This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days.'
stale-pr-message:'This PR is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 10 days.'
LocalAI serves both single-host users who want to install and try models quickly and experienced developers, ML engineers, system administrators, and DevOps operators who manage production hosts or distributed clusters. The interface must support first-time discovery without hiding the runtime state, configuration, and control that returning operators need.
### Brand Personality
Capable, easy to use, and trustworthy. The interface should make sophisticated local-AI infrastructure feel understandable and under control. It should be direct and calm rather than playful, ornamental, or intimidating.
### Aesthetic Direction
Use LocalAI's established technical, editorial design language: Geist typography, compact information density, sharp geometry, deep blue-black surfaces, action blue, mint for healthy/local/live state, and amber only for decisions requiring attention. Support both dark and light themes. Avoid generic card dashboards, decorative gradients, glass effects, and visual noise.
### Design Principles
1. Use progressive disclosure to serve newcomers and operators in the same workflow: make the common path obvious, then reveal operational depth in context.
2. Organize navigation around user intent and lifecycle state, not implementation concepts or nested containers.
3. Give each resource one canonical home; expose discovery, installed state, and runtime state as clear views of that resource instead of duplicating management surfaces.
4. Keep operational status visible and trustworthy through precise labels, explicit scope, and actionable state—not decoration.
5. Preserve information density for expert use while flattening navigation and reducing repeated summaries, tabs, rails, and panels.
Organisations running LocalAI, listed by the people who run it.
If your organisation uses LocalAI and you are happy to say so publicly, open a
pull request adding a row to the table below. That pull request is how we know
we have permission to list you, which is why we do not add anybody ourselves.
You do not need to be a large company, and you do not need to disclose anything
sensitive. A sentence on what you use it for is more useful to other readers
than a logo.
## How to add yourself
1. Add a row to the table, in alphabetical order.
2. Use your organisation's usual name and a link to your site.
3. Say briefly what you use LocalAI for, and whether it is in production.
4. Open the pull request from an account that makes it plausible you speak for
the organisation, or say in the description who you are. We may ask.
To be removed, open a pull request deleting your row, or email
[info@localai.io](mailto:info@localai.io). We will not ask why.
## Who is using LocalAI
<!-- Keep alphabetical. Columns: Organisation | What for | Status -->
| Organisation | What they use it for | Status |
|---|---|---|
| [walcz.de](https://walcz.de) | Self-hosted appliance for a German B2B consultancy: local-only inference on AMD Strix Halo (gfx1151/ROCm), agents with MCP tools, RAG over an internal knowledge base, and a document/bookkeeping pipeline. | Production |
| _Your organisation here_ | | |
## What this list is not
This is not a list of everyone who has ever starred the repository, and it is
not a list of the employers of people who have contributed a patch. Both of
those are easy to scrape and neither means what a logo wall implies.
The website shows two separate things, both of which are checkable without
anybody's permission:
- **Engineers from these companies have contributed code.** Evidence is the
commit history plus the employer on that person's public GitHub profile. It
is a claim about a person, not about their employer.
- **These projects integrate LocalAI.** Evidence is a reference to LocalAI in
that project's own repository or documentation.
Those two lists live in [`website/data/ecosystem.yaml`](website/data/ecosystem.yaml).
This file is the third, stronger thing: organisations that chose to say so.
@@ -8,7 +8,7 @@ Human contributors: see [CONTRIBUTING.md](CONTRIBUTING.md) for the development w
LocalAI follows the Linux kernel project's [guidelines for AI coding assistants](https://docs.kernel.org/process/coding-assistants.html). Before submitting AI-assisted code, read [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md). Key rules:
- **No `Signed-off-by` from AI.** Only the human submitter may sign off on the Developer Certificate of Origin.
- **No `Signed-off-by` from AI.** Only the human submitter may sign off on the Developer Certificate of Origin. One exception: automation a maintainer operates signs off with *that maintainer's* identity, since no other human submitter exists to certify it. See [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md).
- **No `Co-Authored-By: <AI>` trailers.** The human contributor owns the change.
- **Use an `Assisted-by:` trailer** to attribute AI involvement. Format: `Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]`.
- **The human submitter is responsible** for reviewing, testing, and understanding every line of generated code.
@@ -32,6 +32,8 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
| [.agents/adding-gallery-models.md](.agents/adding-gallery-models.md) | Adding GGUF models from HuggingFace to the model gallery |
| [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) | LocalAI Assistant chat modality — adding admin tools to the in-process MCP server, editing skill prompts, keeping REST + MCP + skills in sync |
| [.agents/preparing-a-release.md](.agents/preparing-a-release.md) | Cutting a release: PR labels, `RELEASE_NOTES_vX.Y.Z.md`, the blog post under `website/content/blog/`, and the demo clips under `website/static/media/` |
| [.impeccable.md](.impeccable.md) | Design context for UI/UX work — users, brand personality, aesthetic direction, and design principles |
## Quick Reference
@@ -42,6 +44,7 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
- **Docs (docs-with-code rule)**: When you change user-facing behavior (API endpoints, CLI flags, config keys, or features), update the corresponding page under `docs/content/` in the SAME change, not as a follow-up. A user-facing change without a matching docs update is incomplete. See also the documentation conventions in [.agents/coding-style.md](.agents/coding-style.md).
- **New API endpoints**: LocalAI advertises its capability surface in several independent places — swagger `@Tags`, `/api/instructions` registry, auth `RouteFeatureRegistry`, React UI `capabilities.js`, docs. Read [.agents/api-endpoints-and-auth.md](.agents/api-endpoints-and-auth.md) and follow its checklist — missing any surface means clients, admins, and the UI won't know the endpoint exists.
- **Admin endpoints → MCP tool**: every admin endpoint that an admin would manage conversationally (install/list/edit/toggle/upgrade) MUST also be exposed as an MCP tool in `pkg/mcp/localaitools/`. The LocalAI Assistant chat modality and the standalone `local-ai mcp-server` consume that package; drift between REST and MCP is a real risk. Read [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) — the `TestToolHTTPRouteMappingComplete` test fails until you wire the new tool and update the route map.
- **Releases ship with a post and clips**: a release is not done at the tag. It needs labelled PRs, `RELEASE_NOTES_vX.Y.Z.md`, a blog post under `website/content/blog/`, and a short demo clip in `website/static/media/` for each notable feature. See [.agents/preparing-a-release.md](.agents/preparing-a-release.md).
- **Build**: Inspect `Makefile` and `.github/workflows/` — ask the user before running long builds
- **Backend OS coverage**: a new backend must target every OS it can build for, not just Linux. `.github/backend-matrix.yml` has two matrices — `include:` (Linux) and `includeDarwin:` (macOS / Apple Silicon). Most C/C++/GGML and many Python backends build on Darwin too — wire the `includeDarwin` entry + `backend/index.yaml``metal:` entries, or say in the PR why an OS is unsupported. See the darwin checklist in [.agents/adding-backends.md](.agents/adding-backends.md).
- **Gallery variant ranking**: a gallery entry can declare `variants` (alternative builds of the same weights), and LocalAI ranks the ones a host can run by engine preference first, size second. A new backend that should be preferred on some hardware must be listed in `engineNamePreferenceRules` in `pkg/system/capabilities.go`; the sibling `backendBuildTagPreferenceRules` speaks build tags rather than engine names, and using the wrong table matches nothing without erroring. See [.agents/adding-backends.md](.agents/adding-backends.md).
@@ -218,7 +218,7 @@ LocalAI follows the **same guidelines as the Linux kernel project** for AI-assis
The full policy for this repository lives in [`.agents/ai-coding-assistants.md`](.agents/ai-coding-assistants.md). Summary:
- **AI agents MUST NOT add `Signed-off-by` tags.** Only humans can certify the Developer Certificate of Origin.
- **AI agents MUST NOT add `Signed-off-by` tags.** Only humans can certify the Developer Certificate of Origin. Automation operated by a maintainer is the one exception: it signs off with that maintainer's identity, because there is no other human submitter to certify it.
- **AI agents MUST NOT add `Co-Authored-By` trailers** attributing themselves as co-authors.
- **Attribute AI involvement with an `Assisted-by` trailer** in the commit message:
@@ -265,6 +265,37 @@ The e2e tests run LocalAI in a Docker container and exercise the API:
make test-e2e
```
### Running distributed-mode tests
Distributed mode (several frontend replicas, worker nodes and PostgreSQL) has two suites. Both bring up their PostgreSQL with testcontainers, so Docker has to be available. There is no message broker to bring up: a distributed deployment needs PostgreSQL and the frontends' own HTTP listener, and nothing else.
```bash
make test-e2e-distributed # in-process: services wired directly into the test binary
make test-e2e-cluster # process-level: real local-ai child processes
```
`make test-e2e-distributed` is the fast one (231 specs in roughly 2m20s). It starts one PostgreSQL for the whole run and gives each spec its own database. It runs each spec exactly once, with no retry: `DISTRIBUTED_TEST_FLAKES` defaults to 1 and feeds ginkgo's `--flake-attempts`, which counts *total attempts*, not retries. That is deliberately below the repo-wide `TEST_FLAKES=5`, because this suite exists to catch nondeterministic cluster behaviour and a retry hides exactly the failure it is meant to catch. Raise it locally when bisecting something unrelated.
`make test-e2e-cluster` runs `local-ai` as real child processes, one per frontend replica and one per worker, so a spec can kill a replica and assert what the survivors do. Budget about 13m30s (one measurement: 802.5s of specs in a 13m26s target run, which includes building the binary): several of its 21 specs wait out real staleness and health-check windows. An earlier record of 6 specs and roughly 509s is superseded. It needs a built binary and the mock backend:
```bash
make build build-mock-backend
make test-e2e-cluster
```
Two environment variables steer it:
| Variable | Purpose |
|---|---|
| `LOCALAI_E2E_BINARY` | path to the `local-ai` binary (default: `local-ai` in the repository root) |
| `LOCALAI_E2E_LOG_DIR` | directory for the per-process logs (default: a Ginkgo temp dir) |
Set `LOCALAI_E2E_LOG_DIR` when debugging. A cluster failure is unreadable without the individual frontend and worker logs, and Ginkgo only tells you which assertion failed.
A missing binary skips the cluster specs locally but fails them whenever `CI` is set, so a build problem cannot turn the CI job green without ever starting a cluster. `LOCALAI_E2E_REQUIRE_BINARIES=1` forces that failing behaviour anywhere; `LOCALAI_E2E_REQUIRE_BINARIES=0` forces the skip back on even under CI.
Both suites run in `.github/workflows/tests-e2e-distributed.yml`, on pull requests and on every push to `master`. The `paths-ignore` filter is on the pull-request trigger only, so a master push always runs them.
### React UI tests and coverage
The React UI (`core/http/react-ui/`) is covered by Playwright e2e specs, gated by a **monotonic line-coverage ratchet** (`make test-ui-coverage-check`, run in CI). The metric is non-deterministic — a fast local box reads higher than a slow CI runner for the same code — so a small tolerance is unavoidable.
cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os darwin -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)
cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os darwin -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME) --app-version $(LAUNCHER_APP_VERSION)
cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os linux -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)-linux && mv LocalAI.tar.xz ../../$(LAUNCHER_BINARY_NAME)-linux.tar.xz
cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os linux -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)-linux --app-version $(LAUNCHER_APP_VERSION)&& mv LocalAI.tar.xz ../../$(LAUNCHER_BINARY_NAME)-linux.tar.xz
@@ -161,7 +158,7 @@ local-ai run https://gist.githubusercontent.com/.../phi-2.yaml
local-ai run oci://localai/phi-2:latest
```
To test a running LocalAI server from the terminal, open an interactive chat session from another shell. Inside the prompt, `/models` lists installed models and `/model <name>` switches between them.
To work with a running LocalAI server from the terminal, start the built-in agent from another shell. It answers questions, reads your files and runs commands on your machine, asking you to approve anything that changes state. Inside a session, `/models` lists installed models and `/model <name>` switches between them. See the [Terminal agent](https://localai.io/docs/features/terminal-agent/) docs.
```bash
# Terminal 1
@@ -195,7 +192,7 @@ For more details, see the [Getting Started guide](https://localai.io/basics/gett
- **August 2025**: MLX, MLX-VLM, Diffusers, llama.cpp now supported on Apple Silicon
- **July 2025**: All backends migrated outside the main binary — [lightweight, modular architecture](https://github.com/mudler/LocalAI/releases/tag/v3.2.0)
For older news and full release notes, see [GitHub Releases](https://github.com/mudler/LocalAI/releases) and the [News page](https://localai.io/basics/news/).
For older news and full release notes, see [GitHub Releases](https://github.com/mudler/LocalAI/releases) and the [blog](https://localai.io/blog/).
## Features
@@ -212,7 +209,7 @@ For older news and full release notes, see [GitHub Releases](https://github.com/
- [Built-in Agents](https://localai.io/features/agents/) — Autonomous AI agents with tool use, RAG, skills, SSE streaming, and [Agent Hub](https://agenthub.localai.io)
- [Backend Gallery](https://localai.io/backends/) — Install/remove backends on the fly via OCI images
@@ -231,20 +228,21 @@ Most backends wrap a best-in-class upstream engine. A handful of them are native
| Backend | What it does |
|---------|-------------|
| [vllm.cpp](https://github.com/mudler/vllm.cpp) | From-scratch C++20 port of vLLM for text generation: paged KV cache, continuous batching, prefix caching, safetensors + GGUF loading, engine-enforced structured output, on CPU, CUDA, Metal and Vulkan |
| [vllm.cpp](https://github.com/mudler/vllm.cpp) | From-scratch C++20 port of vLLM for text generation: paged KV cache, continuous batching, prefix caching, safetensors + GGUF loading, engine-enforced structured output, on CPU, CUDA, Metal and Vulkan. Also serves MiniMax-H3 joint video+audio generation |
| [parakeet.cpp](https://github.com/mudler/parakeet.cpp) | C++/GGML port of NVIDIA NeMo Parakeet ASR (tdt/ctc/rnnt/hybrid), with cache-aware streaming transcription |
| [moss-transcribe.cpp](https://github.com/localai-org/moss-transcribe.cpp) | C++/GGML port of OpenMOSS MOSS-Transcribe-Diarize: joint long-form transcription, speaker diarization and timestamping in a single pass |
| [moss-tts.cpp](https://github.com/mudler/moss-tts.cpp) | C++/GGML port of the OpenMOSS MOSS-TTS family: text-to-speech (MOSS-TTS-Local v1.5, 48 kHz stereo) with reference-audio voice cloning, through the MOSS-Audio-Tokenizer neural codec |
| [magpie-tts.cpp](https://github.com/mudler/magpie-tts.cpp) | C++/GGML port of NVIDIA's Magpie TTS Multilingual 357M: 22.05 kHz mono text-to-speech in 5 voices and 9+ languages, with the NanoCodec neural codec and tokenizer/G2P embedded in a single GGUF |
| [ced.cpp](https://github.com/localai-org/ced.cpp) | C++/GGML port of the CED audio-tagging models: sound-event classification (527-class AudioSet) over REST and the realtime API for live recognition |
| [voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp) | Speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion), replacing the Python speaker-recognition backend |
| [voxtral-tts.c](https://github.com/mudler/voxtral-tts.c) | Voxtral Realtime 4B speech-to-text in pure C |
| [voxtral-tts.c](https://github.com/mudler/voxtral-tts.c) | Mistral Voxtral-4B-TTS text-to-speech in pure C: 20 preset voices across 9 languages, 24 kHz WAV output, no dependencies beyond libc |
| [vibevoice.cpp](https://github.com/mudler/vibevoice.cpp) | Native port of Microsoft VibeVoice for TTS (voice cloning) and long-form ASR with speaker diarization |
| [face-detect.cpp](https://github.com/mudler/face-detect.cpp) | Face detection, recognition, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace), replacing the Python insightface backend |
| [free-splatter.cpp](https://github.com/localai-org/free-splatter.cpp) | Pose-free 3D reconstruction (FreeSplatter): turns a handful of plain photos into 3D Gaussians, no camera poses or GPU required |
| [trellis2.cpp](https://github.com/localai-org/trellis2cpp) | C++/GGML port of Microsoft TRELLIS.2: single-image to textured 3D mesh (GLB with PBR materials) |
- [Integrations & community projects](https://localai.io/docs/integrations/)
- [Installation video walkthrough](https://www.youtube.com/watch?v=cMVNnlqwfw4)
- [Media & blog posts](https://localai.io/basics/news/#media-blogs-social)
- [Blog: release write-ups, benchmarks and engineering notes](https://localai.io/blog/)
- [Examples](https://github.com/mudler/LocalAI-examples) — including the [realtime voice assistant demo](https://github.com/localai-org/localai-realtime-demo) (Go client for the Realtime API with tool calling)
## Team
@@ -317,10 +315,6 @@ Past sponsors
A special thanks to individual sponsors, a full list is on [GitHub](https://github.com/sponsors/mudler) and [buymeacoffee](https://buymeacoffee.com/mudler). Special shout out to [drikster80](https://github.com/drikster80) for being generous. Thank you everyone!
## Star history
[](https://star-history.com/#go-skynet/LocalAI&Date)
## License
LocalAI is a community-driven project created by [Ettore Di Giacinto](https://github.com/mudler/) and maintained by the [LocalAI team](#team).
# Gated on cublas because the variable means nothing to any other build, so a
# stray CUDA_MAJOR_VERSION in the environment must not break `make clean` or
# a CPU build. It does still error for `BUILD_TYPE=cublas make clean`, which
# is the right trade: that invocation is asking about a CUDA build tree.
$(error CUDA_MAJOR_VERSION=$(CUDA_MAJOR_VERSION) has no architecture list here (12 and 13do). Leave it empty for a native build, or pass CUDA_ARCHITECTURES explicitly.)
$(warning audio-cpp: libomp not found at '$(LIBOMP_PREFIX)'; building without OpenMP (single-threaded host DSP). Install it with `brew install libomp`, or set LIBOMP_PREFIX.)
CMAKE_ARGS+= -DENGINE_ENABLE_OPENMP=OFF
endif
else
# Portable Linux CPU. Upstream wires this to GGML_BACKEND_DL +
# GGML_CPU_ALL_VARIANTS + $ORIGIN rpath, so one build serves every CPU
Loaded 100 of 1448 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
