Resolve the existing model directory before choosing a missing file. On
macOS, resolving the missing leaf fails and otherwise compares an aliased
path with a canonical allowed directory. Keep this test on the Upload
error branch instead of the containment rejection branch.
Assisted-by: Codex:GPT-6
Resolve allowed directories before comparing them with resolved files.
Otherwise staging rejects valid files under macOS temporary paths.
Cover aliased roots, sibling paths, and symlinks escaping the root.
Assisted-by: Codex:gpt-6
Virtual model names have no primary file to anchor the worker path.
Companion assets still stage successfully, but relative options retain
an incorrect model directory and fail to load.
Derive the worker root from successfully staged option assets when the
primary path is absent. Cover Buffalo packs, files, directories,
overrides, and failed transfers. Document the frontend upgrade.
Assisted-by: Codex:gpt-6 golangci-lint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Merge current master and adapt incoming tests to the distributed APIs.
The sound mock now implements NewClientForNode and uses WorkerLocalAddress.
The upload finalization test supplies the existing direct test dialer.
Assisted-by: Codex:gpt-6
A worker can retain all model bytes with an unfinished-upload marker.
Retries then start at zero and repeatedly fail with HTTP 416.
Verify the existing bytes and finalize same-file retries at full size.
Reuse the normal integrity checks so corrupt content cannot be accepted.
Add regression coverage and document worker recovery.
Assisted-by: Codex:gpt-6 golangci-lint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The general e2e target builds a container, not the host binary that
cluster specs require. Run only its own package; dedicated distributed
and cluster targets retain the full subpackage coverage.
Assisted-by: Codex:GPT-6
Accept original embeddings and timestamps so clients can restore faces
when the in-memory store restarts. Derive stable IDs from exact vectors
to make registration retries preserve identity without duplicate entries.
Assisted-by: Codex:GPT-6 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): stage sound detection audio
Sound detection passes frontend temporary paths directly to remote
workers, unlike transcription. Stage the WAV before classification so
CED can read it without a shared temporary directory.
Preserve the original request for retries and propagate staging errors
without calling the backend. Cover staging, request preservation, and
error handling with regression tests.
Assisted-by: Codex:GPT-6 golangci-lint
* test(distributed): verify routed sound staging
Call sound detection through the client returned by SmartRouter.Route.
This checks interface dispatch through both routing wrappers, rather
than constructing FileStagingClient directly.
The test fails without the sound-staging override and passes with it.
Assisted-by: Codex:GPT-6 golangci-lint
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Community-maintained packagings that currently track releases —
Homebrew, ALT Sisyphus and the Gentoo local-ai overlay — with a note
that versions may lag. Placement and scope as discussed in the issue.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
* ⬆️ Update leejet/stable-diffusion.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(stablediffusion): adapt streaming options
Upstream now selects segmented weight streaming automatically and removes the stream_layers field. Keep the old LocalAI option as a no-op for existing model configurations.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update antirez/ds4
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(ds4): link upstream image helpers
The ds4 bump adds vision calls to the engine object. Link the new image preprocessing object into every backend target.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Preserve heartbeat checkpoints and backend readiness across the tunnel
transport changes. Update incoming tests for the renamed worker address
fields and health monitor arguments.
Assisted-by: Codex:gpt-6
* Update containers.md to fix podman image qualification
Signed-off-by: Alex Mazzariol <alex@alex-maz.info>
* docs(containers): clarify Podman image names
Podman can reject short image names when no registry is configured. Explain why the examples use fully qualified Docker Hub names.
Assisted-by: Codex:gpt-5.6
---------
Signed-off-by: Alex Mazzariol <alex@alex-maz.info>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The mutex round the realtime transport double added a snapshot accessor for each
recorded slice. Only the event one has a caller, so make lint refuses the build:
realtime_doubles_test.go:64:25: func (*fakeTransport).recordedAudio is unused (unused)
No spec has ever read the audio log, before the mutex or after it, so the
accessor is deleted rather than nolinted and the struct comment says where the
next one comes from. audioLog stays written, because a double that silently
discarded what a coordinator sent it would be a different double.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Probing every candidate before handing it out is right, and it made sixteen
specs that were never about the kernel depend on it. They build a supervisor
directly, name the ports they expect literally, and those literals sit inside
Linux's default ephemeral range, so with the real probe each one asks this host
whether 50051 is bindable at that instant.
The first full -race run over ./core/... and ./pkg/... after the probe landed
went red on four of them, and holding 50051, 50052, 50060 and 50061 from
another process turns eleven red deterministically. Nothing was wrong with the
allocator in either case: something else on the machine held a port, which is
the situation the probe exists to survive.
So the specs that assert bookkeeping now inject a probe that always says yes,
and say why once. The two specs that are about the probe leave the field unset
and keep asking real sockets, which is what still fails if the probe is removed.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
None was introduced by this branch and all three are in test code, which is
what made them survive: every suite passed on every run and only the race
detector said otherwise. A known-failing -race run is worse than a noisy one,
because a real race raised by production code lands in the same report and is
read as one of these.
galleryop: gatedModelManager guarded the recorded names and not the gate
channel itself. A spec frees the parked worker by closing the gate and
installing a fresh one, on the spec goroutine, while the worker goroutine reads
the field to park on it. The channel is now read and replaced under the same
mutex, and cleanup closes idempotently.
pkg/model: two specs swapped xlog's package logger to capture output and
swapped it back on cleanup. xlog.SetLogger writes an unsynchronised global, so
the restore raced with the backend process watcher, which logs while a process
is stopping; the captured bytes.Buffer was written by that goroutine and read
by an Eventually at the same time. SetLogger is now called once for the whole
test binary, from init, before a goroutine exists to race with, and a spec
swaps the DESTINATION under a mutex through a routing slog.Handler. Per-spec
level filtering is preserved deliberately: one of these specs asserts that a
debug emission is filtered OUT and would pass vacuously against a handler that
recorded everything.
openai: fakeTransport appended to its event and audio logs from the response
and turn coordinators' goroutines while a spec ranged over them. Both are
behind a mutex and are read through snapshot accessors; the fields are renamed
so a raw read from another spec file does not compile.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The claim queue's attempts counter grew without bound and nothing read it. At
the default two-second poll a permanently undispatchable row cost about 43000
UPDATEs a day, and it cost more than writes: rows are claimed oldest first, so
the oldest stuck row was re-claimed ahead of every newer one on every tick and
held a dispatch slot while it failed. One poison row starved the queue behind
it.
No dead letter, and that is the decision rather than the omission. Read
settleClaim: the only outcome that releases a claim is one where NOTHING was
learned about the work. No agent worker was connected, the tunnel broke, a peer
could not be reached, the stream was refused before the request body left this
replica. Not one of those is a worker saying it ran the job and it failed, and
an attempt ceiling would turn "the fleet was away long enough" into a job
failure nobody reported, which is the collapse this whole design exists to
prevent pointed at work instead of at nodes. The one verdict available here,
that no build of any worker serves this kind, is already settled as an answer.
So the retry stays unbounded and the RATE does not. Each release stamps the row
with the earliest it may be claimed again, doubling from two seconds to a cap
of sixty, computed in the release statement from the row's own attempts count
and stamped on the DATABASE clock, because that is the clock competing replicas
order the queue on. Queued work becomes claimable again within one cap of the
fleet returning, and a stuck row no longer holds the head of the queue. A claim
released by the reap carries no delay at all: that work was never handed to
anyone, so there is nothing to back off from.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A frontend keeps two caches of one http.Client per worker: the control
client's, built on the first verb issued to a node, and the HTTP file stager's,
built on the first file staged to it. Both are keyed by node ID and neither was
ever pruned. Their own comments said so and named what a fix would need, a
signal that a node has left, which did not exist when they were written and
does now.
The map slot is the smaller half. Each entry holds an http.Transport whose idle
connections are streams on that worker's tunnel, kept until IdleConnTimeout
even after the tunnel is gone, so ForgetNode closes them rather than leaving
them to the collector. Both are registered on the deployment's one departure
notifier, as two subscribers and not one: a node can be in either cache without
being in the other, and a single hook would say only that some client was kept.
ForgetNode is on the FileStager interface rather than on the one implementation
that has state to drop, so a stager that grows a per-node map later cannot be
added without answering the question, and so registerDepartureEvictions can
take a FileStager and still fail to compile if the registration is deleted. The
S3 stager's is a documented no-op that deliberately does not forward to the
control client, which registers itself: forwarding would evict a cache it does
not own, twice per departure, and the second drop would not appear in the
subscriber names the wiring spec reads.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
HealthMonitor.misses holds one consecutive-failed-probe count per (node,
model, replica) and nothing ever removed an entry whose row had gone. It is
the only per-node state in a frontend that grows on model churn rather than on
fleet size, so a deployment that loads and unloads models for months
accumulates an integer per tuple it ever probed and gives none back.
There are four ways a row stops being visible to the pass, not one. A node
departs and the pass skips its probes; a node goes offline or unhealthy on a
stale heartbeat and the pass skips it entirely; an operator sets a node
draining; or the row is removed by an unload, a scale-down or an eviction, and
nothing tells this monitor.
So the bound is the pass itself, and not a subscription on the departure
notifier. The notifier evicts the caches a DEPARTURE invalidates and it keeps
that one meaning; this reads a different fact, that there is no longer a row
to count misses against, and covers all four cases with one rule. A row the
pass could not probe is marked seen before the probe, so an unreachable worker
still leaves its streak exactly as it was rather than having it forgiven; a
pass that could not list the fleet prunes nothing, since it observed nothing.
Forgetting only ever delays a reap by up to the miss threshold and can never
cause one.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The backend port allocator allocated from its own bookkeeping alone. That
bookkeeping records what this worker did with a port, and the collision it
cannot see is with something this worker never did: the default base port is
50051, inside Linux's default ephemeral range of 32768 to 60999, so the kernel
hands ports in this range to outbound connections and to anything that binds
port 0. A backend handed one of those dies on bind, and the frontend sees a
backend that will not start.
Every candidate is now probed by binding the exact address the backend will
listen on, in all four allocation branches: the key's own port, the free pool,
a grown port and a stolen one. Probing the free pool matters as much as
probing a grown port, because a port this worker released is exactly as
available to the kernel as one it never used.
A candidate that fails the probe is quarantined rather than blacklisted, since
whatever holds it is usually an ephemeral connection that gives it back, and
its affinity claim is dropped so an unbindable port does not stay reserved for
the key that last held it. Exhaustion now says how many candidates were
skipped, which is what tells an operator "something else is in my range" from
"my range is too narrow".
This does not remove the race and cannot: between the probe and the child's
bind the kernel can still give the port away. It removes the far larger window
in which the allocator hands out a port the kernel gave away minutes ago,
which was the whole of the observed one-in-three harness flake. The e2e
harness comment that recorded the missing check is corrected, and the docs say
how to move the range out of the ephemeral one entirely.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Skills and RAG collections had no cross-replica invalidation, and the two
builders that would have published one were deleted earlier in this branch
because nothing called them. Wiring one now would be wrong, not merely late.
Both features are derived entirely from the frontend's own state directory.
A skills.Service indexes <state dir>/skills, a collections backend enumerates
<state dir>/collections and holds one handle per collection it found there,
and no replica reads or writes another replica's copy of either. In
distributed mode PostgreSQL carries a skill's NAME and description in
skills_metadata, and nothing else: Get, Search, Export and the resource verbs
all read local files. So a peer told to drop a cache entry would rebuild it
from a directory that does not hold the change. For a postgres-engine
collection it would be worse than a no-op, since re-deriving one on a replica
with no local index file yields a collection that answers with an empty file
list against a populated vector store. What is missing is shared storage, not
a broadcast.
Recorded rather than left silent: the two cache fields say why nothing
invalidates them, a distributed frontend logs the limitation once at startup,
and the docs name the two deployments that avoid it. The new spec pins the
premise, so a change that moved either directory onto storage every replica
mounts reddens and the decision gets taken again.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
GET /api/cluster/peer authenticated with the deployment's shared
registration token and took the dialling replica's id from ?id= on trust.
Every worker holds that token, so anything holding it could open a peer
link as any replica: relay through it to every worker tunnel that replica
owns, displace a real replica's inbound link by declaring its id, and
point the roughly 31 GiB per-session receive window at one replica.
Validating the id against the instances table does not fix this, because
the attack declares a real replica's id. So the route now checks two
credentials and needs both. The shared token still says the dialler
belongs to this deployment; a new per-replica credential says which
replica it is.
The credential follows the per-node worker credential rather than
inventing a second mechanism: crypto/rand.Text, stored only as a hex
SHA-256, compared in constant time, with no fallback to the shared token.
It differs in the stronger direction. A worker's credential is minted by
the frontend and handed over once; a replica writes its own instances
row, so it mints its own secret, publishes only the hash in the same
statement that publishes its address, and never sends the plaintext
anywhere but the peer dial.
A peer that presents no credential is refused, not waved through. An old
replica and an attacker holding the shared token send the same request,
so accepting the first accepts the second; there is no safe downgrade
here, only a quiet one. The refusal is made loud instead, on both sides,
naming the upgrade rather than the network. On the documented
frontend-first order a new replica still dials an old one; an old replica
cannot dial a new one, which costs relayed requests that land on a
not-yet-restarted replica and surfaces as no route, never as absence.
A rejected peer gets its own sentinel, ErrPeerRejected, whose unwrap
chain carries ErrPeerUnreachable as well and no absence sentinel at all.
Keeping the older sentinel means no existing consumer changes behaviour;
the cause stays out of the chain, so absence cannot escape through it and
nothing can read an authorization failure as a worker that went away.
One consequence beyond the fix: a replica with no advertised address has
no instances row, so it now cannot dial out either. It was already
unreachable inward. The startup error and the docs say so.
Registry.Register, NewMembership, NewPeerPool, PeerHandler and
RegisterClusterRoutes all gained required arguments, so the identity
cannot be dropped without a compile failure.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Removing the broker left one thing carrying every broadcast family in the
product: PostgreSQL LISTEN/NOTIFY, in core/services/pgbus. It is covered
thoroughly in process by test-e2e-distributed, and it was covered nowhere at
all by real binaries: grepping the six Cluster spec files for pgbus,
bus_messages, LISTEN and NOTIFY returned zero hits. Registration, model staging
over the tunnel and inference through both the owner and the relay paths were
already proven by real processes; the carrier that now carries everything else
was not, so a deployment whose replicas each published to themselves and heard
nobody would have left every suite green.
Two specs, both on two frontends and no workers against one PostgreSQL,
publishing at frontend 0 and reading at frontend 1.
1. A gallery operation admitted at one replica, read out of the other, with the
queued state observed before the terminal one.
2. A broadcast of about 9.3 kilobytes, which PostgreSQL refuses as a
notification payload, making the round trip byte for byte through the
bus_messages spill table.
The family is a gallery operation for one property nothing else on this carrier
has: the answer a peer gives is held in memory ALONE. GET /models/jobs/<id>
reads galleryop's statuses map, which on a peer is filled by the
gallery.*.progress subscriber and by nothing else, because the only other
filler, Hydrate, runs once at startup and every operation here is created long
afterwards. Every other family has a durable table behind it that a peer would
converge through anyway, and a spec on one of those cannot separate "the
broadcast arrived" from "the row was read".
That is then made checkable rather than argued. The gallery_operations row is
written when the gallery worker DEQUEUES an operation, so an operation still
waiting in the queue has NO row, and both specs assert zero rows while the peer
is already answering with the operation's own bytes. Both also read the
instances table and require the reading replica to be a different live instance
from the publishing one, so "the other replica" cannot decay into a spelling of
"this replica".
Holding the queue is what cluster.Options.Galleries is for. The gallery worker
runs one operation at a time on an unbuffered channel, so an install parked
inside a gated index fetch parks everything behind it; without that the
admission broadcast and the terminal one are separated by two database round
trips and no HTTP poller could see between them. The option also turns the
startup estimate warmer off, because a second fetcher filling the process-wide
index cache would leave the operation never blocking and the spec passing on an
ordering nothing enforced.
The spill spec is written against a failure this branch has shipped three
times: a size-limit spec that cannot fail. The oversized body is an ordinary
element name that the real consumer decodes and surfaces, so it is not a body
the decoder would have refused at any size. The size is ABSOLUTE at 9000 bytes
rather than derived from the cap, and a one-byte control operation in the same
run is required to leave no spill row, so moving the 8000-byte cap in either
direction reddens the spec. pgbus.FitsInline, which shares its encoder and its
comparison with Publish, is asked about both payloads and must answer
differently. The spilled row is then decoded and its element name compared byte
for byte against what frontend 1 answers.
The terminal assertion in spec 1 does not re-check the element name: a terminal
status does not carry one, because updateError in galleryop.Start builds a
fresh OpStatus holding only the error. It asserts the two fields that status
does carry, in the relation that one place writes them.
Attacks run, each alone, each reverted, each behaving as predicted. Neutering
the pg_notify in pgbus.Publish so every replica knows only what it did itself
reddens both specs at frontend 1, which answers 500 for an operation it was
never told about; the bus_messages row assertion still passes under it, which
is right, since the row is written before the notification. Releasing the queue
gate reddens spec 1 at the gallery_operations count, because the operation is
dequeued and the row appears. Shrinking the oversized name to 100 bytes reddens
spec 2 at FitsInline; inverting that guard so the run reaches the row check
reddens it there instead, with no bus_messages row written, which is what makes
the row a statement about size.
test-e2e-cluster is 26 specs in 933.8 seconds of Ginkgo time, 15m37s wall. The
two additions cost 7.0 seconds together, 5.0s and 2.0s: they start no workers,
so they pay for no registration, and what they wait on is a broadcast rather
than a threshold. test-e2e-distributed is unchanged at 223 plus 8 specs, 130.6
seconds. The budget comment and .agents/building-and-testing.md move from 24
specs at 897 to 907 seconds to 26 at 933.8.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Qwen3-style chat templates append the opening <think> tag to the *prompt*
when thinking is enabled. The model therefore never generates it and emits
only the reasoning text plus the closing </think>.
sglang's ReasoningParser keys off the opening tag:
in_reasoning = self._in_reasoning or self.think_start_token in text
if not in_reasoning:
return StreamingParseResult(normal_text=text)
so with such a template the entire completion — reasoning and answer, the
raw </think> in between — is returned as content and reasoning_content
stays empty, no matter how reasoning_parser is configured.
sglang's own OpenAI server handles this via
force_reasoning = (self.template_manager.force_reasoning
or self._get_reasoning_from_request(request))
This backend has no template manager, so derive the same signal from the
rendered prompt: if it ends with the detector's think_start_token, the tag
was prefilled and the parser is constructed with force_reasoning=True.
Structured decoding is the exception, and it matters: a grammar applies
from the first token, so the model cannot emit the closing tag even though
the template opened the block. The whole completion is schema output and
belongs in content — forcing there files it as reasoning and returns an
empty answer. Measured against a JSON-schema code audit: 10107 characters
of "reasoning", zero content. sglang's own server keeps the two apart for
the same reason; its grammar backend owns the reasoning prefix when a
reasoning parser is configured.
force_reasoning is only passed when it is meant to be True, so detector
defaults (DeepSeek-R1 already defaults to True) are untouched, and a
prompt without a prefilled tag behaves exactly as before — which matters,
because forcing unconditionally makes an answer generated with thinking
off disappear into reasoning_content.
The construction is factored into _new_reasoning_parser() so the streaming
and non-streaming paths, which previously built the parser separately,
cannot drift apart.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
With `template.use_tokenizer_template: true` the sglang and vllm backends
render the prompt themselves via `tokenizer.apply_chat_template()`, and they
hand it plain string content. A chat template only emits the model's own media
tokens when the content is a list of parts, so the rendered prompt carries no
`<|vision_start|><|image_pad|><|vision_end|>`. The pixels do reach the engine
(`image_data` / `multi_modal_data`), but both engines locate them by scanning
the prompt for that token, so they are discarded silently: HTTP 200, no
warning, and the model answers as if no image had been attached.
Add `attach_media_parts()` to the shared `python_utils` helper and call it in
both backends: the last user turn is rebuilt as
`[{"type": "image"} * n, {"type": "video"} * n, {"type": "text", ...}]` before
templating, which makes the template emit the placeholders. The pixels keep
travelling out of band exactly as before.
Text-only requests are untouched - with no media the helper returns None and
the original string-content path runs unchanged. If a template cannot iterate
content parts (a text-only model), the parts render is caught and the request
falls back to the previous string-content prompt instead of failing.
Signed-off-by: Tai An <antai12232931@outlook.com>
vLLM's engine-based reasoning parsers derive their initial state from the
chat template kwargs. Qwen3Parser:
chat_kwargs = kwargs.get("chat_template_kwargs", {}) or {}
self.thinking_enabled = chat_kwargs.get("enable_thinking", True)
Constructed as ReasoningParser(tokenizer) the flag defaults to True, so the
parser starts in the REASONING state. A completion produced with thinking
disabled contains no tags at all, and every reasoning parser shape then
reports the whole answer as reasoning:
- engine-based parsers classify it by initial state;
- BaseThinkingReasoningParser hits its documented "may not generate start
token" fallback and returns (model_output, None).
Either way `content = c if c is not None else generated_text` turns that
into a duplicate: a Qwen3 model answering "391" with thinking off comes back
as reasoning_content="391" AND content="391".
Measured against Qwen3.5-MoE on vLLM 0.28, non-streaming:
before thinking on reasoning=202 content="391"
thinking off reasoning="391" content="391" <- duplicated
after thinking on reasoning=192 content="391"
thinking off reasoning="" content="391"
Forward the kwargs the prompt was rendered with, which is what vLLM's own
OpenAI server does; parsers that do not accept the argument keep the plain
constructor.
_split_reasoning() covers the older parser shape, which has no initial state
to set. It only reclassifies when the parser exposes a start/end token pair
and neither the completion nor the prompt ever opened a reasoning block.
Truncated reasoning (block open, end token never arrived) stays reasoning,
and parsers without that token pair are left untouched.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
#11772 exempted Temperature from the zero-filter in both backend adapters,
because proto3 has no field presence and an explicit 0 is indistinguishable
from "unset". Seed has exactly the same property and is still filtered:
if proto_field != "Temperature" and value in (None, 0, 0.0, [], False, ""):
continue
A caller pinning `"seed": 0` for a reproducible run therefore gets a random
seed instead, with no error and no log line — the one case where the failure
is invisible precisely because the request looked deliberate.
Both adapters now share a named tuple of fields whose zero is meaningful, so
the next one is added in one place rather than as a second special case.
Deliberately left filtered: top_k, top_p, min_p and the penalties. Their zero
is not a value a caller means — sglang disables top_k with -1, not 0, so
forwarding 0 there would turn a default into an invalid argument.
Verified on the sglang backend (Qwen3.5-MoE, arm64): with the temperature fix
alone, two identical requests at temperature 0 are byte-identical, but pinning
seed 0 has no effect until this change.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
A quantization job that runs with no client on its progress stream stays
"queued" forever, in the API and in state.json, while the finished artifact
sits on disk. state.json was written once by StartJob, and the only code that
advanced a job afterwards lived inside the stream callback of StreamProgress,
so job state depended on somebody watching it.
The backend's progress stream cannot simply gain a second reader: each job owns
one queue.Queue and QuantizationProgress pops from it, so two consumers split
the updates rather than both seeing them. The stream has to be opened exactly
once per job.
StartJob now starts watchProgress on the application context (the request
context is done as soon as the handler returns). That goroutine is the single
reader: it applies each update to the job -- in the cross-replica store and in
state.json, terminal statuses still winning over late updates -- and republishes
it in-process. StreamProgress becomes a pure reader over that fan-out and no
longer loads a backend or opens a stream. A client attaching to a job that has
already finished, including one hydrated from disk after a restart, gets a final
event built from the stored job instead of blocking.
Two paths used to end a client's stream by breaking the gRPC connection and now
release it explicitly: StopJob kills the backend, so it publishes the stopped
event itself; and a stream that ends without a terminal update means the backend
is gone, so the job is recorded as failed rather than left running forever.
Signed-off-by: Tai An <antai12232931@outlook.com>
The Intel backend installs PyTorch XPU wheels, but Qwen ASR only
checked CUDA and MPS. Every Intel model therefore loaded on the CPU.
Select XPU when available and place the model on xpu:0. Keep the
existing CUDA, MPS, and CPU placement behavior.
Assisted-by: Codex:GPT-5 [apply_patch] [gh]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
WhisperX silently returned a plain transcript when diarization lacked
the Hugging Face token required to load pyannote. Reject that request
clearly so callers do not mistake missing speaker labels for a
successful diarization.
Convert WhisperX seconds to the nanosecond duration unit used by the
transcription API.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
importNpmLock turns the same-version hono override into a file: tarball
that conflicts with the direct dependency (EOVERRIDE). Pass
--legacy-peer-deps so the flake build can proceed without
touching package.json (open #11633).
Fixes#11804
Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
LLM-jp 4 provides a recent Japanese and English reasoning model on the supported Llama architecture. Add its official Q4 and BF16 GGUF builds so hosts can select the fidelity that fits.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add Q4_K_XL and Q8_K_XL llama.cpp builds with the shared vision projector. Enable the preserved MTP head for speculative decoding.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the text-only Q4_K_M build for private red-team, blue-team, and security operations workloads. Configure the supported Gemma 4 model for llama.cpp with its verified Hugging Face checksum.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add Q4_K_M and Q8_0 llama.cpp builds plus the official F16 GGUF for Mixedbread mxbai-embed-large-v1.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The new importance-matrix builds reduce Laguna S 2.1 from the existing 96 GB default to 73.9 GB or 54.4 GB. Add both as selectable llama.cpp variants.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
DFM Mimir is a new permissively licensed Danish and English HRM-Text model. LocalAI already ships a vLLM version with native support for its architecture.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the artifact-backed VoxCPM2 model for multilingual speech synthesis, voice design, and controllable voice cloning.
Assisted-by: Codex:gpt-5.6 [Codex]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the Q4_K_M and Q8_0 GGUF builds for a compact Qwen3.5-based reasoning and tool-use distillation. The variant pair lets LocalAI choose higher fidelity when the host has enough memory while keeping the Q4 build as the installable fallback.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a compact NVFP4 build with its embedded MTP head and vision
projector. This gives Blackwell-capable llama.cpp hosts a speculative
option alongside the existing Q4 and Q8 builds.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the original-dtype GGUF for LocalAI\x27s audio.cpp backend. The entry exposes multilingual voice cloning and records the model license.\n\nAssisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the compact LightOnOCR model to the OCR section with Q8_0 and F16 builds. Each build uses its matching vision projector.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the official F16 GGUF build of the compact Apache-2.0 Qwen3-family chat model with its native 2K context limit.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
- api-errors.md documented LOCALAI_SUBTLEKEY_COMPARISON (missing the
KEY underscore); the code defines LOCALAI_SUBTLE_KEY_COMPARISON, so
the documented variable silently did nothing
- cli-reference.md documented a --csrf flag / $LOCALAI_CSRF env that
do not exist, with inverted semantics; the actual flag is
--disable-csrf (LOCALAI_DISABLE_CSRF), 'Disable CSRF middleware
(enabled by default)'
- the PR template's 'Signed commits' anchor pointed at a CONTRIBUTING
section that does not exist; repointed at the Commit messages
section
- the longcat-video backend README linked a docs page that was never
committed; replaced the dead link with plain text
- formal-verification/README.md used ../../../ for five in-repo
packages (escaping the repo root); fixed to ../
Tasks 1 to 17 are proven by unit and integration specs and by two e2e passes
taken mid-flight. This is the pass that boots the real binaries with every
carrier in place and none of the old one, and it does so on the topology the
feature was built for rather than on the one-worker shape the rest of the
cluster suite uses.
Two frontends and two workers is the configuration that matters. With each
worker's tunnel landing on a different replica, the owner path and the relay
path are live at the same instant against one roster, one scheduler and one
health monitor, so a routing mistake has somewhere to show up instead of
hiding. It is also the only shape in which "killing a replica re-homes only ITS
worker" can be stated at all.
Three scenarios, all 2x2:
1. Both workers served from both replicas. No broker as a property of the
ARTIFACT (debug/buildinfo reports no github.com/nats-io module, with the
module count asserted non-zero so a stripped binary cannot pass vacuously),
no broker in either worker's live /proc environment, and no advertised
address on either worker. One completion over the owner path and one over
the relay, plus the mirror image through the other replica, plus four
control-plane listings covering both paths for both workers.
2. The replica owning worker 0's tunnel is killed with that tunnel blocked.
Leg 1 asserts nothing and only waits for the killed instance to leave the
live set, because before that it still reads as a live owner and the
scenario is not yet about absence. Leg 2 then holds a window inside the
reconnect grace requiring that nothing acted on the absence. Leg 3 requires
the re-home and inference again. Worker 1 keeps serving throughout.
3. The suite-wide negative control. Both tunnel dials refused while
registration and heartbeats flow, both workers refused at both replicas
naming the routing fact and not a departure, nothing reaped and both
heartbeats fresh. Then ONE tunnel is restored and exactly one worker
recovers while the other stays refused.
Which worker served is read back from node_models rather than assumed: the two
models are pinned to one worker each through PUT /api/nodes/:id/labels and
POST /api/nodes/scheduling, and every assertion requires the model to be on the
expected node AND absent from the other. That the relay hit a non-owner is read
from the production Owner query before the request and re-read after it.
Attacks run, each alone, each reverted, each behaving as predicted:
hand a worker a broker URL reddens scenario 1's environment leg; point the
module check at gorm.io reddens its artifact leg; start one worker instead of
two reddens all three at the topology guard; delete the relay in
WorkerDialer.Dial reddens scenario 1 on exactly the request sent to the
non-owner while 2 and 3 stay green; a one-nanosecond reconnect grace reddens
scenario 2's leg 2 on the demotion while 1 and 3 stay green; lifting both
blocks at scenario 3's differential reddens its "still unreachable" half. The
brief's "restore the NatsURL validation" attack cannot be applied:
DistributedConfig has no such field left to validate.
Label-orphan arithmetic, counting non-skipped It nodes from --dry-run:
all 256, dist 231, cluster 24, vllm 1, and 231 + 24 + 1 = 256, so no spec is
orphaned by the label filters.
Three test-e2e-cluster runs: 897.0s, 897.8s and 906.6s of Ginkgo time, 24 specs,
15 minutes wall. The only failure across the three was a pre-existing spec dying
at cluster.Start with frontend-1 exiting status 2, which passed in the other two
and is reported as a port-allocation flake rather than a regression.
test-e2e-distributed is 223 plus 8 specs in 131.7s. The budget comment and
.agents/building-and-testing.md move from 21 specs at 800 to 830 seconds to 24
specs at 897 to 907.
The harness gains ProcessEnviron, which reads /proc for any of the three process
families; WorkerEnviron and FrontendEnviron become wrappers rather than being
deleted, so the specs that call them are not re-aimed for a rename.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
- middleware.md: the 'default detector' link used #instance-wide-defaults;
the heading is 'Instance-wide default detector'
- the advanced/reference landing pages linked an ../installation/
directory that does not exist in docs/content; dropped the dead
bullets (deployment content lives under getting-started)
* docs(llama-cpp): clarify multimodal speculative decoding
Update the speculative decoding guidance now that modern llama.cpp backends can combine mmproj-based vision with speculative decoding, including MTP. Document compatibility checks, draft acceptance statistics, VRAM tradeoffs, and a combined configuration example.
Assisted-by: Codex:GPT-5.6-Sol [gh] [OpenStack] [Docker]
Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com>
* docs(llama-cpp): clarify multimodal MTP references
Distinguish the upstream change that removed the general multimodal speculative restriction from the later change that added MTP with explicit vision compatibility.
Assisted-by: Codex:GPT-5.6-Sol [gh] [Docker]
Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com>
---------
Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com>
Distributed mode has not dialled a message broker since the control plane
moved onto the workers' own outward tunnels and every fan-out family moved
onto PostgreSQL LISTEN/NOTIFY. What was left was the dependency itself, and
the code that existed only to feed it.
Dropped from go.mod: nats-io/jwt/v2, nats-io/nats.go, nats-io/nkeys,
nats-io/nuid and testcontainers-go/modules/nats, along with the fourteen
indirect requires that only the NATS testcontainer pulled in. go.sum carries
no nats line either, so the removal is not the partial kind where the require
goes and the checksum stays.
Deleted with them: pkg/natsauth in full, the broker client's remaining
options and TLS files, the per-node JWT minting on both the register and the
approve path, and the natsauth.Config parameter threaded through the node
routes. The credential manager is renamed and stripped rather than deleted,
because it still holds the tunnel token that every re-registration rotates.
The bus flags stay accepted and ignored, and are now hidden, on every command
that had them, so an existing unit file, compose file or Helm values file
still starts on the day of the upgrade. What is not kept is the validation
that REQUIRED one: a distributed frontend started with no bus URL is no
longer fatal. The TLS paths lose type:"existingfile" deliberately, so a
certificate deleted along with the broker cannot fail a startup.
One operator-visible behaviour change: --nats-require-auth no longer makes an
agent worker wait through admin approval. Ask for that wait with
--distributed-require-auth, which already implied it. It is documented in the
migration section and pinned from both sides.
A deployment now needs PostgreSQL and the frontends' own HTTP listener, and
nothing else.
coverage-baseline.txt moves from 54.2 to 62.0.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Every carrier had already moved and no process opened a bus connection, but
the surface an operator reads still described a deployment with a broker in
it: a compose service, a 220-line credential-generation script, two CI steps
pulling a container nothing started, two flag tables offering --nats-url, an
architecture diagram with a NATS box wired to the workers, a join-command
generator in the Nodes page that emitted --nats-url for agent workers, and a
test suite that stood a NATS server up for specs that no longer used it.
That is the one way this programme could still fail invisibly. Every test
passes, every binary works, and every production deployment goes on running
and paying for infrastructure that carries nothing.
Nothing in this repository starts a NATS server any more. The compose file is
four services, the docs say to shut the broker down and what to keep, and the
e2e suite runs on one PostgreSQL container.
The three LOCALAI_NATS_*_TIMEOUT env vars are KEPT, and are now documented
twice as being kept. They were never broker settings: each names a control-RPC
budget the frontend applies to a worker, still read and still enforced. They
carry the prefix only because they arrived with the bus, and renaming them
would break every existing deployment for cosmetics.
The agent worker's join command was the last surface still emitting the flag,
two tasks after the agent worker stopped dialling. The Playwright spec that
covered it asserted the opposite of what is now true, so it is inverted rather
than deleted, and it reads the rendered command string rather than the
component's variables: the variables are what the fix removes, so a spec
reading them would have stopped compiling instead of failing, and a compile
error is not evidence about what an operator is shown.
nats_jwt_test.go and its helpers are deleted. They pinned a real server
ENFORCING the minted permissions. The CONTENT of those allow lists is still
pinned, untouched, by pkg/natsauth's own suites, including the spec that
refuses to let the agent lists go empty, since an empty allow list in NATS
means unrestricted. The enforcement half is retired rather than moved:
enforcement is a property of a connection, and nothing opens one.
The suite's own NATS container goes with them, which the brief left for the
next task. Removing the pre-pull while BeforeSuite still ran the image would
have defeated the step rather than cleaned it up, and this change removes the
last reader of TestInfra.NC. agent_native_executor_test.go and
mcp_ci_job_test.go are moved onto infra.Bus() instead of deleted: they were
the last two specs building a bridge and a dispatcher on a client nobody uses,
which is exactly the drift TestInfra.Bus's own comment warns about.
cluster.Options.NatsURL is now fed a deliberately dead address rather than a
live container's. Frontends and agent workers still receive LOCALAI_NATS_URL,
because that is the coverage for the promise that an existing command line
still starts; sourcing it from a running server would have let a regression
that actually dialled it pass. The control in cluster_control_test.go keeps
its assertion and loses its explanation, which claimed the deployment had a
bus and no longer could.
One latent spec race surfaced and is fixed: the background-run spec waited for
a COUNT of events and then read a snapshot for the terminal status, which is
the last event of a run and therefore always arrives after the count is met.
Its immediate twin had already been fixed this way. Nothing in production
changed.
pkg/natsauth keeps its files. It is reachable from production only through the
natsauth.Config parameter thread, and that thread is the next task's.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Task 4 gave agent workers tunnels and deliberately left the NodeType skip in
HealthMonitor.tunnelDeparted, with a spec asserting that an agent node whose
presence reader answers PresenceGone is NOT marked unhealthy. That spec was
scaffolding. It was true while an agent worker took its jobs and its verbs over
the message bus: a departure row for one said nothing about whether it could
work, and an early bug in the new tunnel client could otherwise have demoted a
fleet of healthy agent workers.
There is no bus. An agent worker is reachable through its tunnel and through
nothing else, so a departed agent tunnel means exactly what a departed backend
tunnel means: no live replica holds it, the departure has outlived the reconnect
grace, and that is a routing fact the scheduler and a reaper may act on. The
skip would now hide the only symptom an unreachable agent worker has. This is
the deliberate removal Task 4's M6 predicted, and task-4-report.md is where that
mutation already stands recorded red against the spec this commit deletes.
The skip existed at ONE site. router_liveness.go has none: its candidates come
from queries that already filter node_type = 'backend'. The two skips in
managers_distributed.go stay, because an agent worker still runs no backend
processes, so it has no backend to list and no backend op to apply.
Two node types can depart now, which is why the second half exists. Before this,
one type could depart and every per-node cache a departure left stale was
dropped from wherever its owner happened to notice, so a reader could not tell
which caches a demotion invalidated by reading the demotion path. Departure gets
ONE notification point. DepartureNotifier is edge triggered, because the monitor
runs on a ticker and a departed node stays departed; its subscribers are NAMED,
because what has to be caught is a forgotten cache and a count can say only that
one of four is missing; and NewHealthMonitor takes it as a required positional
argument, so a caller that does not pass one fails to compile.
Four caches subscribe: prefix-cache affinity in every model, probe freshness at
every address, in-flight staging operations, and the per-node breakdown of every
open gallery operation. The prefix-cache one is registered only when
prefix-cache routing is enabled, so --distributed-prefix-cache=false stays a
true no-op. The notification carries the node's name as well as its id, because
the staging tracker keys on the name and the other two key on the id, and a
subscriber should not have to read the registry from inside an eviction hook.
A departure notification is an act on absence, so it fires only on the routing
fact. A tunnel lost inside the grace, a worker that never dialled, a presence
query that failed and a stale heartbeat all announce nothing, asserted per node
type. The stale-heartbeat branch is excluded on purpose: it already marks the
node offline, which deletes its rows and runs the registry's replica-removed
hooks, so firing there too would double-evict and make the notification mean two
different things at its subscribers.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
agent.<name>.cancel was the last family on a message bus, and the only
reason an agent worker dialled one. Its subscriber is the worker running
the execution, and a worker has no database, so the family could not move
to the PostgreSQL fan-out carrier: a cancel published there would reach no
worker while reporting that it had been sent.
It is a control verb now. An agent worker mounts workerctl.PathAgentCancel
on the loopback control plane behind its tunnel and applies the cancel to
the same registry the executor registers a run on. The frontend issues it
through nodes.AgentControlClient.CancelAgentRun.
That call is a FAN-OUT and not a pick, because nothing records which worker
holds a given execution: the claim row names the claiming replica, and it
is deleted when the run ends. Every agent worker a live replica can reach
is asked over its own tunnel, relayed by the peer mesh when a peer holds
it, and each worker answers only for itself.
The answers stay apart, which is why this family was held back. A cancel a
worker made is nil. A cancel some worker could not be asked is
ErrAgentCancelUndelivered, which is neither a refusal nor a missing run. A
cancel every reachable worker declined to own is ErrAgentRunNotOnAnyWorker.
A deployment with no agent worker is ErrNoAgentWorker. Neither new sentinel
wraps ErrWorkerUnroutable and neither is a worker answer, so nothing is
reaped, demoted or evicted because of a cancel.
A worker in the ABSENT CONNECTION condition, one whose tunnel was lost
inside the reconnect grace, counts as undelivered. It is not retried in the
call and not queued: a retry would spend a budget the caller did not
choose, and a queue would need durable state whose only consumer is a run
whose control stream went with the tunnel. A worker whose departure has
outlived the grace is the one routing fact a caller may act on and is
excluded, or a single retired agent node would make every cancel
undelivered for ever.
The fan-out reads a different node set from the pick. A draining worker
takes no new work but is still finishing what it holds, so it is offered
the cancel; a pending one is refused by the tunnel route on every dial and
is not.
With that, nothing in LocalAI connects to NATS. The agent worker's dial,
its credential ladder and its refresh loop are gone, and so is the
frontend's cancel carrier. LOCALAI_NATS_URL is accepted and ignored
everywhere, and distributed mode no longer requires it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Nothing in the tree publishes, subscribes, queue-subscribes or requests through
the MessagingClient interface any more, so it is deleted rather than shrunk to
Broadcaster: two exported names for one method set in one package is an
invitation for the next author to pick whichever the surrounding file already
imported.
$ grep -rn 'messaging\.MessagingClient' --include='*.go' .
core/services/syncstate/syncstate.go:54: // It is messaging.Broadcaster rather than messaging.MessagingClient because
(one hit, a comment; no live referent. The naive grep in the plan also matches
prose and the local test type names fakeMessagingClient and
countingMessagingClient, so it can never be empty.)
*messaging.Client is shrunk to exactly Broadcaster plus its own lifecycle.
QueueSubscribe, QueueSubscribeReply, SubscribeReply, Request, Conn and the
package helpers QueueSubscribeJSON and RequestJSON go with it; none had a
production caller. Deleting the methods rather than only the call sites is what
makes putting a family back on this carrier a build error instead of a line that
compiles, publishes successfully, and is delivered onto a carrier the deployment
is being taken off. Conn is in that list because while it existed every other
name was one c.Conn().X() away; the flush-and-verdict that its real consumers
needed is now ConfirmRoundTrip, which keeps the NATS JWT permission specs armed.
The client, its options and its TLS plumbing are NOT deleted, and both processes
stay on the bus. agent.<name>.cancel is the one fan-out family that could not
move: its only subscriber is the agent worker, which has no database and cannot
join the PostgreSQL carrier at all, so a cancel published there would reach no
worker and be reported as sent. The frontend passes the client to
newFanoutBridges as its cancelCarrier and the worker subscribes on it, so
--nats-url stays required on agent-worker. Both go with the tunnel cancel verb.
The struct field is renamed Nats -> CancelCarrier to say what it is for, and
agentpool loses the messaging.Publisher it held only to be non-nil: it never
published on it, and it was gating whether a frontend runs agents distributed or
in an in-process pool. Retiring the bus would have flipped every replica back to
the in-process pool silently. The gate now reads the agent store, which is the
dependency the mode actually requires.
Also deletes four subject builders with no production publisher
(SubjectFineTuneProgress, SubjectFineTuneCancel, SubjectCacheInvalidateSkills,
SubjectCacheInvalidateCollection), the queue and request/reply halves of the
shared test double, and the e2e specs that were their only callers. Every
surviving subject is now pinned to its exact literal, because a subject is a
cross-version wire format and a rename that looks internal stops half a fleet
hearing the other half.
Docs: distributed-mode.md and cli-reference.md no longer claim NATS carries the
agent-worker job subjects, the frontend's cross-replica events, or an agent
worker's real work.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Gallery progress and cancel, the operation cache's start and end, the model
and backend cache invalidations, staging progress, and the prefix cache's
observations and invalidations now travel on the LISTEN/NOTIFY carrier. No
subject is published or subscribed on messaging.Client anywhere in the tree,
which is what makes retiring that package a deletion rather than a migration:
$ grep -rn 'natsClient\.Publish\|nats\.Publish\|\.Nats\.Publish\|QueueSubscribe\|SubscribeReply\|\.Request(' \
--include='*.go' core/ pkg/ | grep -v _test \
| grep -v 'c\.Request()\|ctx\.Request()\|Request()\.Context' \
| grep -v 'core/services/testutil/fakebus.go'
core/services/messaging/client.go:168,170,172,227,234,236,250,252,254,268,269,287
core/services/messaging/interfaces.go:21,22,23
Every remaining hit is inside core/services/messaging itself. The production
reads of the NATS client are now three, all of them the documented agent-worker
exception: Close on shutdown, the agent pool's publisher, and the agent-cancel
carrier passed to newFanoutBridges.
Prefix-cache observations publish like every other family rather than through a
method that refuses a message too large for a notification. The plan proposed
such a refusal on the reasoning that a long prompt makes a chain of thousands of
entries; ExtractChain caps a chain at Config.MaxDepth blocks, MaxDepth is a
constant with no operator knob, and the chain reaching Sync.Observe has one
source, the router's own extraction hook. A worst-case observation is a few
kilobytes against an 8000-byte cap, so the hot-path spill the refusal was
designed to avoid cannot occur, and shipping it would have added the programme's
only deliberate message drop to guard a condition that cannot arise. pgbus gains
FitsInline instead, a predicate that shares one size decision with Publish and
decides nothing, and core/application refuses at startup to wire a prefix cache
whose configured depth would put every observation over the cap.
The carrier choice is no longer stated at four sites. StagingTracker.SetPublisher
and SubscribeBroadcasts become one SetBroadcaster, so a tracker that publishes
where its peers are not listening cannot be spelled; prefixcache.Sync gains
SubscribeBroadcasts, which reads the carrier it publishes on; and the gallery
service and the operation cache are wired by methods on DistributedServices that
name no carrier at all, so the NATS client beside it cannot be handed over.
OpCache.SetMessagingClient and GalleryService.SetNATSClient are renamed to
SetBroadcaster so a missed call site fails to compile.
Two pre-existing defects that the two-real-carrier specs surfaced are fixed. A
progress tick published before a cancel and delivered after it cleared Cancelled
and left the operation reading as still running on that replica; mergeStatus now
drops a stale tick rather than merging it. GetStatus and GetAllStatus handed out
the stored OpStatus pointer while the broadcast subscribers mutated it in place,
so an /api/operations response could be marshalled mid-write; both now copy.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Five of the six families whose subscriber is an open HTTP response rather
than a process-lifetime cache now travel on pgbus: jobs.<id>.progress,
jobs.<id>.result, jobs.<id>.cancel, agent.<name>.events.<user> and
responses.<id>.cancel. Both ends of each move together, so there is no
state where a publisher is on one carrier and its subscriber on the other.
agent.<name>.cancel does NOT move, and the plan was wrong about why. Its
only subscriber in the tree is the agent worker, which has no database and
so cannot join the PostgreSQL carrier at all. Publishing that cancel on
pgbus would have lost every cancel of a worker-run agent while returning
nil, which reports a cancel that reached nobody as a cancel that was sent.
EventBridge now names its cancel carrier separately, a frontend replica
sets it to the carrier the worker reads, and it stays there until a cancel
rides the worker's tunnel like every other verb addressed to a worker.
The carrier drops at 256 rather than blocking, which is not safe on its own
for a result: a lost result has no successor message. It is not the only
path. The claiming replica persists the terminal line before it releases
the claim, and an open progress stream re-reads the job row once after
subscribing and then periodically, so a dropped terminal broadcast costs
promptness and never the answer.
Both per-request subscriptions close in a defer instead of on one return
path, and pgbus grows Subscribers() so the leak they would otherwise cause
can be asserted. It has no other symptom: only the first subscriber of a
channel issues a LISTEN, so a leaked filter just adds one closure per
notification for every stream the replica has ever served. Subscribe now
issues its LISTEN before it registers, which makes that count a readiness
signal rather than a figure to compare against itself.
Two rules that were stated at several sites and pinned at none are now one
each. The re-broadcaster is built beside the dispatcher and the bridge and
handed to the dispatch loop, so no line is left that can point it at a
carrier nobody subscribes to while every spec stays green. The set of
statuses a job never leaves is one exported set that the SSE bridge and the
store both read. The last hand-written subject filter in production code
became messaging.SubjectAgentEventsWildcard.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
syncstate.Config held one carrier field typed as the NATS client, so a
pgbus.Bus could not be handed to a SyncedMap at all: it satisfies
messaging.Broadcaster and not MessagingClient. The durable re-hydration
path built for the responses map therefore had a NATS-only consumer and
nothing in the build said so.
The field becomes Bus messaging.Broadcaster, SubscribeJSON moves to its
own file and relaxes its parameter to Broadcaster, and the four adopters
fan out over PostgreSQL LISTEN/NOTIFY: fine-tune jobs, quantization jobs,
agent tasks with their per-tenant children, and Open Responses metadata.
A new spec proves it on a real database, over two Bus instances on two
pinned listener connections: a Set and a Delete carry, a payload past the
8000-byte notification cap comes back byte identical through the spill
row, two families sharing one LISTEN channel stay separate, and a
terminated listener re-hydrates a row written while it was gone.
The five sites that each chose a carrier for an adopter are collapsed
into one DistributedServices.Broadcast() accessor. Five field reads were
five chances to leave one family on NATS with nothing failing, because
messaging.Client satisfies Broadcaster too. The accessor also refuses to
hand out a nil pgbus.Bus wrapped in a non-nil interface, which every
adopter would read as "broadcast" and dereference on the first Set.
SetTaskSyncNATS and SetJobSyncNATS are renamed to SetTaskSyncBus and
SetJobSyncBus so a missed wiring site fails to compile.
The response metadata table gains a retention of its own, defaulting to
24 hours. It inherited the Open Responses store TTL, which defaults to 0
meaning no expiration. Zero is defensible for a map that dies with the
process and is not for a table: the table grew for the life of the
deployment and a restarting replica re-hydrated every response the
cluster had ever created. A row that names its own expiry is still judged
on that column alone, and "this row is dead" now has one SQL spelling
that PurgeExpired deletes by and ListUnexpired is the negation of, so a
hydrate cannot resurrect what a sweep has already retired.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Every AgentJobService built its tasks SyncedMap with the name "agent.tasks",
and there is one service per user. So every tenant published on and subscribed
to the same subject, state.agent-tasks.delta, and SyncedMap.apply scopes
nothing: a task tenant A created was written into tenant B's in-memory map on
every replica, and ListTasks reads that map. Nothing repaired it short of a
process restart.
The subject now carries the tenant in a token of its own,
state.<name>.<tenant>.delta. Four tokens where the unscoped builder makes
three, deliberately: SubjectMatches compares token count before anything else,
so a tenant's subject and the cluster-wide one cannot cross-match, and neither
can two tenants. Putting the tenant inside the name token would not do that,
because the sanitizer folds '.' to '-' and the only filter that could then span
tenants is state.*.delta, which spans every other family too.
The rule is stated once. subscribeFilters calls publishSubject rather than
restating the subject, so a map cannot end up publishing scoped and subscribing
unscoped, which would leak exactly as before while every publish assertion
passed. The one case that decides on its own is the cluster-wide administrative
view: it hydrates from every tenant's rows, so it also takes the per-tenant
wildcard, or it would be stale the moment any tenant wrote. A tenant hydrates
from its own rows and applies only its own deltas. PerTenant defaults to false,
so finetune, quantization and the responses store keep the subject they have.
The second half of the same defect was the delete. taskStoreAdapter.Delete
called DeleteTask(id) and JobStore deleted by primary key with no user
predicate, reachable from DELETE /api/agent/tasks/:id, which takes the id off
the URL. A tenant who learned another tenant's task id destroyed that tenant's
row. The user id now travels with the id and lands as a user_id predicate.
Empty stays the administrative any-owner scope, the same thing an empty id
already means for ListTasks and ListJobs. A foreign delete removes nothing and
returns no error: not yours and not there are the same answer to the caller,
and neither is a store failure.
SetUserID rebuilds the tasks map for the same reason SetTaskSyncNATS does.
GetJobs happens to set the user id first, nothing enforced it, and with the
order reversed the map would be built with an empty tenant and put that user's
tasks back on the cluster-wide subject.
Both halves predate this programme; they are surfaced here rather than caused.
Neither is fully closed for a deployment with the agent pool off, where the
task routes are still served by one cluster-wide service that every
authenticated caller shares; that is a separate gap and it is documented.
testutil.FakeBus grew a real defect this was the first change to trip:
Unsubscribe matched on the filter string, so with two subscribers on one
filter, closing one deafened the other. Subscriptions now carry an id.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The responses.metadata SyncedMap had no durable Store, so its reconnect
re-hydrate replaced nothing. That was survivable while responses converged
through deltas on a broker that mostly stayed up. It is not survivable on a
carrier whose listener is one pinned PostgreSQL session: every response created
while the subscription was down stays invisible on that replica forever, and the
symptom is a 404 from one replica and a 200 from another for the same
response_id.
State that must survive a gap now lives in a response_metadata table, and the
notification only says it changed. The map writes through on a Set and reads the
table on hydrate, on reconnect and on reconcile, so the gap closes instead of
becoming permanent.
The row carries the whole projection as JSON rather than one column per field. A
column-per-field schema would be a second definition of what a peer may act on,
and the two would drift the first time syncedResponse gained a field: the map
would broadcast the new field and hydrate without it, so a replica that had
reconnected would serve a different response body from one that had not, with
nothing failing anywhere. Only PayloadJSON is ever decoded; owner_replica and
owner are indexed copies for an operator reading the table by hand.
A missing row and an unreachable database are different facts. Every store and
adapter method returns a driver failure as an error and never as an empty
result, and syncstate replaces nothing when its source errors, so an outage
leaves the map holding what it had rather than blanking it into a cluster-wide
404.
Liveness is the database's clock, spelled expires_at IS NULL OR expires_at >
now(), because every replica hydrating from this table must agree on which rows
are live and a Go-side cutoff makes that a property of whichever process asked.
The test container shares the host clock, so no behavioural spec can tell the
two apart; the statement shape is pinned instead. The constructor refuses a
non-PostgreSQL handle, because an unguarded now() on the single-binary path
reads as a missing migration.
A ticker sweeps expired rows every five minutes on each replica, and Close waits
for it rather than racing it. Note that the sweep removes nothing while
LOCALAI_OPEN_RESPONSES_STORE_TTL is 0, which is the default: with no TTL nothing
ever expires and the table grows for the life of the deployment. The docs say so
plainly.
EnableDistributed takes the store positionally and last, so a call site that
forgets it fails to compile rather than silently restoring the deltas-only map
this change exists to replace. A nil store there is refused by name: it is
reached only from the distributed branch of route registration, so it is a
wiring bug and not a deployment shape.
What still never leaves the owning replica is unchanged: the resume buffer and
the CancelFunc. The write-through is one row per response state change, not one
per generated token.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The AI-assistant policy says an AI agent must never add a Signed-off-by
trailer, because only a human can certify the DCO. That is right for the
case it was written for: an assistant helping a contributor who then
signs off themselves.
It does not fit automation a maintainer runs. Those pull requests have no
human submitter, so nothing ever signs and the DCO check blocks them
permanently. Sixty-one open pull requests from the maintenance bot are in
exactly that state, every one of them correctly following the documented
rule.
Carve out the case: automation a maintainer operates signs off with that
maintainer's identity. The maintainer certifies the DCO, as they do for a
commit they typed by hand, because they configured the automation, own
its output, and take responsibility on merge. The Assisted-by trailer
still records that a model wrote the code, so provenance is unchanged.
Keep the exception narrow. An assistant helping an outside contributor
still must not sign off, and a bot must not sign for anyone but its
operator, including on a contributor's branch it pushes to.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The three NATS queue groups jobs.new, jobs.mcp-ci.new and agent.execute are
gone. Dispatching work is now a row in a work_claims table, taken by one
frontend replica with SELECT ... FOR UPDATE SKIP LOCKED and driven on an agent
worker as a streaming control RPC over that worker's tunnel. Exactly-one
delivery among competing consumers is a database problem, not a broker feature.
An agent worker has no database, so it never claims; it executes what the
claiming replica hands it.
A claim must not outlive the replica that took it. The reap releases a claim
whose owner is no longer a live replica in the instances table, on the database
clock, and never asks how long the claim has been held. A job that legitimately
runs for an hour on a heartbeating replica is left alone, while a claim whose
owner stopped heartbeating becomes claimable again within one liveness window.
A replica with no advertised address has no instances row at all, so it refuses
to claim rather than have its work reaped out from under it mid-run.
The settle rule is stated once, in settleClaim, and every exit path calls it. A
transport failure releases the claim and never completes or discards it; only a
decoded reply line completes it. That line is deliberately not
cluster.IsWorkerAnswer, which accepts the stream refusals a worker's tunnel
writes before any request body reaches its control server: completing on those
would discard work that never ran.
The terminal line is persisted before the claim is completed, so a store that
refuses leaves the claim standing rather than leaving the job running for ever.
That is the dropped-result defect fixed structurally rather than by retry.
This also surfaces a pre-existing gap rather than causing one: no worker has
ever served plain task jobs, and publishing them into an empty queue group left
them running with no trace. Such a claim is now failed with a reason.
Removes QueueWorkers, --agent-subject and --agent-queue, and narrows an agent
worker's minted JWT by agent.execute and jobs.mcp-ci.new.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A worker has no database and cannot NOTIFY, and it does not need to: every
message it sends is produced inside a handler the frontend invoked, so
there is always an open control response to write on. This adds the two
ends of that, and the authorization decision that sits between them, and
nothing that dispatches yet.
workerctl.Envelope gains Subject, a REQUEST and not an instruction. Empty
means the line is for this caller alone, which is what every pre-existing
progress line is, and omitempty keeps those lines byte-identical for an
older reader. It qualifies a progress line and never a reply line: a reply
is the worker's verdict about the work, and there is no version of
"publish my verdict for me" this control plane has to carry.
nodes.MayBroadcast is the replacement for pkg/natsauth's allow list, and
the inversion is the point. NATS read an EMPTY allow list as NO
RESTRICTION, which is why phase 3 refused to delete the backend branch and
spelled it {"_INBOX.>"}. This one reads an empty list, and an absent node
type, as DENY EVERYTHING, and a table-driven spec pins that by emptying
the agent entry and asserting all three of its subjects are then refused.
Matching goes through messaging.SubjectMatches, the one definition in the
tree, so a filter that fires here fires on the carrier.
nodes.Rebroadcaster.Handle returns a bool and never an error. A refused or
failed re-broadcast is logged and the RPC continues, because the RPC's
outcome is the worker's verdict about the work and a publish failure says
nothing about it. The return shape is asserted at compile time in the file
that states the rule, so changing it to an error does not compile.
ControlClient.CallStreaming's progress callback now takes the line's
subject alongside its raw bytes. The client no longer decodes progress at
all: what a line means is a question about the verb and whether a named
broadcast may be made is a question about the node, and it knows neither.
Both moved into installProgressBridge, which every one of the three
streaming call sites in unloader.go goes through. A line naming a subject
is dropped there rather than delivered as install progress, because
backend.install and backend.upgrade are a backend worker's verbs and a
backend worker is allowed no subjects.
agents.StreamPublisher is the producing end, a messaging.Publisher writing
NDJSON envelopes onto an in-flight control response and flushing each one,
so a tick reaches the frontend while the handler is still running.
Serialized, because two concurrent encodes on one http.ResponseWriter
interleave bytes and tear the framing.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Two things made every end-to-end result on this branch unreliable.
make test-e2e-cluster did not rebuild ./local-ai. It only checked that a
file by that name existed, so an edit that was never rebuilt ran the whole
suite against an older build while reporting on the working tree, and a
missing binary skipped every spec and exited 0 with "Test Suite Passed".
The target now depends on a new e2e-binary target, which is the plain go
build CI already used rather than make build, since that one also builds
the React UI this suite never touches. The harness carries the other half:
localAIBinary now FAILS, locally as well as under CI, when the binary is
older than the newest non-test Go source in the tree, which covers a run
started with LOCALAI_E2E_BINARY or by invoking ginkgo directly. Test files
are excluded from that scan because they compile into the ginkgo suite and
never into local-ai. The CI job drops its own build step and the env var so
that one place owns the build and it happens after protogen-go.
test-e2e-distributed was audited for the same hole and has none: ginkgo
compiles that suite from the working tree on every run and it execs no
prebuilt binary. build-mock-backend already rebuilt unconditionally.
"Worker tunnel under load" bounded the worst probe inside the bulk transfer
window against the worst probe under the empty-load window. A max over n
samples is a biased estimator when the two n differ, and here they always
do: the bulk window is by construction longer and draws several times as
many chances at an unrelated scheduling outlier. Anything loading the box
widens that gap, so the spec reddened on what else was running: 255ms
against a 161ms budget with make lint beside it, 57ms alone. It now bounds
the probe COMPLETION RATE instead, which is the statistic a serialised
session actually moves and a mean over dozens of samples in both terms, so
a uniform slowdown cancels in the ratio. Measured with make lint running:
0.97 direct and 0.60 relayed against a floor of 0.125.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
nodes.<id>.backend.stop was the last worker-facing NATS subject, and it existed
only because ONE publisher had not moved. An agent worker already mounted
workerctl.PathBackendStop on the tunnel it holds, and a backend worker already
took its stop there, so RemoteUnloaderAdapter branched on NodeType to pick a
carrier for a verb both kinds of worker served the same way.
The branch is gone, and with it nodeTypeOf and its NodeTypeBackend default,
which removes one of the ten NodeType branches left to sweep. The adapter loses
its messaging.MessagingClient outright rather than keeping an unused field: it
now holds no publisher, so re-routing any verb back onto the bus is a change to
the struct and to every caller of the constructor, and does not compile until
all of them agree. messaging.SubjectNodeBackendStop and subjectNodePrefix are
deleted, the agent worker's subscription with them.
pkg/natsauth drops the per-node backend.stop grant from the agent SUB list. That
is a narrowing of eleven entries to ten, never to nothing: NATS reads an EMPTY
allow list as unrestricted, so the coverage spec asserts both that the retired
subject is no longer covered and that the queue subjects an agent worker lives
on still are. The e2e half proves it against a real enforcing server: one spec
subscribes successfully on an agent-minted JWT, the next is refused the retired
subject on a JWT minted the same way.
Both halves of the old split were pinned, so both pins are re-aimed rather than
deleted, and the two node types are asserted separately rather than as one
parameterised case, because only two cases can show that the two used to differ.
Three assertions that the adapter published nothing are deleted instead: with no
publisher to hold, no change could ever redden them.
The CLI's handler set moves into agentWorkerControlHandlers so a spec can stand
it up and post to it. That wiring was a bare literal no spec pinned, and
deleting the subscription made it the ONLY carrier for backend.stop: a dropped
field would have been a 404 the frontend reads as a worker too old to serve the
verb, and nothing in the repo would have noticed.
Mutations: the agent branch restored off the control route reddens two specs;
the backend branch restored, separately, reddens five; PathBackendDelete in
place of PathBackendStop reddens nine across both node types; dropping the CLI
wiring line reddens the new wiring table; re-adding the allow-list entry reddens
the unit spec and the JWT e2e spec; and restoring the publisher for real does
not compile.
Four comments this change falsified are fixed, in core/cli, pkg/model and the
distributed-mode docs, which now say both kinds of worker serve
POST /v1/control/backend/stop and what each does with it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Two independent breakages on master make every open pull request red,
for reasons unrelated to the changes under review.
The e2e backend suite stopped compiling. Reply.message is `bytes` in
backend.proto, so res.GetMessage() returns []byte, and strings.ToUpper
wants a string. Every other call site in the file already converts.
tests/e2e-backends sits behind a build tag, so `go build ./...` never
compiled it and the breakage reached master unnoticed.
The darwin vllm build stopped resolving. Upstream vllm-metal deleted
its old dev tags and re-versioned to track the vLLM release it targets,
so the pinned wheel 404s. The coupled vLLM release also moved out of
upstream's install.sh into .github/vllm-release-tag.commit, and the
wheel's platform tag moved from macosx_11_0 to macosx_15_0.
Read the wheel name from the release's own asset listing rather than
composing it from a hardcoded platform segment, so a platform-tag
change cannot silently 404 again, and resolve the vLLM version from
the new metadata file with a fallback to the legacy installer. The
bump script and the extractor learn the same two-source lookup, so the
next nightly run converges on the pin checked in here instead of
reintroducing the break.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(downloader): make file:// installs reachable again
DownloadFileWithContext already has a branch that copies from a local
file, but it could never run. Before reaching it the function decides
whether the destination is fetchable with
} else if !os.IsNotExist(err) || !URI(url).LooksLikeHTTPURL() {
and LooksLikeHTTPURL is http(s) only, so any URI resolving to a local
path is rejected there. Falling through requires the destination to be
missing AND the source to be an HTTP URL, which a file:// source never
is -- leaving the local-source branch below unreachable.
A first import always has a missing destination, so importing
file:///path/to/model.gguf always failed, with an error that listed
file:// among the supported schemes (#11701).
Name the local-source condition once as URI.hasLocalSource and use it
both to admit the destination and to pick the source, so the two cannot
drift apart again.
Signed-off-by: Tai An <antai12232931@outlook.com>
The recommended sampling parameters for a model family were applied at
install and then never took effect. Two things went wrong on the way to
disk.
They were written as top level keys. ModelConfig embeds PredictionOptions
under the "parameters" yaml key, so temperature, top_p, top_k, min_p,
repeat_penalty and presence_penalty are only read from there. At the top
level they parse without error and are then ignored for the life of the
model.
They were also merged in after the YAML had already been marshalled. The
only re-marshal sat behind the artifact binding, which an entry carrying
files: never reaches, so for those entries the defaults were computed and
then dropped before anything was written.
Neither failure was visible in normal use. ApplyInferenceDefaults runs
again at load time and fills the same values from the same table, so the
model ends up tuned correctly while the file on disk pins nothing. It
surfaces when someone edits one of those values expecting it to win, or
when a family is absent from inference_defaults.json and there is nothing
to refill from.
Both install paths are covered: an entry carrying files:, and one that
binds a primary artifact instead.
The empty base spec asserted that the authored parameters block landed
verbatim. It now checks the authored keys individually, because the family
defaults are merged into that same block.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
Two specs in the distributed e2e suite have been red since 1cf847f29, which
stopped workers advertising an address and removed every fall-back to the
node's own endpoint. After that commit a replica row must name the loopback
endpoint of its own backend process: DistributedModelStore.Range refuses to
list a replica whose backend process is unnamed, and SmartRouter treats an
unnamed warm row as naming no process and cold-loads instead.
1cf847f29 updated the unit specs under core/services/nodes for the new
contract but not tests/e2e/distributed, and the phase's closing verification
ran test-e2e-cluster rather than test-e2e-distributed, so nothing reported it.
The task brief named six later commits as candidates and called 671785621
known good; both are wrong, and 671785621 fails these two specs as well.
The fix is the scripted input, not the guard. Both specs wrote an empty
address, which is a row this release cannot produce, since installBackendOnNode
refuses an install that names no address.
The dedup spec is repaired rather than merely un-reddened. With an empty
address its DB row was dropped by the unnamed-replica guard before Range ever
consulted the seen-set, so deleting the dedup check left it green: it asserted
nothing. With the endpoint named, removing that check reddens it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
mcp.tools.execute and mcp.discovery were the only NATS subjects that
combined a queue group with a reply, and no carrier in this design
provides both. They never needed one: a queue group is a way of choosing
a subscriber, and choosing is a query.
The frontend now lists the approved, non-draining agent nodes, asks the
node_connections table in one joined statement which of those tunnels a
live replica holds, prefers one this replica holds so the call skips the
relay hop, and issues an ordinary control RPC on the path task 4 already
mounted. A peer-held tunnel is reached through the relay. That is a
choice a broker's hidden balancing could not make.
The selection reads presence and nothing else. It is filtered only on
node type and on the two statuses an operator controls, never on a health
verdict written on another clock, because refusing a worker that is
connected and answering is the same defect as picking one that is gone.
An empty fleet answers ErrNoAgentWorker, which is deliberately neither
ErrWorkerUnroutable nor anything cluster.IsWorkerAnswer accepts: nothing
was asked of any worker, so no reap guard may act on it.
A reply carrying an Error is the worker's own answer and is returned
unchanged; it is never offered to a second worker, which would turn "this
MCP server rejected your arguments" into "the fleet is broken" and could
run a tool twice. A call that never reached a worker is retried against a
different pick, at most three times, and whatever error is finally
returned is returned unwrapped so its identity survives the loop.
MCP prompts and resources now answer 501 in distributed mode instead of
an empty 200. They are served only from sessions the frontend holds, and
in distributed mode it holds none. That gap predates the removal of the
bus and is not closed by it; this only stops it being silent.
Agent workers keep every other subject, including nodes.<id>.backend.stop.
Their minted JWT loses the two MCP subjects and keeps a non-empty allow
list, because NATS reads an empty one as no restriction at all.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Phase 2 gated agent nodes out of tunnel credentials at the mint site.
That was right while nothing dialled into an agent worker: a credential
would have replaced nothing, and the gate was structural rather than a
second check that could drift. It is wrong now that the frontend needs to
reach an agent worker by RPC.
attachTunnelToken mints for backend and agent nodes and CLEARS for
anything else, through one tunnelEligible predicate rather than two
conditions that can be widened separately. ConnectHandler still never
reads NodeType, so an empty hash is still what refuses an ineligible
node.
An agent worker now starts a loopback control server behind the same
bearer check a backend worker uses, and holds one tunnel whose only
stream tag is http: it runs no backend processes, so the grpc tag has
nothing to route to and is not offered. Its MCP tool, MCP discovery and
backend.stop verbs are served from ONE implementation reached by both the
bus and the tunnel, so a frontend cannot get different bytes depending on
which carrier delivered.
The tunnel is an ADDITION. --nats-url is still required, and agent jobs,
MCP execution, MCP CI jobs and nodes.<id>.backend.stop all still travel
on the bus.
Absence semantics are unchanged. An agent node now has a real
node_connections row whose departure ages past the grace, so the node
type check in HealthMonitor.tunnelDeparted stopped being an optimisation
and became the rule; its comment says so, and the spec that pins it is
shown red under a mutation that deletes the check. The scheduler needed
no change: every placement query already filters node_type = backend, so
an agent node never reaches nodeMayTakeWork.
Shared rules moved to one site each. The request bounds, the POST-only
check and the unknown-path 404 live in workerctl and are called by both
worker packages; the bearer check that guards every extra route is one
function in core/services/nodes used by both server constructors.
workerctl.AllPaths splits into BackendPaths and AgentPaths, with AllPaths
as their deduped union, because a backend worker does not mount the agent
verbs and asserting otherwise would fail a correct worker.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ds4): build CUDA kernels for the target architecture
The ds4 backend compiled its CUDA objects with no -arch. Upstream's Makefile
leaves CUDA_ARCH empty and its `cuda` target refuses to build without one,
offering `cuda-spark` (sm_121) and `cuda-generic` (native) instead. We invoke
its object targets directly, which bypasses that guard, so nvcc fell back to
its default architecture and the kernels ran as JIT'd PTX on the real GPU.
On GB10 (sm_121) that silently corrupted inference: any prompt over roughly 128
tokens produced text unrelated to the input and never closed its thinking
block, so content came back empty and the chat showed only reasoning; longer
prompts failed with "cuda decode failed". It also cost close to two orders of
magnitude of prefill throughput. Measured on one box, same model, same prompt,
same GPU, upstream ds4 at the pinned commit, differing only in the nvcc flags:
make -B ds4 (archless, as we build it) garbage output 4.21 t/s
make cuda-spark (compute_121a/sm_121a) correct output 325.70 t/s
Select an architecture list from CUDA_MAJOR_VERSION, which the backend matrix
already declares for both ds4 cublas entries but Dockerfile.ds4 never forwarded.
Upstream's CUDA_ARCH takes a single value, so it cannot express the fat binary
these images need; NVCC_ARCH_FLAGS is overridden instead, since a command-line
assignment wins over its `:=`. The lists are copied from vllm-cpp rather than
invented so the two CUDA images cover the same GPUs, with l4t/arm64 covering
Orin, Thor and GB10. An empty CUDA_MAJOR_VERSION keeps upstream's `native`
behaviour for local developer builds, and no CI runner has a GPU to enumerate.
DS4_CUDA_HAVE_MXF4 is deliberately left unset: upstream defines it only for
single-arch sm_120/sm_121 builds and guards it with a plain #ifdef rather than
__CUDA_ARCH__, so it cannot be combined with older archs. It gates an optional
MXFP4 indexer fast path whose #ifndef branch returns 0 and falls back cleanly,
so omitting it costs speed on GB10, not correctness.
Assisted-by: Claude Code:claude-opus-5
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
* test(ds4): cover the multi-batch prefill regression
The architecture fix has no automated guard: every existing e2e spec uses a
short prompt, and the miscompiled backend answered short prompts correctly.
The corruption only appears once a prompt spans more than one prefill batch,
so the whole suite passed against a backend that produced garbage in normal
use.
Add an opt-in "long_prefill" capability to the backend e2e suite that sends a
prompt well past one batch with a known needle and asserts the answer still
reflects it, and document in the ds4 guide why the build must never omit an
nvcc architecture, how to check which flags a configuration resolves to
without compiling, and how to run the new spec.
Assisted-by: Claude Code:claude-opus-5
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
---------
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
Propagate gRPC cancellation into DS4 prompt synchronization and poll it at decode boundaries.
Stop on failed stream writes and skip parser finalization and KV persistence for abandoned partial requests.
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
PostgreSQL holds undelivered notifications in a queue it shares with every
session on the server, and it kills a listener that stops draining. Two failure
modes follow, and both are silent: a carrier that blocked on a slow resolver
would lose its connection and with it every later broadcast, and a carrier that
reconnected without re-registering would be connected and deaf.
The receive and dispatch halves were already separate. What was missing is
everything around them. The listener path moves into listener.go and gains a
carrier-level Dropped() so a replica that is behind can be seen; the queue depth
and the spill retention become Config fields with exported defaults; the LISTEN
session gets an application_name so an operator can count listeners in
pg_stat_activity and a spec can drop exactly one of them; and OnReconnect fires
after the re-LISTEN, on a goroutine of its own, because a callback re-hydrates
from a database and must never run on the path whose only job is to drain.
That callback is reached through an optional interface assertion, so deleting
its invocation compiles and every adopter silently stops converging. The spec is
the only guard, and it is named in a comment at the site.
The slow consumer is proved through the transport rather than a seam: an ACCESS
EXCLUSIVE lock on bus_messages stalls the resolver's spill SELECT for exactly as
long as the spec holds it, and the listener is shown still draining and dropping
while it does. The dropped connection is a pg_terminate_backend matched on the
carrier's own application name.
Neither Dropped nor IsConnected is on messaging.Broadcaster, and a spec asserts
that over the interface type. Both are facts about a frontend; the conditions a
scheduler acts on are facts about a worker, and no consumer holding the
interface can read one as the other.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Unsubscribe decided a channel had lost its last subscriber under one
lock and issued the UNLISTEN after releasing it. A Subscribe on the same
root could decide to LISTEN in that window, and the two reached the
connection in the wrong order: the root ended up not listened with a
live subscription on it. It does not heal, because the next Subscribe
sees the registration already there and never re-LISTENs, so the whole
root stays deaf on that replica until the connection drops.
The decision and the statement it implies now happen under one lock,
held across both, at both call sites. A second lock and not the
registration lock: issuing waits on the listener goroutine, delivery
takes the registration lock, and holding that across the wait deadlocks
the carrier.
The race is spec'd through a barrier seam rather than by racing
goroutines. The natural window is microseconds wide, and a spec that
waits for it to open passes by luck; the seam scripts the interleaving,
so the spec decides in both directions.
Resolving a spilled message moved off the listener. PostgreSQL keeps
undelivered notifications in a shared, fixed-size queue, so a listener
that stops draining it can block COMMIT for every publisher on the
server, not only this one. The listener now only drains; one resolver
goroutine reads the row back and dispatches, which also keeps a spilled
message and an inline one on the same subject in the order they were
published.
Three wiring lines that could be deleted with the suite staying green:
the sweeper's start is now pinned by a Config interval, and the two
lines that carry the bus into the deployment now refuse to boot when
either is missing. A subscription can also report what it dropped, so
the party that missed a message is the party that can see it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Distributed mode needs an operator to run a NATS cluster. This adds the
carrier that replaces its fan-out half, so a deployment eventually needs
PostgreSQL and its own HTTP listener and nothing else.
pgbus holds one PostgreSQL session per replica, pinned for the life of
the process because LISTEN registrations belong to one backend session
and a pooled handle would lose them on the next checkout. Publishes go
out on the pool with pg_notify.
Subjects map onto a channel by their first token, from a closed set of
roots. A subject outside the set is refused at publish AND at subscribe
rather than mapped to a channel of its own: a channel name is capped at
63 bytes, and one LISTEN per job id would be unbounded. Refused rather
than dropped, because a subject that goes nowhere and reports nothing is
the class of defect this work exists to remove.
PostgreSQL refuses a notify payload of 8000 bytes or more, and several
subjects on this bus exceed that in normal operation: a job result
carries a whole LLM output, a gallery progress event carries one entry
per node. Those are written to a row and the notification carries the
id. What is measured against the cap is the ENCODED notification, not
the caller's payload, because the subject and the envelope travel too.
The filter grammar is not respelled here. Subscribe asks
messaging.ValidFilter and delivery asks messaging.SubjectMatches, which
makes this the first production caller of a matcher that had only test
doubles. New refuses a DSN that names a different database from the
pool: that pairing publishes successfully, delivers nothing, on every
replica, and reports no error anywhere.
Nothing publishes on it and nothing subscribes yet. The construction is
wired anyway, because the DSN has exactly one legitimate source and a
setting that decides whether any broadcast is delivered should not be
invented by whichever call site is migrated first.
Delivery is at-most-once, like NATS core. Nothing downstream may read a
message it did not receive as evidence about a node: a carrier that
cannot deliver is not a worker that is gone.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Three copies of "does this filter match this subject" lived in the tree: one
in testutil.FakeBus, a byte-identical second inside galleryop's own private
fakeBus, and a third in pkg/natsauth with different semantics. The first two
are doubles the specs publish through, and the carrier the pgbus work is about
to add needs the same rule in production. Two spellings drift, and the drift
reads as a peer that receives an event on one replica and not on another.
messaging.SubjectMatches is now the only definition either double uses. The
natsauth copy stays: it matches a NATS server allow list, so it has to
implement the '>' tail wildcard this one deliberately refuses, and Task 16
deletes that package anyway.
'>' is refused rather than implemented because no surviving subscription uses
it, and a caller who writes one must get no messages rather than silently
getting every message on the prefix. The refusal is checked BEFORE the
filter == subject fast path: a verbatim port checks equality first, and then
the filter "a.>" matches the literal subject "a.>", which is the contract
leaking. One table row pins that ordering and it is the only row that does.
messaging.ValidFilter refuses an empty filter, a '>' filter and an empty token
so a subscriber is told at subscribe time instead of staying silently empty for
the life of the process. FakeBus.Subscribe calls it, which is what keeps the
double honest about what the carrier will do, and three new testutil specs pin
that wiring: the previous state of the tree had no spec at all that failed when
the double's wildcard routing was replaced by exact matching in any package the
plan named.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Five cluster specs that run the binaries an operator runs, plus the repair
of eighteen specs phase 2 left red.
The eighteen were router_tracking and full_flow, failing since 1cf847f29 on
"reported backend installed but named no address for the process". Two
contracts had changed under them: an install reply that names no
worker-local address is refused rather than substituted, and a frontend with
no worker dialer reaches no backend at all. Nobody noticed for a phase
because phase 2 verified with --label-filter='Cluster', which excludes both
suites. ServeBackendLifecycle and tunnelBackendClients state both facts once
for every spec.
The transport double is the part that matters. It translates a refused
connect into cluster.ErrStreamTargetUnavailable, which is what a real worker
answers when its backend process has died and what IsWorkerAnswer lets a
reap guard act on. A bare ECONNREFUSED reaches those guards as "no route" and
reaps nothing, so a double returning the raw syscall error could never fail
the way production fails; putting it back reddens the stale-record spec and
nothing else.
The new specs cover: a backend worker with no bus URL in its /proc environ
registering, being scheduled onto and serving inference; a backend install
and a backend listing driven through the replica that does NOT own the
worker, with the owner read through the production Owner query and re-read
after; that install's progress proven to arrive before its terminal reply,
made deterministic by a gallery server that holds the worker's fetch open so
a reply cannot exist yet; a worker whose tunnel is genuinely gone, waited for
rather than assumed, losing nothing inside the reconnect grace and re-homing
after; a heartbeating worker with a permanently dead tunnel losing its
healthy status while an agent worker in the same cluster keeps it; and the
suite's negative control, where a control RPC to a tunnel-less worker fails
naming the missing route, reaps nothing, and succeeds the moment the tunnel
returns.
Every scenario was attacked. The churn one was WRONG on the first attempt
and only the attack found it: its hold window sat entirely inside
cluster.InstanceLiveness, so a killed replica still read as a live owner
throughout, presence was "connected", and the spec passed with the reconnect
grace set to a nanosecond. It now blocks the tunnel before the kill and waits
for the ownership row to actually empty. Attacks that redden the rest:
posting at the owner, writing the install reply before the work, collapsing
PresenceReconnecting into PresenceGone, removing the non-backend node-type
guard, and not blocking the tunnel. Agent workers turn out to be protected
twice over; no single mutation reaches them.
Harness: Options.AgentWorkers and Options.ReconnectGrace, WorkerEnviron
(read from /proc, because Cmd.Env is the harness agreeing with itself),
NatsURL, FrontendBackendsDir, AgentWorkerName, PostJSON, and a node String()
so a failing roster assertion is readable instead of several hundred bytes
rendered as numbers.
Budget: 20 specs at 787 to 808 seconds over three runs, up from phase 2's 591
to 612. --timeout goes to 30m so a loaded runner reports a cause rather than
a spec name.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A local-ai worker no longer opens a bus connection. connectNATS and its
spec are gone; Run registers once, starts its tunnel, arms /readyz on that
tunnel, and heartbeats. The worker's bus credential flags (--nats-jwt,
--nats-user-seed, --nats-require-auth, the three TLS flags) and
Config.NatsAuthRequired go with it. --nats-url stays, accepted and
ignored, so an existing worker command line still parses.
/readyz was the thing most likely to wedge a tunnel-only worker: it
required a live NATS link, so a worker with no bus would have reported
itself unready forever. nodes.NATSReadiness becomes nodes.TunnelReadiness
over a local interface{ Connected() bool }, and worker.Tunnel gains
Connected(), backed by a mutex-guarded session field the loop publishes
and clears. A closed-but-not-yet-cleared session reads as disconnected:
the loop waits for every in-flight stream before it clears the field, and
the probe must answer not-ready through that wait.
The heartbeat gate is DELETED rather than re-pointed at the tunnel. The
heartbeat is the worker's own answer that its process is alive; whether
the frontend can reach it is a separate fact the frontend already holds
and ages against LOCALAI_WORKER_RECONNECT_GRACE. Withholding the
heartbeat would report an unreachable worker as an absent one on the one
path with no grace, where the health monitor marks it offline and its
pending backend ops are deleted behind it. heartbeatLoop is given no view
of the tunnel, so a gate cannot be added back without changing its
signature.
Removing the NATS credential manager from this path also removes a defect
it carried: its refresh loop re-registered on a timer to renew a JWT, and
Register CLEARS a node's NodeModel rows. Any backend worker running on
frontend-minted credentials had its replica rows deleted roughly every
18 hours.
Of core/cli/workerregistry, everything survives. The manager is still
used in full by core/cli/agent_worker.go, which still needs NATS: Acquire,
Provider, RefreshLoop, HasCredentials and TunnelToken are all untouched.
The backend worker simply calls RegisterFullWithRetry directly now.
WorkerPermissions is documented as serving agent nodes, and its non-agent
branch narrowed to _INBOX.> on both sides. It is NOT deleted: NATS reads
an empty allow list as no restriction, so returning nil would upgrade
every JWT the frontend still mints for a backend node from its own inbox
to the whole account.
Agent workers keep the bus everywhere: their CLI flags, their
subscriptions, the agent branch of WorkerPermissions, and the compose
service with its LOCALAI_NATS_URL and depends_on: nats.
Also corrected two flags the Nodes page advertised that do not exist
(--distributed-nats, --distributed-db), and a log line plus several
comments that still named a bus the code no longer touches.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Removing "Presence: clusterRegistry" from the options literal in
initDistributed left all seven suites and tests/e2e/distributed green.
The predicate was right and its input was silently nil, which returns the
deployment to absence being decided by nothing, with no log line and no
failing request. That is the fourth finding of this exact shape in this
phase.
The two assignments move out of a twenty-field literal into
distributedSchedulerOptions, a named function a unit spec can reach.
Deleting either is now red. The health monitor takes its presence reader
and grace as a required positional pair instead, so deleting those does
not compile at all. requireAbsenceWiring then refuses to start a
distributed frontend whose scheduler or health monitor has no source of
absence, because refusing to boot is the only symptom either failure has.
With a fresh heartbeat and a permanently gone tunnel there was no reaper
at all. A heartbeat says the worker's supervisor is alive; it says
nothing about whether anything here can reach that worker's backends,
because those are reached over the tunnel. A proxy that stops upgrading
WebSockets, a rotated registration credential or a reconnect loop longer
than the grace left a node listed healthy forever while every request for
a model already loaded on it failed "no route to that worker", and every
reaper keyed on the heartbeat. The health monitor now reads presence from
the same place and against the same window as the scheduler and demotes
such a node. That also ends the 15s re-promotion: the demotion arm
returns before the recovery arm, so the scheduler's demotion is no longer
undone on the next tick, and recovery needs the tunnel back rather than
just the heartbeat.
The demotion is status-only. MarkOffline would DELETE the node's rows,
and deleting rows on a presence read would give any future defect in that
read the widest blast radius in the system for nothing the demotion does
not already deliver.
LRU eviction is the third path that commits work to a node, and it read
only the stored status. A node full enough to be an eviction target is
exactly the node the VRAM and idle selectors never offer, so
pickReachableNode structurally cannot cover it. It now runs its chosen
node through the same nodeMayTakeWork predicate, demotes it and evicts
again rather than handing back an install that cannot land. Presence is
read after the transaction and not inside it: reading it inside would
hold a FOR UPDATE lock across a query needing a second pooled connection,
which is how concurrent evictions deadlock a pool.
Also: a router built with a presence reader and no grace now has its
documented default pinned by a spec rather than only claimed by a
comment; ageDeparture asserts RowsAffected, since an UPDATE matching
nothing succeeds and the inside-the-grace spec returned the same verdict
either way; the scheduler comment that still described the bus is
corrected; the docs stop conflating heartbeat recovery with tunnel
recovery and name the third reader; and an overlong rewrapped line in
membership.go is folded.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The scheduler decided whether a worker had gone away from
nats.ErrNoResponders: one frontend's observation that nobody answered IT
within a request budget. Two replicas asking in the same moment could
disagree and demote each other's workers, and a worker re-homing its
tunnel between replicas looked identical to one that had died.
SmartRouter now reads cluster.Presence instead. Only PresenceGone -- no
live replica holds the tunnel AND the departure has outlived the
reconnect grace -- excludes a node from placement, and it is a fact every
replica reads identically from the database. PresenceReconnecting,
PresenceUnknown and a failed presence query are all non-verdicts and
place work as normal: excluding on a database hiccup would cost the fleet
its capacity for a reason that has nothing to do with any worker.
nodeAnswersOnBus is deleted. It excluded on a sentinel no control RPC can
produce, so it decided nothing while PingNode cost a relayed round trip
per scheduling decision to feed it. PingNode goes with it, from the
adapter and from NodeCommandSender. isRequestTimeout drops nats.ErrTimeout:
every verb this adapter sends now travels over the worker's tunnel.
The predicate is named nodeMayTakeWork rather than nodeHasRoute. "Route"
is ErrWorkerUnroutable in this package, the condition nobody may act on;
PresenceGone is the one a scheduler may. Spelling them the same way is
the collapse this work exists to prevent.
Also folds in ReapStale's return rename: it counts connection rows
CLEARED, never rows deleted, and reading it as a delete count would make
a worker that is re-dialling right now look forgotten.
The spec pinning that a message merely quoting "nats: timeout" is not a
timeout was scripting a SUCCESSFUL reply carrying the phrase, which comes
back with a nil error and never reaches the classifier. Restoring the
string match left it green. It now scripts a 5xx whose body carries the
phrase, and asserts that the phrase reaches the classifier as a
precondition.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The rule "a body this worker could not parse is a non-2xx, never the
worker's answer" is written at three exits in control_files.go and only
ensure was pinned. Turning stage's or listdir's decode exit into a
200-with-error left worker and nodes entirely green, and what that
converts is a frontend's malformed request into the worker's own verdict
about a file, which passes cluster.IsWorkerAnswer and reaches a reap
guard. The production code was already right; nothing held it there.
The e2e NATS JWT spec was asserting the opposite of the code and passing.
It published nodes.<id>.files.in and called it an allowed subject after
that grant was deleted, and it could not tell: a permission violation
does not close the connection, so FlushTimeout and IsConnected both stay
happy. It now reads LastError, the way its sibling always has, and
asserts the denial plus the one publish right a backend worker has left.
Also pinned, each mutation-verified alone: the CreateTemp branch (an
existing staging-tmp at 0500 reaches it without a seam), the walk's
context check (a caller that gave up must fail the listing, never be
answered with a short one), and the cache and data directory layout.
The data directory was derived twice, once in worker.go and once for the
listdir verb; worker.go now reads the same helper, so a move cannot leave
a verb listing files the file server does not serve. The per-verb RPC
ceiling moves from an argument at five call sites into fileRPCBudget,
so no site can name the wrong one, and the two values are asserted.
The body-cap table now holds both directions locally and with two
different claims: a body exactly at the cap proves the bound is a
ceiling and not an off-by-one, and an absolute megabyte proves the cap
stays above real gallery traffic. Only the second notices a cap shrunk
to 64 KiB.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The four nodes.<id>.files.* subjects were the last commands a
serve-backend worker took off the bus. They are now HTTP routes under
workerctl.Prefix, on the same loopback server and behind the same bearer
check as the ten lifecycle verbs, so the frontend reaches them through
the worker's tunnel.
files.listdir is the verb this matters most for. Its reply had to fit a
payload the bus would carry, which put a wide model directory close to
the limit; a response body has no such ceiling, so nothing truncates the
listing at either end. A short listing reads to the frontend as files
the worker does not have.
S3NATSFileStager becomes S3FileStager and calls ControlClient, which
means every failure now lands in the bucket phase 3 exists to keep
straight: a route this frontend could not use is unroutable and nothing
may act on it, while the worker's own answer, including "that file is
not there", is evidence a caller may act on. Each RPC's deadline is
DERIVED FROM the caller's context rather than started fresh, at every
one of the five call sites, so a caller that gave up stops the RPC too.
A worker started without an object store mounts no file verb at all and
answers 404, which is the same answer a build too old to know them
gives. The subjects and the backend worker's files.> publish grant go
with them; a backend worker now publishes nowhere but its own inbox.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review fix round 1. Two blocking findings and seven non-blocking; both
blocking ones are M12's shape again, and this time on the invariant
itself. No production behaviour changes here: everything below was
already correct and merely unpinned, so re-inserting the defect left all
679 specs green. The only non-comment edits are one struct-field comment
and one log message.
"A failed control RPC no longer demotes a node" is stated three times in
this package and was pinned once, at ListBackends. Putting MarkUnhealthy
back at either op-drain site passed. What that buys in production is the
fleet-wide eviction this phase exists to prevent: MarkUnhealthy removes a
node from ListDuePendingBackendOps AND from scheduling, so a frontend
replica that has just lost its tunnels demotes every node it holds an op
for, for a reason that is about the frontend. The reconciler's is the
worse of the two, being a background loop nobody is watching. Both now
have a spec, each with the recorded op failure as its negative control so
"still healthy" cannot pass by nothing having happened.
The sweep the review asked for found four more rules stated at more call
sites than they were pinned at, and two the review had not:
The still-installing surfacing at the manager layer has two call sites
and was pinned at InstallBackend. Dropping it from UpgradeBackend
reported a spent budget as GREEN SUCCESS: the admin sees the upgrade
finished while the worker is still re-pulling gigabytes.
The agent-node skip has two call sites and was pinned at ListBackends.
Without it the fan-out enqueues a row for every agent node, and an agent
worker serves no control plane, so that row can never drain: it retries
until the dead-letter cap.
The still-installing conversion has three call sites and was pinned at
two; the legacy force-install fallback was the gap. Its budget was
unpinned too, so the new spec asserts both, on the upgrade budget rather
than the install one, since the fallback re-fires an install as part of
an upgrade.
The carrier split has two call sites and was pinned at one. Hardcoding
NodeTypeBackend in UnloadRemoteModelContext passed, and an agent node
holding a node_models row would then have its stop sent over a tunnel it
does not hold, fail, and leave the row behind. The new spec unloads a
model held by one node of each kind and asserts each stop went to that
node's own carrier and to no other.
router_nats_liveness_test.go asserted demote-on-absence, which production
can no longer produce, and its header described the pre-cutover world.
The exclusion is unreachable by construction rather than by argument:
cluster, the package supplying every control-path dial error, does not
link nats.go at all. The file now says that, and gains the assertion that
IS load-bearing, a table naming each sentinel a control RPC can answer
with and requiring that none of them excludes. Widening the exclusion to
ErrWorkerUnroutable reddens four of its entries plus the real-adapter
scheduling spec.
unroutable keeps no budget-first guard and the reason is now written at
it: unlike controlFailure it reads one already-recorded error rather than
racing a live deadline, and an expiry is not in streamRefusals, so it
falls to the umbrella without one. The two implement the same split at
two layers and each now names the other.
Fourteen comments still described the bus. Among them the reconciler
saying a drain would "churn NATS every tick", a spec comment naming a
subject builder this branch deleted, and the agent-skip comment
explaining the skip by a subscription that no longer exists.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The ten backend and model lifecycle verbs stop being NATS requests and
become HTTP calls on the worker's own control routes, reached through
that worker's tunnel on the `http` stream tag that already carries file
staging. Nine subject builders and the per-op install-progress subject
are deleted with their entries in the worker's NATS permissions; the
request and reply DTOs are untouched, so a body on the wire is byte for
byte what the subject carried.
This closes the merge gate Task 3 left open, which was worse than lost
commands. Once the worker stopped subscribing, PingNode was still asking
nodes.<id>.backend.list and nodes.<id>.models.running, so EVERY healthy
worker answered no-responders, nodeAnswersOnBus read it as absence and
pickReachableNode demoted it on the scheduling path. PingNode is a
control RPC now, and no control RPC can produce ErrNoResponders, which
is the only error that exclusion acts on. Two specs drive
pickReachableNode against a real adapter and a worker answering over its
control plane, which is the only arrangement that can see the difference:
the router's own double never touches a transport and stayed green for
the whole window the defect was live.
How a control RPC FAILS is the whole of this change, so it is decided in
ONE function reading ONE table. A worker's answer passes through
unwrapped, so cluster.IsWorkerAnswer still sees it and a reap guard may
act on it; everything else is wrapped in ErrWorkerUnroutable so nothing
can. There is no third branch, because a third branch is how the eight
collapses on this branch happened: each was a site that decided for
itself which errors were evidence. A 404 under the prefix is its own
sentinel, because it is the worker stating a deployment fact about
ITSELF rather than a verdict about a backend, and only the legacy
upgrade fallback may act on it.
The caller's budget is checked FIRST. A timeout is not a verdict: a
refusal arriving in the instant a deadline expires would otherwise be
reported as the worker's non-transient answer, which reaps a row, and
nothing orders the two timers.
A 5xx and an undecodable body are transport failures, not answers. An
empty ModelsRunningReply means "this worker is running nothing", which
the reconciler acts on, so it must never be manufactured from a body
that would not parse. A stream that ends before its reply line is the
same rule one layer up: a tunnel dying mid-install is not the worker
saying the install failed.
backend.stop is split by node type rather than moved. Agent workers hold
no tunnel, so they have no control plane to serve, and they still
subscribe to nodes.<id>.backend.stop to drop cached MCP sessions; that
subject and its agent permission both survive. It is the honest
intermediate state until agent workers hold tunnels too.
A failed control RPC no longer demotes a node anywhere. ErrNoResponders
meant "not on the bus"; a control failure means "this frontend could not
route to it", which is equally what a healthy worker re-homing its tunnel
between replicas produces. Absence is a fact read from the database, and
the scheduler starts reading it in a later task.
The rolling-update fallback re-fires a DESTRUCTIVE force-reinstall, so
it runs only on the worker's own 404. Its negative direction was pinned
at the admin call site and unpinned at the reconciler's, where widening
the condition to any error left all 676 specs green: a background drain
nobody is watching would then force-reinstall every queued backend the
moment a replica lost its tunnels. Three specs cover it, arranged so the
force install IS reachable in the negative case and a fallback that
fired would show as a call and a drained row.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review fix round 1. Seven non-blocking findings; the blocking one is a
merge gate for Task 4 rather than anything in this diff, and the report's
concern about it is corrected: until Task 4 lands, PingNode probes two
subjects no serve-backend worker subscribes to any more, so every healthy
worker reads as absent and is marked unhealthy on the scheduling path.
The rune-boundary cut in truncate was true behaviour with nothing
holding it: a byte-wise mutation survived all 201 specs. isRuneStart is
replaced by utf8.RuneStart, the same predicate the cluster package uses
for this rule, and two specs pin it, one with a rune straddling the
bound and one with a rune ending exactly on it so the fix cannot be
"always walk back".
unloadModel answered Success:true whatever Free did. That is the worker
saying "done" about work it did not do, and the frontend's only caller
is EvictLRU, so a false yes told the scheduler VRAM had been released
and let it place the next model on a node still holding the old one. It
now reports the failure, following stopModelExact, which is the honest
pattern already in this package. Still a 200: the worker answered, only
its verdict is negative. An address with nothing loaded still answers
success, which is a true answer rather than a claim about work done.
NewDebouncedInstallProgressPublisher had no production caller after the
last commit, only its own spec. Deleted rather than wired: wiring it
would publish every event on two carriers, which is what the carrier
decision exists to avoid. Its specs now run against the sink, plus one
that pins the identity stamped on each event, since the subject used to
carry the op and node id and now nothing but the body does.
The install progress wiring was exercised by no spec, because with no
gallery nothing ever invokes the download callback. The guard moves into
startProgress, shared by install and upgrade, which also emits one
resolving event before any gallery work. That is worth having on its
own: a cold install spends minutes on a manifest and a progress stream
with nothing on it is indistinguishable from a broken one. It also makes
the wiring observable end to end, and four specs now drive the real
installBackend and upgradeBackend over HTTP with no override.
model/stop and backend/stop keep taking Background rather than the
caller's context, and the sites now say why. model/stop is the
acknowledged stop path: it reserves the process, frees it, kills it,
waits for exit and releases the port, and abandoning that because the
caller hung up would leave a process marked stopping, a port not
returned to the allocator and a row nothing reconciles. In
stopBackendExact the Free is a courtesy before a kill that happens
anyway. model/unload differs because Free IS the operation there.
A route set with no prefix or no registrar is now a startup error rather
than a silent no-op: a server that comes up healthy while every route
the caller registered answers 404 is, through a tunnel, indistinguishable
from a version skew. And the AllPaths spec no longer claims to catch a
constant that was never added to the set, which it cannot; it asserts
the whole set instead, which catches a verb dropped from it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Ten NATS subscriptions on the worker become ten HTTP routes under
/v1/control/, served on the loopback HTTP server the worker already runs
and reached only through the tunnel's existing `http` stream tag.
The carrier is the tag that already exists rather than a new one. A new
tag would have had to invent correlation, per-request deadlines,
unbounded payloads and a progress stream, and each of those is a place
this branch has already put a defect. It would also have added a fifth
entry to the worker's stream-refusal vocabulary, which decides what a
frontend reaps on and took eight fixes to settle. Riding `http` means a
control RPC to a worker another replica holds takes the same relay the
inference path takes, which is the path that has been measured.
The request and reply DTOs are untouched, so a body on a control route
is byte-for-byte what the corresponding subject carried. No subject was
deleted: agent workers still subscribe to nodes.<id>.backend.stop.
Install and upgrade stream. They answer application/x-ndjson: zero or
more {"progress":...} lines carrying the same event the per-op NATS
subject carried, then exactly one {"reply":...} line, always last. That
deletes the 8000-byte notification cap structurally instead of
reproducing it on a new carrier: a progress line is written into the
response the caller is already reading, so there is nothing to size and
no subscribe-before-request window. The debouncer is shared with the
NATS publisher rather than forked, so the ~4/s tick bound is one fact.
A verb's own failure is a 200 with Error set, never a 5xx. The frontend
maps a transport failure onto "no route to that worker", which nothing
may act on, and the worker's answer onto evidence a reap guard may act
on; answering 500 for a failed install would put the worker's verdict
in the bucket reserved for a broken link. Only a request that could not
be read or routed is non-2xx.
Control RPCs carry the caller's budget. r.Context() replaces four
context.Background() calls at the gallery-install sites, and the one
pre-existing fixed timeout on model.unload is now derived from the
caller's context so a shorter budget is honoured. No timeout is invented.
The inner `go func()` in the install and upgrade handlers is deleted
rather than nested: it existed because one subscription served every
install, and over HTTP each request already has its own goroutine.
Per-backend serialization stays lockBackend, which is what actually
prevented two requests racing the gallery directory.
Bounds against a boundary the worker now serves: every body is capped at
8 MiB before any decode; the 404 echoes at most 128 bytes of the request
path, cut on a rune boundary so a half rune cannot travel downstream as
a replacement character; non-POST is refused before the body is read so
a probe cannot fire a command; the streaming responses set nosniff.
The routes mount through nodes.AuthenticatedRoutes, which hands the
registrar a private mux and puts the whole prefix behind the same
constant-time bearer check as the file routes. The worker's HTTP server
now takes the supervisor as a required parameter, so there is no way to
start it without the control plane mounted.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The retention a worker's departure is kept for is now derived from the
reconnect grace, so a purge can never outrun the window Presence measures
against. Nothing pinned that. The sweep could be reverted to pass the constant,
or the setter emptied out, and the suite stayed green either way: the specs
covered the arithmetic helper, and the fix is the wiring.
The loop now has a spec of its own. It departs two workers either side of the
difference between the floor and the derived retention, and the row that must
go is what witnesses the sweep running at all, so the row that must stay cannot
survive by nothing happening.
The default grace goes from 60s to 90s. Two of the worker's ceiling backoffs is
60s, but the failed dial between them costs its handshake timeout too, which
puts the worst case at 70s, and the backoff resets only after a session long
enough that a replica accepting a dial and then dying denies it. So the ceiling
is reachable exactly during the rolling restart this window exists for, and 60s
sat on the edge of it. Too short reports a live worker as gone and costs a
model reload; too long reaps a dead one later. The cheaper mistake is the long
one.
A held row whose owner is dead and whose stamp is stale is the state a rolling
upgrade actually produces, and it was the one state no spec built. It has an
answer now, and the two ways to get this wrong land either side of it: reading
the stamp first says gone, reading held-ness without the liveness join says
connected.
Two comments claimed more than the code did. There IS a grace at which a live
worker is reported as gone, which is the point of it being a duration; and the
switch that reads held-ness first is only a partial second gate, since with the
SQL gate gone and a dead owner it answers gone rather than reconnecting.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A worker whose tunnel is gone is not, by that fact, a worker that has left.
Absence is what makes the scheduler stop placing work, reap the worker's rows
and evict its models, and one of those paths runs during inference, so the
deployment needs to tell a worker re-homing between frontend replicas from one
that is really gone before anything acts.
Registry.Presence answers that in one joined statement, with four values and
not a boolean: unknown when there is no row at all (this package cannot tell a
worker that has never dialled from one whose departure aged out, and must not
guess), connected while a live replica holds the tunnel, reconnecting while the
departure is inside the grace, and gone once it is older. Only the last is a
verdict a caller may act on.
Held-ness is asked FIRST and the departure only refines it, in the SQL and
again in the switch that reads it. Every writer here clears disconnected_at in
the statement that writes the owner, but that is a property of these writers
rather than of the table: a replica running a binary from before the column
existed re-claims without clearing the stamp, so during a rolling upgrade a
held row carries an old departure, and a read that consults the stamp first
reports a connected worker as gone for the whole upgrade.
Both windows are computed by the database, for the reason every other window in
this package is: they are compared across replicas, and replicas disagreeing
about whether a worker is gone is the flapping this branch exists to remove. No
behavioural spec can see the difference, since the test container shares the
host clock, so the statement shape is pinned instead.
The grace is an operator's knob, defaulting to twice the worker tunnel's
maximum reconnect backoff. That made the fixed departure retention wrong: an
operator raising the grace past it gets a purge that deletes departures before
the grace elapses, so a worker that is gone reads as unknown forever and
nothing ever reaps it. The retention is now derived from the grace, with the
old constant as its floor.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The membership tick's call to PurgeDepartedBefore was the only production
wiring this change introduced, and removing it left the suite green. A
retention nothing applies is a departure that never ages out, which is
the state the sweep's held-ness filter exists to make reachable at all.
A spec now ages a released row past DepartedRetention on the database
clock, starts a real Membership, and waits for the row to go.
Release and Deregister leaned on "no owner id is ever empty" to avoid
touching an already-departed row, which is the accident Owner refuses to
lean on. A departed row keeps its epoch and carries an empty owner, so a
release or a deregistration naming an empty id matched it and stamped a
fresh departure over the old one, making a worker that left long ago look
like one that has only just gone. Both now filter on connectionIsHeld.
The comment on DisconnectedAt claimed a held row never carries a
departure. A binary from before this column existed claims without
clearing the stamp, so a rolling upgrade produces exactly that row. The
comment now says what holds, and says to ask held-ness first and read the
stamp second.
The sweep's vocabulary follows the code: it records departures where the
comments still said it deleted rows, and its log line separates the
instance rows it deleted from the connection rows it left behind.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Releasing a worker tunnel deleted its node_connections row, so "this
worker's link dropped a moment ago" and "this worker has never connected
here" were one observation: no row. Nothing above could tell a worker
re-homing between replicas from a worker that is gone, and any grace
period built on top would have had nothing to measure from.
The row now survives a departure. Release clears owner_instance_id and
stamps disconnected_at on the database clock; the membership sweep and
Deregister do the same for every connection a dead or departing replica
held; Claim clears the stamp in the same upsert that writes the owner, so
a reconnect is never observed half-applied. PurgeDepartedBefore deletes a
departure once it is older than DepartedRetention, and the membership
tick owns that schedule.
Owner and OwnerRow report a departed row as ErrNoConnection, through the
one predicate connectionIsHeld, the way instanceIsLive is the one
predicate for replica liveness. This change records the departure and
does not interpret it: how long ago it happened is nobody's answer yet.
The sweep only clears rows that are still held. An empty owner is in no
instance's id, so without that filter every heartbeat would restamp every
departed row and no departure could ever age out.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Web Chat Settings left the System Prompt field empty but still treated a
blank/whitespace value as an explicit system turn. That satisfied
tokenizer chat templates' messages[0].role == system check and suppressed
the model YAML system_prompt on fresh chats.
Omit empty/whitespace system messages in the React and Alpine UIs, strip
them server-side, and inject config.SystemPrompt for tokenizer-template
models when the request has no real system turn.
Fixes#11834
Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
* fix(distributed): evict only when a node is known to be full
scheduleNewModel asked the registry for a free replica slot and treated
every error as "this node is full", so a control-plane database slow
enough to time out the lookup evicted a healthy loaded model. The
evicted process died, a peer frontend still holding its address dialled
the dead port and retried, and the model thrashed between nodes. The
comment on the branch already said it meant a full node; the code never
tested for it.
Evict only on ErrNoFreeSlot. Any other error now returns and names the
lookup that failed, so a slow database degrades into a diagnosable
load failure instead of into lost work.
An audit of the rest of the router found one branch of the same shape:
node selection discarded the error from its last-resort finder, so a
database timeout there also produced a nil node and evicted for it.
That path now returns unless the finder said gorm.ErrRecordNotFound,
which is the only answer that means the cluster had no node to give.
No other destructive branch in router.go fires on a generic error.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): checkpoint heartbeat writes instead of writing every beat
Every heartbeat UPDATEd backend_nodes. Six nodes at a ten second beat is
roughly 52,000 writes a day against a six-row table, and that churn is
what turned a blocked autovacuum into a 460 MB table whose six-row scan
cost 867 ms and timed out the queries that place models.
A beat carrying only a fresher timestamp now waits for the checkpoint
interval. Each reported field is compared against the value last
persisted rather than tested for presence, because a worker sends its
disk figures on every beat and presence alone would suppress nothing.
A node's first beat, a changed total VRAM, total disk or GPU vendor,
and a free VRAM, RAM or disk reading that has moved more than 256 MiB
from the persisted value all still write at once. A node that is not
active is never suppressed, because it recovers only when the health
monitor sees a fresh timestamp.
The persisted column is up to one interval stale by design, so the
stale-node threshold moves from 60s to 5m to cover it.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): fail worker readiness when a held backend is unreachable
The readiness gate tracked only the NATS link, so a worker whose backend
processes had died still answered /readyz with 200 and kept receiving
loads. One node did exactly that during an incident: it reported healthy
while its backend port refused connections, and every load routed to it
failed.
Readiness is now the NATS link and, for each backend process the worker
believes it is running, a short dial of its recorded address. A worker
holding no backends stays ready, because idle is a healthy state.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): keep a starting backend out of the readiness dial set
A backend process is inserted into the supervisor map with its gRPC
address already recorded, but the address refuses connections until the
gRPC server binds, which the startup poll allows up to 30 seconds for and
which takes 10 to 15 seconds on a slow node. The new data-path readiness
probe dialled that address straight away, so a worker answered /readyz
with 503 for the whole of every cold backend start. The container
HEALTHCHECK absorbs that, but a Kubernetes readinessProbe at 10s does
not, and the worker would leave rotation each time it loaded a model.
The skip for a stopping process had no counterpart at the other end of
the lifecycle. Backend processes now carry a serving flag, set where the
startup health-check gate succeeds, and the probe dials only processes
that are serving and not yet stopping. backendStartStillValid becomes
markBackendServing: the check and the mark must share one lock hold, so
the flag can only ever land on the entry the key currently owns.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(distributed): export control-plane database health gauges
Four transactions wedged on a corrupt index held the vacuum horizon open
for 42 days. Nothing measured it, so the first symptom anyone saw was
models failing to load six weeks later, by which time a six-row table
had grown to 460 MB.
Export the oldest xmin age, the longest open transaction, and the dead
tuple ratio on the registry tables. The first is the number that would
have caught it: it sits near zero in health and was 21,002,291.
Sampling is scrape-driven behind a cache, and a failed sample reports
the last good values rather than failing the scrape, because these
gauges matter most when the database is already struggling.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): rate-limit failed control-plane database samples
The cache advanced its clock only on a successful sample, so once the
database started failing every scrape retried the query immediately.
That turned the cache off in the one regime it exists for: a retry
storm at scrape cadence aimed at a database already in trouble. A
catalog read that consistently exceeds the 5 second timeout also paid
that cost on every scrape, with all scrapes serialised behind the
sampler mutex.
Time every attempt rather than every success, so failures and timeouts
cost the same interval as good samples. Whether a good sample exists
moves to its own field, keeping the gauges absent until the first
success and holding the last good values through later failures.
Also note in the runbook that pg_stat_activity cannot see prepared
transactions or replication slot xmins, so a healthy-looking xmin age
does not by itself rule out a blocked horizon.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(distributed): pin that a failing database evicts nothing
Exercises the real distributed stack against a control-plane database that
refuses the router's slot lookup, and asserts the scheduler reports the
lookup it could not answer instead of falling through to eviction.
The failure is injected with privileges rather than a statement timeout. A
timeout set with ALTER DATABASE also breaks AutoMigrate, and it leaks into
every later spec in the suite unless it is reset, so the spec would end up
testing the migration rather than the scheduler. Instead the spec creates a
dedicated login role, points a second gorm handle at it, and revokes that
role's SELECT on node_models.replica_index. This has to be a separate role:
the test container's owner is a PostgreSQL superuser, and superusers bypass
every privilege check, so revoking from CURRENT_USER is recorded and then
ignored.
The revoke is scoped to one column on purpose. Revoking the whole table
would also blind node selection, which runs first and has a guard of its
own, so the scheduler would never reach the slot lookup this spec is about.
Leaving every other column readable lets selection succeed and lands the
refusal exactly on NextFreeReplicaIndex, which plucks replica_index. The
grant is restored from BeforeEach via DeferCleanup, so a failing assertion
or a panic cannot hand the next spec a role that cannot read.
Reverting the eviction guard fails this spec, which is the point of it: the
router then reports "no replica slot on keeper and eviction failed" for an
error that was never evidence the node was full. The surviving-row
assertions are secondary under this injection, because the eviction path
reads whole node_models rows and the same revoke blinds it too; a comment
in the spec says so, so nobody mistakes them for the load-bearing ones.
Also documents why the vector store and the control plane must not share a
database: the removable-tuple cutoff is per database, not per table, so one
transaction left open anywhere stops autovacuum reclaiming the node
registry, and a six-row table bloats into hundreds of megabytes. The note
names LOCALAI_AUTH_DATABASE_URL and LOCALAI_AGENT_POOL_DATABASE_URL as the
two knobs that must differ, and the localai_control_plane_oldest_xmin_age
gauge as the way to see it coming.
grep for StaleNodeThreshold and HealthCheckInterval in
core/config/runtime_settings_registry.go returns no matches: the
distributed duration knobs are not exposed as runtime settings, so the new
heartbeat checkpoint interval follows them and needs no registry entry.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): close the review gaps in the heartbeat and health path
The stale-node threshold moved from 60 seconds to 5 minutes in this branch
because checkpointing makes last_heartbeat up to one checkpoint interval
behind by design. Two things were left inconsistent with that. NewHealthMonitor
still fell back to a hardcoded 60 seconds when handed a zero threshold, so any
future caller that stopped passing the configured value would mark every
healthy, beating node offline on every cycle. And the threshold itself had a
flag-name constant but no AppOption, no CLI field and no env binding, so an
operator who widened --node-heartbeat-checkpoint had no way to widen the
threshold to match. The fallback now tracks config.DefaultStaleNodeThreshold,
and --stale-node-threshold / LOCALAI_STALE_NODE_THRESHOLD is wired the same
way its sibling is.
Heartbeat suppression compared the RAW reported free VRAM against the
snapshot, but the column persists capAvailable(raw, ceiling). On any node with
a VRAM budget set, whose actual free VRAM oscillates above that ceiling, every
beat looked material while the persisted value never moved: suppression was
defeated on exactly the nodes an operator had configured, and the write
amplification this branch exists to remove came straight back there. The
comparison and the snapshot now both hold the capped figure, so they measure
the same quantity as the column.
Fixing that needs the ceiling, and reading it cost a SELECT on every beat,
including suppressed ones. The skip decision therefore moved ahead of the
updates map and now reuses the ceiling cached on the last durable write, while
the write path still re-reads it before capping anything. A ceiling that
changed inside the checkpoint window can cost one extra or one late write; it
cannot persist a wrong figure. A suppressed beat now costs no query at all.
Also: the operations section now says to grant pg_read_all_stats to the
LocalAI role, because PostgreSQL blanks backend_xmin and xact_start for
sessions owned by other roles, and the transaction that wedged the horizon in
the incident was a co-located vector store connecting as a different role, so
without the grant the new gauge sees only our own sessions. The compose
healthcheck comment now describes readiness covering the backend data path,
and the control-plane gauge registration records the otel.SetMeterProvider
ordering it depends on.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): resolve the gauge's table names through gorm
The dead-tuple gauge queried pg_stat_user_tables against a hardcoded list
of three table names. Those three do not agree on where their name comes
from: BackendNode and NodeModel take gorm's default pluralisation, while
GalleryOperationRecord overrides TableName, and gallery_operations
already had a constant of its own that the list duplicated.
A literal list keeps compiling after any of that moves, and the query
then matches nothing. The failure is silent and it points the wrong way:
a dead-tuple ratio that matched no rows reports the same numbers as a
cluster with no bloat, so the gauge would look healthiest exactly when it
had stopped working.
Ask gorm what each model is stored as instead, which follows a TableName
override and the default pluralisation alike. A spec pins that the
override really is consulted: naive pluralisation of the type would give
gallery_operation_records, so the resolution cannot quietly stop asking
the model.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Clamp requested generation to the usable context after prompt sync while preserving the legacy 256-token fallback for omitted limits.
Constrain each speculative MTP cycle to the remaining request budget so accepted tokens cannot advance beyond the visible output limit.
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
The worker re-classified a failure a local service had already classified.
classifyServiceFailure preserved exactly one of the four refusal codes,
which was faithful to its own comment for as long as there was one worth
keeping; once ErrStreamNotServed existed, a service returning the code
whose whole job is to say "I learned nothing" had it promoted to
ErrStreamTargetUnavailable, which every reap guard acts on.
ErrStreamTagUnknown was promoted too, and cost nothing only because both
sides of that one reap. No in-tree service produces either, which is the
same "unreachable, therefore safe" argument that let the request-frame
merge survive a whole phase, and LocalService is exported.
The cause was a fifth site enumerating the vocabulary by hand, so the fix
is one table. streamRefusals pairs each sentinel with its wire code and
with whether a frontend may act on it as evidence about a backend, and the
writer, the reader, IsWorkerAnswer and the new IsStreamRefusal all read it.
A fifth code is now taught to every one of them at once.
The codes are also pinned against literals written out in a spec, the way
this branch already pinned the NATS vocabulary. The round-trip table
cannot see a rename, because a rename moves the writer and the reader
together; an unrecognised code is deliberately not the worker's answer, so
renaming "unavailable" would turn every crashed backend on a tunnelled
worker into a row nothing can ever reap, silently and with the suite green.
Three comments the previous fix falsified, corrected:
- tunnelHeaderTimeout still said the window bounds only framing the
frontend writes immediately after opening the stream. That is true on the
direct path and false on the relay path, and it was the argument for
treating an expiry as the frontend's fault.
- classifyServiceFailure's deny-list is three causes, not two: on a dial
error net.Error.Timeout also covers ETIMEDOUT and EAGAIN. Both are kept
deliberately, because reaping a wedged or resource-starved backend is the
eviction this phase exists to prevent, and ECONNREFUSED still reaps.
isReadTimeout is renamed reportsTimeout, which is what it asks.
- The operator table named three refusals and said a refusal is acted on.
It now lists four, with when each is sent and whether the row is reaped.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Making a worker's refusal reaping evidence created a defect one layer
along, at the producer. The worker refused a ReadStreamRequest failure with
ErrStreamRequestInvalid and its own comment said "Includes the deadline
above expiring", which was harmless while every refusal reached the
frontend as "no route" and became a reap the moment one of them did not. So
a request frame that had merely not ARRIVED yet was reported as a
non-transient verdict about a backend.
It is reachable on the relay path, which carries most production traffic:
the worker's header timer starts when the OWNING replica opens the stream,
while the frame is written by the DIALLING replica only after the relay's
acceptance travels back to it, so a whole peer-link round trip runs inside
that window, on a link this design deliberately loads with multi-gigabyte
artifacts beside token streams. For a long-deadline caller the endpoint is
ConnectionEvictingClient, which stops the model across the fleet. It also
falsified the "neither clears on its own" argument that licensed the reap.
There is now a fourth refusal, ErrStreamNotServed, for what a worker could
not serve for a reason of its OWN. It is deliberately outside
IsWorkerAnswer, so it reaches a consumer under the no-route umbrella and
reaps nothing, which is the same treatment an unrecognised code already
gets. Four producers move onto it: a request frame that timed out (a
malformed one stays a verdict, because that is a frontend bug no retry
fixes), both SetReadDeadline failures, which are facts about the stream and
not about a target nothing has dialled yet, and WriteStreamRefusal's
default for a reason nobody classified.
classifyServiceFailure keeps ErrStreamTargetUnavailable as its default on
purpose: inverting it would make errno enumeration the single point of
failure for the reap, and a miss there is a row nothing can ever delete.
What it gains is a deny-list of two causes that are provably this worker's
own clock or its own context.
Also:
- The read-site caller-deadline guard in the handshake was unpinned: the
existing seam spends the budget before the handshake starts, so only the
write could ever fail. A spec whose deadline falls between the request and
the reply pins it, and each guard now reddens on its own.
- The documented worker-first failure line omitted the JSON error envelope
the old frontend returns, so an operator grepping it found nothing.
- The peer-link disclosure names the aimable per-session receive window in
all four places, and LastDialErrorOf records why a third consumer must go
through IsWorkerAnswer rather than roll its own list.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A worker that refuses a stream has answered, and cluster.Dial keeps the three
tunnelproto sentinels out of the ErrNoRoute umbrella precisely so a consumer
can act on that. No consumer did. Since workers stopped listening, a backend
process that crashed on a healthy worker is no longer a dead listener's
codes.Unavailable: the worker refuses the stream with
ErrStreamTargetUnavailable, gRPC flattens it into Unavailable anyway, and
nodes.unroutable reported the whole thing as "this frontend has no route".
Every reap path then answered ProbeUnknown and left the row, so the replica
slot never freed and at the default MaxReplicasPerModel=1 the only cleanup
left was LRU eviction of models that were working.
isWorkerAnswer is exported as cluster.IsWorkerAnswer, so the errors the dialer
keeps out of the umbrella are by construction the errors the consumers treat
as the worker answering. nodes.unroutable and pkg/model's transportFailure
both use it; ConnectionEvictingClient, the site reached during inference, goes
through transportFailure rather than asking the transport directly. A reply
code this frontend does not recognise is still not an answer, so a newer
worker's vocabulary costs a retry and not a replica.
The reap guards keep the allow-list rather than requiring ErrNoRoute: an
unrecognised dial error must mean "no route", never "the backend is gone".
Also in this final pass over the branch:
- Docs: recommend upgrading FRONTENDS first, with the symptom of each order.
Workers-first fails now that a 4xx registration is a verdict rather than an
outage, so an old frontend's "address is required for backend workers" makes
each restarted worker exit and drains the fleet a node per restart.
- Docs: LOCALAI_WORKER_TUNNEL=false is a fatal startup error, not a degraded
mode, in both places that described it; and a frontend rollback needs every
worker restarted, because re-registration force-clears the address columns.
- A replica with no advertised address now says so every five minutes and
names the workers only it can reach, instead of one startup warning for a
cost paid for the life of the process.
- callerRanOut's rule now holds at all three siblings, so an expired caller
deadline stops reading as a broken tunnel; probeHealth's withdrawn reason
for using the raw client is corrected; the dead DoOrCached is deleted and
its coverage kept on DoOrCachedResult; sweepLeakedInFlight enumerates the
outcomes that reach it.
- The peer route's self-declared id is recorded as a phase-3 deferral, in the
handler, in the isolation claim it narrows, and in the operator docs.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review round 1 on the end-to-end proof. Zero blocking items, eleven
non-blocking, and three of them turned out to be production defects rather
than notes on the report.
The one that matters is a misclassification the phase is built to prevent. A
dial carries the caller's deadline down to the socket, so when the budget runs
out the socket's timer fires and the error travels back up through the
WebSocket handshake and the multiplexer. The context's cancellation is a
separate timer whose func the scheduler has to run before ctx.Err() stops
returning nil, and nothing orders the two. Under contention the socket's error
is back in PeerPool.Open first, ctx.Err() reads nil, and a peer that is
listening and healthy is reported as ErrPeerUnreachable to a caller that simply
ran out of time. An unreachable peer is a fact a caller may act on and an
expired deadline is not, and core/services/nodes routes around a replica it is
told is unreachable.
callerRanOut answers that question in one place: ctx.Err() when it is set, and
otherwise the wall clock against the caller's own deadline. That is sound
because it is the same instant the socket compared itself against, so if the
socket's timer fired this comparison is past it too. The ambiguous instant
resolves towards the caller, which is the direction that never blames a peer.
The spec that caught it, peerlink_test.go's "blames the caller's deadline",
was red in three of seven -race runs and had been since Task 5, which is often
enough to read as noise and is why single-run verification never saw it. Rather
than leave the proof to a coin flip, a second spec makes the window
deterministic: Open is handed a context whose deadline has passed and whose
cancellation has not been delivered, against an address nothing is listening
on, so the dial fails for real. It reddens without the fix.
The peer link's yamux windows were applied to one end only. A receive window is
advertised by the side that RECEIVES, so configuring the dialler alone tunes
exactly one direction, and the direction left on the 256 KiB default is the one
that carries a relayed model artifact INTO the replica that owns the worker's
tunnel. That is the largest thing the link ever moves and it is the direction
the load measurement exercises: the review read it as flowing toward the
dialler and it does not. PeerLinkConfig is now exported and used on both ends.
Measured, same box, 128 MiB staged through the relay against the same transfer
without one: the relayed path cost 1.6x to 2.0x the direct path's transfer
window before, and 1.06x to 1.25x after.
The SSRF reachability spec could be fooled into reporting an SSRF that did not
happen. It bound the victim on 127.0.0.2 at an ephemeral port and required
127.0.0.1 at the same port to refuse, so any other spec in the run holding that
number made the dial succeed; red one run in seven, green five of five in
isolation. It now picks from below the kernel's ephemeral range, the same fix
the harness got for the adjacent-port collision.
The rest are the specs and the report saying what they mean.
Scenario 1's advertisement assertion could not tell "the worker advertises
nothing" from "the JSON key moved", which matters because removing the
advertisement is the change it covers. It was green against a renamed key. The
roster now keeps the raw key set beside the decoded fields and the spec
requires both keys present before reading them as empty.
Scenario 4's refusal-body check was a four-way disjunction admitting bare
"tunnel", "not connected" and "unroutable". Those alternatives were inert and
each would be satisfied by refusals that say nothing about routing, in the one
assertion the whole negative control rests on. It is "no route" alone.
The head-of-line gate bounded the worst probe by the whole transfer window,
which admits about eightfold degradation and loosens as the box slows. It is
now half the window, plus a scale-free ratio against the worst probe under the
SAME cold load with nothing to transfer, which is the control that isolates the
transfer from the load. Not tighter than that, and the reason is measured
rather than cautious: under a concurrent -race suite the worst relayed probe
reached a fifth of its window, so a quarter-window gate would have had 1.2x of
margin, and a spec that fails one run in three is worse than no spec.
The report entry printed p90 and p99 off samples of twenty, where both land on
the same element and p99 often lands on the max, so one number appeared three
times under three names. A quantile is now printed only when the sample can
separate it.
Two claims in the report were wrong and are withdrawn rather than softened.
Scenario 2's race is closed by the trailing re-read of the owner, not by the
pre-assertion the report credited: a move to the non-owner mid-request would
serve directly and still return 200, and only the trailing read reddens on it.
And "the median request is unchanged" holds on this box and not on the
reviewer's, where the relayed median rises up to 82% and p99 up to 3.5x. What
survives on both is structural: the worst probe is a small fraction of the
window in which bytes are moving, so the session interleaves rather than
serialising. Sharing a session with a bulk transfer costs latency; it does not
cost service.
The disk footprint note undercounted, and the reviewer lost a run to a full
disk on this box, so it is worth having right: two bulk models seeded into two
frontends and staged to the worker is about 768 MiB, not 512 MiB.
Left alone deliberately: the worker's backend port allocator still hands out
ports without checking they are free, and its default range still overlaps the
kernel's ephemeral range. It is confirmed, it is out of scope here, and it is
being tracked as a named follow-up rather than fixed under an e2e task.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): link librdma from the static ggml-rpc build
ggml-rpc gained an Apple RDMA transport in this llama.cpp range and
declares its librdma dependency with target_link_options(ggml-rpc
PRIVATE "LINKER:-weak_library,..."). Link options are not a usage
requirement of a static library, so the llama-cpp-grpc variant, which
builds with BUILD_SHARED_LIBS=OFF, dropped the flag and left every
ibv_* symbol of transport-apple.cpp undefined when grpc-server linked
on darwin.
prepare.sh now re-declares the same weak link as INTERFACE on the
ggml-rpc target, so the flag reaches whoever links the static library.
The append is guarded on a marker for repeat runs, and on
GGML_RPC_RDMA_APPLE, which the turboquant and bonsai forks lack.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Everything this phase built was proven by unit and integration specs. This is
the first run of it against the real binaries: a frontend replica per process,
a worker that binds nothing routable, real inference over the result.
Four scenarios, each with the question "what would make this pass if the tunnel
were doing nothing" answered rather than left open.
A worker with no advertised address is reached through its tunnel. The roster
is asserted to report it advertising nothing, so there is no address a frontend
could have dialled instead, and node_connections is asserted to name the
replica that serves the request.
A request landing on the replica that does NOT own the worker is relayed to the
one that does. With N replicas behind round robin that is (N-1)/N of production
traffic, so it gets the FIRST request for its model: the backend install, the
file staging on the http tag, and the gRPC load and predict all cross the
relay. Which replica owns the tunnel is read from the ownership table through
the production Owner query and mapped to a frontend index through the address
the harness pins per replica; the non-owner is derived from that reading and
asserted to be a non-owner immediately before the request, rather than assumed
from the harness default. Sending the same request to the owner reddens it.
Killing the owning replica re-homes the worker onto the survivor. The worker
dials a balancer rather than a replica, because LOCALAI_REGISTER_TO is resolved
once at boot and is the tunnel endpoint as well as the registration one: aimed
at a single replica, a worker has nowhere to reconnect to when that replica
dies, and the re-home cannot happen at all. Removing the kill reddens it.
And the negative control for the whole suite, which is why the other three mean
anything. Frontend and worker share a host here, so every backend port the
frontend names in a stream target is one it could have dialled directly; if it
did, the first three would pass with the tunnel inert. LOCALAI_WORKER_TUNNEL is
no longer usable for this, because it is a fatal startup error and a worker that
never started says nothing about a worker reachable some other way. The balancer
answers the tunnel connect path itself instead, leaving a worker that registers,
heartbeats, reports healthy and holds no tunnel. It is asserted to have dialled
and been refused, asserted to be held by nobody, and then asserted unreachable
with the refusal naming the missing route. Then the block is lifted, nothing
else changes, and the same request succeeds: that is what attributes the refusal
to the tunnel rather than to any of the ordinary reasons an e2e inference fails.
The fifth spec measures the head-of-line blocking this phase deferred three
times. 128 MiB crosses the session while a warm model is probed back to back,
direct and relayed. Median latency is unchanged, the worst probe is about 3x the
baseline median and about a seventeenth of the transfer window, and the transfer
runs at 415-490 MB/s direct and 222-268 MB/s relayed. A session that
head-of-line blocked would park a probe for the length of the window. Leave the
yamux windows untuned; and note this is loopback, so it says the multiplexing
does not serialise and says nothing about a link with a bandwidth-delay product.
The load spec is measured against a control that the first version did not have.
It passed with the bulk artifact cut to 4 KiB, because the window it read probes
against was mostly cold-load overhead: it would have reported a clean bill on a
session carrying no large message. The same cold load now runs twice, once
empty and once bulk, and the difference between the windows is asserted to be
real before any latency is read from it.
Two defects on the base commit came out of this.
cluster_peerlink_test.go has been red since the relay landed, deterministically,
in isolation and in the suite. It asserted that an accepted peer stream is
refused at once, on the premise that phase 1 installs no relay. The relay
correctly waits fifteen seconds for a frame naming the worker, and the spec's
budget was five. It now writes a relay request for a node no replica holds and
asserts the refusal is ErrNotOwner and specifically not ErrNoConnection, which
is a stronger spec than the one it replaces and the only thing in the e2e suite
that exercises the relay's refusal path.
The harness handed a worker's own HTTP port to a backend process. It took two
ports from freeport and used one as the gRPC base and the other for the file
transfer server; freeport returns adjacent ports often, and the backend
allocator hands out base, base+1, base+2, so the second backend started on a
worker was regularly given the HTTP server's port and died with EADDRINUSE. No
spec had started two backends on one worker before, so it had never fired; the
load spec starts five and it failed about one run in three. Each worker now
reserves a contiguous bind-probed block laid out the way production lays it out,
below the kernel's ephemeral range, with LOCALAI_GRPC_MAX_PORT bounding the
allocator to it. The underlying production defect is not fixed here and is
recorded in the report: allocatePort never checks that a port is free, and its
default range overlaps the ephemeral range on every Linux box.
Constraint 6, whether distributed mode should now refuse to start without an
advertised address, is DEFERRED, and the comment and the docs that described the
cost were understating it. A replica with no advertised address writes no
instances row, and Owner joins a connection against a live instance, so a worker
whose tunnel lands there is unroutable from every OTHER replica while being
registered and healthy. Refusing to start would still be wrong, because the
deployments it would break are single-host ones with no peers to be unreachable
by, and telling those apart at startup is a design with its own specs. Both
places now say what actually happens.
Suite wall clock 592s for 15 specs, up from 502s for 10 of which 2 were red. The
CI budget of 20 minutes does not move.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review round 1 on the change that stopped workers listening. One blocking item
and seven notes.
LOCALAI_WORKER_TUNNEL=false was the blocking one, and the ruling was to make it
fatal rather than to correct the comment that still promised it fell back to the
advertised address. There is no fallback left: a worker on this branch
advertises nothing and binds only loopback, so turning the tunnel off leaves it
reachable by nothing while it registers, heartbeats and reports healthy, and the
scheduler keeps placing models on it. That is the worst available failure shape,
so a new Config.validateStartup refuses it before prefetch, registration and
NATS, while the worker is still invisible to the cluster. It absorbs the
pre-existing empty-registration-token check, which had the same shape and no
spec. The flag is kept rather than deleted so an operator who set it is told the
promise is gone instead of having the setting ignored, and the guard around
StartTunnel is removed, because a branch nothing can take reads as a supported
no-tunnel mode that does not exist.
The justification for erroring on an install that names no address was wrong,
and the review is right that this is the dangerous form of overclaiming, because
the conclusion holds and the mechanism does not. It said the resulting empty
target would be refused as an invalid stream and that the refusal would read as
the worker answering about its backend. Nothing in this repo branches on
cluster.ErrNoRoute, and nodes.unroutable treats any recorded dial error as
unroutable, so that refusal reaches every reap guard as ProbeUnknown and deletes
nothing. The site now stands on what holds, that an install naming no port
produced nothing routable and the failure belongs to the install rather than to
a later probe, and records the retracted claim so nobody re-derives it. This
retracts the same paragraph in the body of 1cf847f29.
The reviewer deleted the whole tryWarmPath unnamed-replica guard and the suite
stayed green, including the reservation release. It is specced now, and the
asymmetry the review asked about is decided at the site: the row stays, unlike
the sibling !alive branch which removes it. That branch has observed a backend
dead; this one has observed only that the row is unreadable, which says nothing
about whether a process is running, and the row is the last record that one
might be, since the acknowledged stop path refuses a stop whose ExpectedAddress
does not match and an empty one cannot be cleaned up through it either.
The cross-version wire claim rested on two struct tags nobody asserted:
renaming only the json keys survived mutation while the gorm column rename went
red through raw SQL. Both keys are pinned now, marshal and unmarshal, per
struct.
A worker-first upgrade showed the operator a status code and not the reason. The
registration client discarded the body, so "address is required for backend
workers" was read off the socket and thrown away, and the ladder then spent four
minutes on a verdict the frontend reached instantly. Refusals now quote the body
and carry ErrRegistrationRejected, and both the ladder and the credential
manager's Acquire stop on the first one. Acquire matters more than the ladder:
it is the default path and its bound is 100 attempts, not 10. 408 and 429 are
deliberately not refusals, since both are the frontend asking for the same
request again.
Also: the stale "not blocked by firewalls" troubleshooting line, which now names
the real cause and the knobs that move the port range; and the inert address
fields on the MCP Node DTO, which the Assistant was still being handed. The
review named http_address there and I removed address too, because it is inert
by the same argument and leaving one of a pair is arbitrary.
Five mutations, all red. Deleting the warm-path guard reddens four specs and
falsifying only its reservation release reddens one, so the two halves are
pinned separately. Renaming only the json keys reddens both wire suites.
Discarding the refusal body reddens two. Dropping the rejection classification
does not fail the suite, it hangs it, which is the operator-visible symptom, so
it is recorded red under a ginkgo timeout.
The verify list is now derived from the diff rather than from the brief, which
is what let the previous round ship a spec asserting 200 where the endpoint
returns 201: nine ginkgo suites, the e2e vet, route auth coverage, the leaf
check, build, the healthcheck shell suite and lint. The two jsx files have no
harness in this worktree and are recorded as the one unverified surface.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A worker now opens no listener on a routable interface and states no endpoint at
registration. Backend processes and the file-transfer server bind loopback, and
the frontend reaches both through the tunnel the worker dials. The bind address
is built from loopbackHost, the same constant the tunnel's grpc tag dials, so
"the worker binds where its tunnel dials" is one fact in one place rather than
two literals that can drift.
All three advertisement sites are closed, not one: the registration body,
RegisterNodeRequest, and the per-backend address in the install reply.
That third one was hiding a live bug. stopModelExact refuses a stop whose
ExpectedAddress does not match what the worker recorded for the process. The
worker recorded 127.0.0.1:port; handleBackendInstall reported advertiseHost:port;
the router stored the reported one and sent it straight back. On any worker whose
advertise host was not 127.0.0.1, every acknowledged model stop failed with an
address mismatch. Nothing caught it because the e2e harness set
LOCALAI_ADVERTISE_ADDR=127.0.0.1, which made the rewrite a no-op. Removing the
rewrite makes the two strings the same by construction.
The brief was wrong about two of the four functions it called dead.
effectiveBasePort is the base of the backend port allocator and resolveHTTPAddr
is the file server's bind address; deleting them would have deleted the port
allocator and the file server. Only the two advertise* helpers were dead, and
addr_test.go is rewritten rather than deleted, because the port arithmetic it
pinned still needs pinning.
NodeModel.Address survives with a narrowed meaning and is renamed
WorkerLocalAddress, along with the install reply field that feeds it. The
frontend still has to say WHICH backend process on a worker it means, and the
port in this string is how it says it: it travels as a stream target and the
worker dials its own loopback. The gorm column and the json key stay "address",
so neither a migration nor an API break rides along. Every fall-back to the
node's address is gone. installBackendOnNode now errors when a worker reports
success without naming one, because substituting the now-always-empty node
address would name an empty target, and the worker refuses that as an invalid
stream, which is classified as the worker answering about its backend. That is
the "a present worker reads as something it is not" class this phase forbids.
DistributedModelStore.Range had the same shape and was already wrong: it built
each remote model's client from the node's base gRPC port, never the port a
backend process listens on, so Free and Status went to the wrong place. It uses
the replica's address now.
BackendNode.Address and HTTPAddress are kept but made provably inert: no writer,
no reader that acts on them, and Register force-clears both on re-registration so
an upgraded worker's stale advertisement does not outlive its own upgrade in the
API and the Nodes page. Dropping the columns is a ~90-site edit across the specs,
the e2e suite, the MCP dto and the UI; it is recorded as a follow-up rather than
folded in here.
A persistent tunnel 401 still does not trigger re-registration, and now for a
reason rather than a deferral. Register CLEARS the node's replica rows, so
re-registering on a 401 would delete a live worker's rows on every retry, and
under the name collision that causes the 401 the two workers would take turns
doing it forever: a credential failure causing model reclamation. It also cannot
fix the named cause, since a collision is indistinguishable from a restart. The
401 log now names both causes and says nothing can reach this worker, which is
true only now that it has no listener.
The container healthcheck did not break the way the brief expected, since the
listener still exists on loopback and the probe runs inside the container. It did
have a real #10987 defect that this change makes the common case: it read
LOCALAI_SERVE_ADDR only, while effectiveBasePort reads LOCALAI_ADDR first, so a
worker on a non-default base port was probed on 50050 and reported unhealthy
while working. It follows the same precedence now.
Docs, the compose file and the e2e harness are updated in step: no inbound rule
or published port is needed for a worker, the two advertise variables are gone,
the remaining address variables are read for their port only, the
firewall-the-file-transfer-port warning is narrowed to the LOCALAI_HTTP_ADDR
opt-out, and the upgrade-order note no longer claims the worker still listens.
The Nodes page showed node.address, which is now always blank, so it shows the
node id instead.
Eight mutations, all red on a named spec, including reverting the loopback bind,
re-adding the address to the registration body, restoring both node-address
fall-backs, dropping the force-clear, storing the endpoint's address again, and
un-fixing the healthcheck. One of them caught a defect in a spec I had just
written: it asserted 200 where the endpoint returns 201, which went unnoticed
because core/http/endpoints/localai is not on the task's verify list. It is run
here.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
I reported that enabling gocritic pushed make lint past 600s. That was wrong,
and it was wrong in a way worth naming: those runs happened right after I
changed pkg/grpc's and core/services/nodes' interfaces, so the Go build cache
was cold for essentially the whole repository including every backend, and test
suites were running concurrently on the same machine. I attributed a cold-cache
full-repo typecheck under load to the linter I had just enabled, and raised it
as a cost without ever timing it against a baseline. A number with no control is
not a measurement.
Measured properly, with the golangci cache cleaned before every run and isolated
GOCACHE directories for the cold ones so the shared cache was not wiped: warm,
base 15s then 7s and current 8s then 7s; cold, base 87s and current 78s running
base first, base 136s and current 79s running current first. The spread between
the two cold base runs is larger than any gap between base and current, so
gocritic with only the ruleguard checker costs nothing measurable.
So the rule stays, unscoped. Scoping it to core and pkg was the fallback for a
cost that does not exist, and adding that configuration would buy nothing.
The one override gets the protection it needs instead. InFlightTrackingClient's
nolint is exactly the kind of thing a later reader tidies away, so it now opens
by saying not to, and states what breaks rather than what is intended:
WrappedBackend embeds the full Backend interface, so adopting it there would
promote every inference method as untracked pass-through, the build would stay
GREEN, and in-flight accounting would silently stop covering whatever was added
next. WrappedBackend's own doc carries the counterpart warning so a reader
arriving from either side finds it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Re-review round 2. One blocking item, and it was a spec I wrote: eight
goroutines raced at the probe cache and nothing made them coalesce, so a
straggler that missed the flight re-entered the probe and double-closed a
channel. It panicked about one run in three and took the four-suite race block
down. The green verification I reported was not reproducible, which means one
green run was never evidence for a spec that coordinates goroutines. Its comment
claimed the probe blocked until every goroutine was inside flight.Do, and that
gap was exactly the panic: the comment described the design intended rather than
the one written.
It is deterministic now rather than tolerant. singleflight.DoChan registers its
channel on an in-flight call under the group's own mutex and returns without
running its function, so calling it while the leader is provably parked inside
the probe joins that exact flight with no window and no dependence on the
scheduler. The spec asserts the join really happened, that the joiner got the
reason and not only the answer, and that the probe ran once; the entered channel
is sent on rather than closed so a second probe fails an assertion instead of
panicking. Twenty runs green under race against the committed code, five out of
five red on the mutation back to a closed-over variable.
The future-decorator gap is closed in the lint gate, but not the way the review
suggested, and the reason is worth recording. HasMethod rejects inline
signatures outright, its method-reference form needs a package ruleguard's own
typechecker can import and that typechecker cannot import this module, and
Implements tests the value method set while every Unwrap is on a pointer
receiver, so it fired on all three wrappers that already had one.
So the safe shape is structural instead. grpc.WrappedBackend gives the same
pass-through method set plus Unwrap on a value receiver, and a decorator that
embeds it is transparent by construction; forgetting stops being expressible
rather than merely discouraged, which is the move loopbackService already makes
in the worker. FileStagingClient and ConnectionEvictingClient embed it and their
hand-written Unwrap methods are gone. The ruleguard rule then only has to catch
the raw embedding, needs no type filter, and cannot misfire. It was verified to
fire on a throwaway wrapper and stay silent on a correct one, and reports
nothing across core and pkg with the baseline disabled.
InFlightTrackingClient is the one exception and says why in a nolint: it embeds
ControlBackend deliberately so that leaving an inference method unwrapped breaks
the build, and WrappedBackend embeds the full interface, so adopting it would
silently restore pass-through for every inference method and delete that
guarantee.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Re-review round 2. One blocking defect, and it was the concern I filed myself
last round and mis-scoped as a future trap. It was live, and it sat on the most
destructive reaping path of the five.
RouteResult.Client is an InFlightTrackingClient, over a FileStagingClient when a
stager is configured. model_router puts that on the cached remote model and
pkg/model's checkIsLoaded asks IT whether the transport failed. Both wrappers
embed grpc.Backend, which does not declare LastDialError, so the type assertion
read nil and the guard added last round fell straight through to the old
eviction. That eviction sends backend.stop over NATS to every node holding the
model and deletes every replica row, where the other sites delete one. The spec
covering it built a bare client by hand, which is why it passed while production
did not.
This is the third time in this task a correct fix was disarmed one layer out, so
the fix is a mechanism rather than two methods. BackendUnwrapper is one line per
decorator, LastDialErrorOf walks the chain, and both consumers now call it
instead of each keeping its own assertion. One implementation, no per-caller
policy to get wrong.
Sweeping every type that embeds or holds a grpc.Backend found a third decorator
the review had not named, and it is itself a reaping consumer of the same
collapsed signal. ConnectionEvictingClient is built for remote models in
initializers.go and its evict callback runs ShutdownModel; it fires during
INFERENCE rather than on a health check, so a tunnel blip mid-request was enough
to stop a model that was loaded and serving. It consults the transport first
now. A locally spawned backend has no custom transport, so that path is
unchanged byte for byte. Everything else touching a Backend is a consumer rather
than a decorator; there is no fourth.
The probe cache joiner shape is pinned. It was the right design last round with
nothing holding it: the mutation back to a closed-over variable passed all 602
specs in the package. Eight goroutines coalesced on a probe that blocks on a
channel now assert every joiner gets the leader's REASON and not just its
answer, which is the difference between a leader declining to reap and its seven
joiners reaping on the leader's own observation.
The LastDialError scope note claimed an exactness it does not have at
checkIsLoaded, which reads a shared long-lived client after releasing opMutex.
It now says which caller is not exact, why the imprecision is accepted there,
and what making it exact would cost.
The four-outcome table in the docs still said a worker with no live owner is
treated as absent and rescheduled, contradicting the code and the paragraph nine
lines below it. None of those outcomes is absence any more, and the table says
so, names the fifth, and points at the heartbeat as the thing that does decide
presence.
Five mutations, each reddening named specs, including the two the reviewer found
surviving.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review round 1 on task 6. Five blocking findings, all with the same root: the
conditions the dialer kept apart were erased one layer out, because every one of
them arrived at core/services/nodes as a gRPC codes.Unavailable, which is also
what a backend process that died produces. Four call sites acted on that by
deleting a replica row, one of them after a single failed probe.
The fifth condition is ErrNoRoute: this replica could not get a request to a
worker's backend, and no claim at all about the worker. A worker's presence is
its HEARTBEAT, which nodes owns; a route is a separate fact that cluster owns,
and the two now differ. They differ in normal operation, not exotically: a
worker that has not dialled its tunnel yet after a frontend-first upgrade is
unroutable on every request while it heartbeats and serves.
Two properties, both mutation-tested. Every failure to resolve or open a route
carries ErrNoRoute, so a consumer has one check to make. No failure carries an
absence sentinel: routeFailure is the single place that rule lives, and it keeps
ErrNoConnection and ErrInstanceNotFound in the message and out of the unwrap
chain, the guarantee unreachableError already made for peers. Everything else
stays matchable, so ErrNotOwner and ErrPeerUnreachable are unchanged for anyone
who can act on them. A worker's own refusal carries no umbrella, because a
worker that answers has demonstrated it is there and that is the only real
evidence on the path.
Crossing the boundary needed a value, not a code. NewClientWithDialer wraps the
dialer and records each outcome; LastDialError hands it back behind a narrow
interface, and nodes.unroutable turns it into ErrWorkerUnroutable with the
cluster sentinels still in the chain. A spec asserts a dial failing with
ErrNoRoute plus ErrPeerUnreachable arrives matching all three and matching
neither absence sentinel.
The sweep found a fourth site the review had not named: pkg/model checkIsLoaded
evicts a remote model on a connection error, and a tunnel dial failure is one.
Four other reap sites were cleared with reasons - inflight and the worker
authoritative pass reap only on semantic answers, scale-down is driven by
last_used, abandoned loads decide on the node's heartbeat. Every fixed site also
grew the opposite spec, so the new check cannot pass by never reaping.
probeCache carries the reason through singleflight rather than a closed-over
variable. A variable is only written by the goroutine that runs the probe, so
the leader would correctly decline to reap while every joiner reaped on the
leader's own observation; a mutation reproduces exactly that.
The docs sentence promising LOCALAI_WORKER_TUNNEL=false restores direct dialling
is gone. There is no such path, so it said the operator could take a worker dark
and call it a rollback. Replaced with the upgrade order that is actually safe.
The deadline spec the reviewer found vacuous now waits on the dial context's own
Done channel before touching the stream, so the armed deadline has really
expired; the mutation that survived for the reviewer reddens it.
Nine mutations, each reddening a named spec, including both halves of
isAbsenceClaim independently.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The tunnel, the fence, the registry and the relay were all built and none of
them carried a byte: every dial from the frontend still went to the address a
worker registered. This is where that stops. One WorkerDialer resolves where a
worker's tunnel is held, opens a stream on it locally or relays through the
owning replica, and hands back a conn past both handshakes; gRPC, the file
stager's HTTP client and the log-streaming WebSocket are all pointed at it.
A worker's address stops being somewhere to connect to and becomes the name of
which backend process a stream is for. It still appears in URLs, logs and
errors, because that is what identifies the process; what it no longer decides
is where the bytes go.
Nothing falls back to dialling it. BackendClientFactory now has exactly one
method, NewClientForNode, and returns an error where there is no way to reach
the worker. The direct-dial constructor was removed rather than kept beside it,
because leaving one on the interface keeps the bypass one word away from every
call site that holds an address, which is all of them.
The second construction path is closed too. DistributedModelStore built remote
models with a nil client, and pkg/model.Model.GRPC then dialled the raw address
lazily on first use - reached in production by ShutdownModel's Free and by the
backend monitor's Status. Those models now carry the tunnel-backed client, and
a model that cannot be given one is logged and not listed.
Four conditions stay unmixable, and one path produces absence: the dialer
answers ErrNoConnection only where Owner's liveness join did. A peer that will
not answer, a stale ownership row, a worker's own refusal and a missing relay
path are each reported as themselves. This matters because nodes ACTS on
absence, and the collapse would have it reclaim the models of a worker that is
connected and busy.
That is not hypothetical. Writing the mutation for it exposed the bug in this
change's own first draft: probeHealth returned bare false when it could not
build a client, and tryWarmPath deletes the replica row on a false probe. A
frontend whose dialer broke would have emptied node_models for the whole
deployment while every model kept running. probeHealth now returns alive and
probed separately, the reconciler gets a ProbeUnknown outcome that neither
advances nor clears a failure streak, and the health monitor skips rather than
counting a miss.
Task 5 left the relay's open timeout at a fixed 15s and said so: no operator
has the information to set it, because the number that matters is the original
client's remaining budget, which is invisible on the relay side. The dialer has
that budget, so it now states it in the relay request frame and the owner takes
the smaller of the two. It can only shorten - a patient client must not be able
to park a relay goroutine and a stream slot on a worker that stopped accepting.
Zero is written as no budget at all, since on the far side the number zero is a
caller with nothing left and would refuse healthy traffic.
Seven mutations, each reddening a named spec: peer-unreachable as absence; the
local-failure guard dropped; max instead of min on the budget; the nil-client
model restored; ProbeUnknown falling through to the reaper; OwnerRow instead of
Owner; probed collapsed into alive. The first budget spec passed for the wrong
reason - a handshake deadline, not the relay - and was replaced by three that
each assert one link, including one where the spec plays the owning replica and
reads the budget out of the frame instead of inferring it from a clock.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review round 1 on task 5. Eight non-blocking items, all addressed.
The classifier read Remote in two predicates with a report-by-default
fallthrough behind them, so reverting either read left the whole suite green:
the error reached the same answer down the other path. A correctness argument
that rests on mutation evidence cannot afford a shape that cannot be mutated in
pieces, so the two predicates collapse into one muxVerdict deciding each error
type once. Falsifying either Remote read now reddens exactly one spec.
Three claims the comments made loudly and nothing tested:
- clearing the header read deadline before the splice. Deleting the clear
left all 49 focused specs green, while in production it is the difference
between a relayed response that streams for an hour and one that dies after
fifteen seconds of quiet;
- the open budget bounding the open and nothing after it;
- closing the worker-side stream when the acceptance reply cannot be
written, which leaks one stream on the worker per failure.
All three are pinned now. The first two share a spec that sets both budgets to
50ms and then watches the conversation outlive them by ten times, which is an
assertion about an event that must not happen and so is the one wait a channel
cannot replace. The third drives the relay with a peer stream that delivers a
request and then fails every write, because no pair of live yamux sessions can
be made to fail that write on cue.
The disjoint-vocabulary argument was specced for the accepted frame only. Both
refusal directions are covered now, and asserted as "not one of the other hop's
sentinels" rather than merely "an error", since reading a relay refusal with the
tunnel's reader always errors and the question is whether it errors as the wrong
thing.
The open budget stays non-configurable, and says so: the number that matters is
how long the original client will wait, which is not known on this side and is
not something a deployment-wide constant can stand in for. The honest fix is the
caller's remaining budget travelling in the request frame, which belongs to the
dialler that has the budget.
Two comments corrected: nothing deadlines the peer stream after the clear, so
the tunnelled protocol's own deadlines cannot be what justifies clearing it; and
the membership sweep deletes departed replicas but reports only how many, so
identifying them is work that would have to be done, not knowledge waiting to be
plumbed. Recorded at muxVerdict: a remote RST that does not ride a
typeWindowUpdate frame yields the bare sentinel and is still silenced, which is
unreachable between two go-yamux peers but keeps the new rule from reading as
unconditional.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
A worker holds ONE tunnel and it lands on ONE frontend replica, so with N
replicas behind a load balancer roughly (N-1)/N of requests arrive somewhere
that cannot reach the worker directly. This is the piece that carries them:
the SessionStore stream handler reads which worker a peer's stream is for,
opens a stream on the tunnel this replica holds, and splices the two.
Splice has had no production caller since phase 1. It has one now, and being
the first caller it settles the two endings phase 1 deliberately left open,
both of which read as normal termination until now:
- a peer-initiated *StreamError{Remote: true}, which yamux builds only from
an RST frame the far side sent (stream.go:432-449); a reset this side asks
for carries Remote: false, and Splice never resets anything, its own Close
sending a FIN;
- a graceful ErrRemoteGoAway, which handleGoAway returns for code
goAwayNormal (session.go:829-833) and close hands unwrapped to every live
stream (session.go:328-337).
Both truncate whatever was in flight. Reporting them as normal termination is
how a half-finished inference comes to look like a short one that completed,
so both are now reported; the local forms stay silent, because those are the
teardown Splice provokes itself. The decision cannot live in a caller reading
Splice's result, since a result already mapped to nil carries nothing left to
reclassify, so it lives at the classifier with the reasoning beside it. The
relay logs it at debug: a client cancelling a relayed request produces one per
cancellation, and the truncation is separately visible to the frontend's own
gRPC or HTTP client.
The relay hop gets its own request and reply frames. They have to be distinct
from the worker tunnel's, because a relayed stream carries both hops' frames
back to back, and a vocabulary shared between them would let a reader applied
to the wrong hop hand back a plausible sentinel belonging to the other. Its
three refusals stay apart for the reason the worker's three do: ErrNotOwner is
a routing fact and the caller should resolve the owner again; unavailable is
infrastructure at this replica and a retry is worth something; bad-request is
the caller's bug. None of them is, or may be built over, an absence error.
One hop, always. A stream naming a worker this replica does not hold is
refused, never resolved and relayed onward, so a stale ownership row cannot
become a loop between two replicas each certain the other holds the worker.
PeerPool is constructed and closed alongside SessionStore, so both halves of
the peer mesh now have an owner and a shutdown.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Re-review follow-up, three items. Two are the overclaiming-comment class again,
and the first is that class with a real defect underneath it.
attachTunnelToken said "enforcement is therefore structural": an ineligible node
never gets a credential, so its hash stays empty and the tunnel route's
empty-hash branch does the refusing. That was true for a node that had always
been an agent and false for one that had not. Register upserts by NAME, so a
backend node re-registering as an agent keeps its ID, and Register's struct
Updates zero-skips the credential column while writing the new node_type. The
early return left the credential the node earned as a backend sitting on a row
that is now an agent, and ConnectHandler never looks at node_type.
Fixed by making the claim true rather than by softening it, because the mint-site
gate was chosen precisely on the grounds that it was structural: an ineligible
node now has its column CLEARED, unconditionally, so the invariant does not
depend on what the row happened to contain. A spec pins it and was red before the
change. Same shape as the Register-upserts-by-name hazard already carried
forward: a name is not an identity.
Second, loopbackHost claimed to be the only host any tunnel stream is ever
dialled on. It is not: fixedService dials whatever Run built it from, which is
this worker's own LOCALAI_HTTP_ADDR, and loopbackAddr rewrites only a wildcard
bind, so an operator who binds the file-transfer server to a routable address
gets a routable dial. The property that matters is narrower and is what the
comment says now: the frontend cannot STEER the dial. The grpc tag builds its
address from a constant and a validated port with nothing from the wire reaching
the dialler, and the http tag ignores its target entirely. Worth stating exactly
rather than summarising, because the argument about what a stream can reach rests
on knowing which hosts are reachable, and an overstatement at that site is what
would let someone conclude the constant alone is doing the work.
Third, a spec named "without allocating it" measured no allocation. It now
asserts the mechanism the defence actually rests on, that the reader consumes the
two length bytes and not one byte of the body, through a counting reader. The
input carries a body on purpose: against input that ends after the header the
assertion would pass with the limit check deleted.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
Review follow-up. One blocking finding and seven others.
The blocking one first, and it is this project's recurring shape: the untested
path. loopbackService is the function whose comment calls the discarded host
"the security property this function exists for", and nothing tested it. The
reviewer replaced its body with a dial of whatever the frontend named, no port
range, and all 131 specs passed. Every spec installed the permissive test
dialler, so the real routing table was exercised nowhere.
It now has specs, and the property is stated as reachability rather than as a
property of the code: a listener on 127.0.0.2 that only the frontend's target
names must NOT be reached. Plus the port-range table, fixedService, loopbackAddr,
tunnelEndpoint, and the table itself, which moved out of Run into tunnelServices
so it can be built without starting a worker. One spec drives a real stream
through that table over the wire, so the routing rules are exercised end to end
at least once rather than only in isolation. The reviewer's mutation now reddens
ten specs, and six narrower ones redden between two and four each, so no spec is
riding on another.
The shape changed too, not only the coverage. The dial address is built from a
loopbackHost constant and strconv.Itoa of a validated int, so nothing derived
from the wire reaches DialContext at all: restoring the hole takes ADDING a data
flow, not deleting a check.
And a taxonomy fix found while specifying it. A port outside this worker's
allocator range was reported as unavailable, which tells a frontend to retry
something that can never work. It is a bad request now, and a backend that is
merely not listening yet stays unavailable, which is the retryable one.
Agent nodes no longer get a tunnel credential. Nothing dials into an agent
worker, so a tunnel replaces nothing for it and no client would open one, and
the gate is at the mint site rather than in the handler: with no credential
minted the hash stays empty and the existing empty-hash refusal covers it, so
enforcement is structural.
Two comments and one doc paragraph said an anonymous registrant gets a "working"
credential. With auto-approve off the node is pending and the credential is
inert, which is the distinction this same change argues three files away to
justify minting for pending nodes at all.
A refusal reason over the frame limit was cut on a byte boundary and could split
a rune. It cuts on a rune boundary now, and the code survives truncation, which
is what keeps a refusal classifiable.
Also: the pending-node spec asserted only that a credential was non-empty, so a
credential derived from the shared token passed it; it now pins per-node-ness the
way the headline spec does. The tunnel handler's citations into nodes.go were
stale before this branch landed, having been written against a file the same
commit was editing, and are by function name now. The static-NATS path says
plainly that an externally forced rotation locks it out until restart, and where
that gets fixed. tunnelproto gained direct specs, including that a read failure
is never reported as a refusal.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
The worker end of the tunnel. It dials wss://<register-to>/api/cluster/connect,
holds one yamux session as the CLIENT, and serves every stream the frontend
opens on it. Nothing dials into the worker, which is the point: no inbound port,
no reachable address.
Each stream opens with a length-prefixed frame naming a tag and a target, and
the worker answers before either side speaks the tunnelled protocol. The reply
is sent on every stream, not only on refusal, because the protocols carried here
are client-speaks-first and a reply sent only sometimes would arrive interleaved
with a response body. Two tags today: grpc reaches a backend process, and only
on 127.0.0.1 within this worker's own backend port range, because a tunnel
terminates inside the worker and letting the frontend name a host would make
every worker a proxy into its own LAN; http reaches the worker's file-transfer
server, whose address the frontend is not asked about.
An unknown tag, an unreachable local service and an unparseable request are
three refusals and stay three on the wire. A frontend gives up on the first and
retries the second. Each is answered AND the stream is ended: a worker that says
why and leaves the stream open has parked the caller on a request nobody will
answer, and a deadline on the far side cannot tell that from a slow worker. The
specs assert the stream ends rather than that an error occurred, which is what
phase 1 shipped in three places and held in none.
Reconnects double from 500ms to a 30s ceiling, each wait drawn between half the
interval and all of it, and the interval returns to its floor only after a
session that LASTED. Resetting on connect is how a rolling restart, where every
dial succeeds and dies moments later, becomes a retry storm against the first
replica back up. Nothing is assumed to survive a reconnect: the credential is
read at dial time, never captured.
And the credential is now real. The tunnel endpoint advertised authenticating a
worker against its own secret, but registration stored the hash of the shared
registration token, so a leak plus a known node ID still opened a tunnel.
Registration now mints a per-node secret, returns the plaintext once as
tunnel_token, and stores only its SHA-256 in a new column; the endpoint compares
against that and does not fall back to the old one. Rotating on every
registration follows from storing only the hash, since a re-registering worker
cannot be told the secret it already holds; its live tunnel is unaffected,
because the credential is checked when a tunnel is dialled and never again.
Unlike the agent API key and the NATS JWT next to it, the credential IS issued
to a node awaiting approval: the tunnel route re-reads the node's status on
every dial and refuses a pending one, so it is inert until an admin acts, and
withholding it would strand every worker that registers exactly once.
A node that has not registered since this change cannot tunnel, and the column
cannot be back-filled because the plaintext only ever existed in the response
that minted it. The boot warning that said tunnels need LOCALAI_REGISTRATION_TOKEN
is replaced: it was true while the tunnel authenticated against that token's
hash, and says the wrong thing now. What is still true, and is what it warns
about instead, is that without one, registration itself is unauthenticated.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The guard added last commit was circular. It cleared the maintenance database's
timeouts by executing SET statement_timeout = 0 on a connection that had already
inherited that database's bound, so the statement clearing the bound ran under
the bound it was clearing. Under the white-box spec's deliberate 1ms that gave
it 1ms, and it failed roughly once in fifty at 8-way concurrency with SQLSTATE
57014. The guard against invisible load-dependent flakes had become one.
The clearing is now delivered as a connection startup option, options=-c
statement_timeout=0 -c lock_timeout=0 on the maintenance DSN, so there is no
statement left to abort. Raising the imposed bound would only have bought
headroom and left the circularity in place. pgx puts every URL query parameter
into settings, options is absent from notRuntimeParams so it becomes a runtime
parameter, and runtime parameters are copied into the startup message
(pgconn/config.go:340-378, 606-617; pgconn/pgconn.go:382-388).
The spec now discriminates on pg_settings.reset_val, the value in force when the
connection started: 0 for a startup option, 1ms for a session SET. A first
attempt using a deliberately slow first statement did NOT discriminate, because
under the circular design the SET is itself the first statement, so by the time
a spec runs anything the session is already unbounded. Reinstating the circular
clearing now reddens the spec deterministically rather than intermittently.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Two advisory-lock specs named their database by literal, ALTER DATABASE testdb.
Once the test helper started handing every spec its own database on a shared
server, that statement landed on the maintenance database and did nothing to the
one the spec was holding, so both specs went green having never reproduced the
condition they exist for. They regress a model-load advisory-lock wedge that has
already shipped to production once, so the previous commit's de-flaking silently
disarmed a regression test for a real deployed bug.
Both sites now read the name back with current_database() and, more importantly,
assert the override actually landed before relying on it. A literal name can go
stale again; an assertion that the setting is in force cannot pass while it is
not. Removing either production override now fails the matching spec with the
real 55P03 and 57014 again.
That literal also meant every CREATE DATABASE and every DROP ... WITH (FORCE)
ran under the 300ms bound it set on the maintenance database, which is a new
load-dependent single-spec flake inside the change that was meant to remove one.
The helper's maintenance connections now pin one connection and clear both
timeouts on it, so no setting a spec makes can bound them, and a white-box spec
imposes the leak deliberately and proves it does not reach them.
Also pins the reclaimOne gate deferral the previous commit added without a test,
by panicking inside the re-claim's own claim statement, and drops the per-dial
empty-token log line to debug now that the boot warning says it once.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review follow-up. Twelve findings, none blocking, grouped here by what they
protect.
Panics. The handler now recovers between the WebSocket upgrade and the
hand-off, the way the peer link next door already did: net/http recovers the
panic but leaves the hijacked socket open, so without this a worker keeps a
session this replica has no entry for and will never detach. The claim gate in
Attach and reclaimOne is now released with defer, so a panic under Claim cannot
wedge one node's gate for the life of the process. SetTunnels gained the
nil-receiver guard its sibling Stop has.
Operability. A deployment with no registration token stores an empty token_hash
on every worker, so every tunnel dial 401s forever on a frontend that looks
correctly configured. That now warns at startup, logs its own line rather than
sharing the "wrong token" one, and is stated in the docs together with the fact
that setting the token later needs the workers to register again.
Authorization. A node still awaiting admin approval is refused with 403. The
rest of /api/node/ gates on nothing, but the two places that hand a node
something durable, its API key and its NATS credential, both refuse a pending
one, and a tunnel is that kind of grant. Draining and unhealthy nodes keep
their tunnels on purpose.
Comments that claimed more than the code. The global auth middleware does run
on this path and then declines to reject; the future per-node secret only lands
without a change here if it lands in TokenHash; the empty-hash guard is
defensive rather than deciding; ClusterPathPrefix is no longer only
replica-to-replica; the docs no longer say a reaped replica re-claims
unconditionally.
And the test harness. SetupTestDB started a PostgreSQL container per BeforeEach
with a readiness deadline it asserted on, which is one chance per spec to fail
one spec inside its setup, anywhere, never twice in the same place: the shape of
the flake seen twice here and never reproduced. It now starts one container per
process and creates a database per call, which is the pattern tests/e2e already
proved. Isolation is unchanged and is now asserted for the first time. All 69
call sites are untouched; the eleven consumer packages run 1404 specs green, and
jobs went from 34.3s to 3.3s, agents from 13.8s to 1.9s, cluster from 97.4s to
37.5s.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
A worker needs no inbound port: it dials GET /api/cluster/connect, the
connection becomes one multiplexed yamux session, and the frontend opens a
stream on it per request. This adds the endpoint that accepts that dial and
attaches it to the tunnel registry.
The dial is authenticated against the NODE's own stored token hash rather than
the deployment's registration token. That is the mechanism, not yet the
isolation, since a worker still registers by presenting the shared token; what
it rules out is the shortcut of comparing against the configured value, which
would have to be unpicked the day workers get their own secrets.
Every refusal happens BEFORE the WebSocket upgrade, so a dialer reads an HTTP
status rather than a handshake error. The route is registered in every
deployment, single-binary ones included, which is what puts it in front of the
route-coverage test that holds that rule in place; with no node registry it
refuses every dial, and tells a credentialed one the frontend has no cluster
rather than that its token is wrong.
A lookup that FAILED is answered as a failure. Reporting a database that could
not be read as "unauthorized" would send a worker re-registering, throwing away
the identity its tunnel and loaded models are keyed by.
Wires the tunnel registry in core/application/distributed.go and hands it to
the membership loop. Without that call the re-claim after a replica is reaped
had no production caller and could never run.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
The gate is justified by being held for one claim round trip, and Attach
held it across the close of the session it superseded. Closing a yamux
session closes the underlying conn and then waits for both its send and
recv loops to exit, and the send loop can be inside a write bounded only
by ConnectionWriteTimeout, so that is a wait on other goroutines. It must
not stand between a worker re-dialling this node and its claim.
The gate is now released after the store and before the close, which also
makes Attach match reclaimOne, where it has always been released
explicitly on every path. This is safe because a superseded session is no
longer reachable from the map by the time it is closed: the next re-dial
replaces an entry that already names the new session.
Pin the re-claim half of the gate too. A worker that re-dials between a
re-claim's commit and its record leaves the row carrying the re-dial's
epoch while the entry carries the re-claim's, so the attachment holding
the socket releases an epoch the row does not have and the row outlives
it, with nothing to sweep it while this replica is alive. Only Attach's
half of the serialisation was asserted; keying the two apart left every
spec green.
Also take the test hook's action under the lock that guards whether it
has fired. It was written from the spec's goroutine and read from
whichever goroutine issued the statement, which is a race in the harness
that pins the serialisation specs.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Two Attach calls for one node both claim, and PostgreSQL serialises the
two upserts, but nothing ordered the two map writes against the two
commits. The entry left installed could be the one whose claim lost the
row, and its Detach then released an epoch the row does not carry, so the
release matched nothing and the row survived the socket.
Nothing swept that row. This replica is alive and heartbeating, so
ReapStale leaves its rows alone, and no reconnect is coming for a worker
that has gone. Owner kept naming this replica as the live owner of a
tunnel it no longer held, and every dialer sent here was answered
ErrNotOwner, which is the relay into a replica that cannot serve the
request that this phase exists to prevent.
Claims for one node now pass through a gate, so claim and record are
indivisible. It is per node rather than one lock over the registry, the
way PeerPool locks per peer: the claim is a database round trip, and a
slow one for a single worker must not hold up Open for every other.
Detach is not gated, because it takes no context and must never park
behind an in-flight database call, and it changes no epoch.
Reclaim takes the same gate, which makes its claim the newest one for
that node, so it records the epoch on whatever attachment is installed
rather than only on the one it listed. Refusing to record onto an
attachment that replaced the listed one would leave that row with nothing
able to release it. The interleave the gate does not cover is Detach, and
a claim whose attachment detached while it was in flight is now released
again rather than left behind.
Also: restore Start's doc comment, which SetTunnels had swallowed; keep
reaping other replicas when this one fails to rebuild its own row, rather
than skipping the sweep along with the re-claim; scope the comment about
an unnoticed dead socket to the keepalive of the session whoever accepted
the tunnel built, since the worker session config does not exist yet; and
pin the sortedness of Held, the nil-session refusal, and both re-claim
interleaves with specs.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Phase 1 left the connection fence with a table and no sockets behind it.
This adds the registry that holds them: Attach claims the node and then
stores the session, Open hands out a stream over the tunnel this replica
holds, Detach releases the claim it was handed, and Held names what this
process is carrying.
The claim is written before the session is stored. A claimant that
installs itself and only then finds it cannot claim has, for that window,
published a tunnel no row records, so Held names it while a peer asking
Owner is told the worker is connected nowhere.
ErrNotOwner is produced at one place, the map miss. It is a routing fact:
some other replica may hold that worker perfectly well. A broken socket
under a held entry is returned as itself, because answering "not held
here" would send a dialer looking elsewhere for a worker this replica is
holding.
Epochs are compared for equality and never ordered. Claim guarantees an
epoch is unique and never reissued; it does not guarantee the later claim
draws the larger number, because the sequence value on the insert path is
drawn before the row lock.
The membership loop now re-claims on re-register, which closes the hole
phase 1 named in ReapStale. A replica that stalls long enough is swept by
a peer, losing its instance row and, in the same transaction, every
connection it owned; Register rebuilds the instance row and nothing else,
so without this it serves workers that every other replica reports as
connected nowhere. Re-claiming draws a fresh epoch, so an attachment
carries two: the token Attach handed back, which is what Detach matches
and which never moves, and the epoch of the row currently held, which is
what Release is given. Collapsing them would leave the re-claimed row
outliving the socket with no caller able to remove it.
A tunnel whose session is already closed is skipped rather than claimed
back, because claiming is an upsert and would take the row from whoever
holds the worker now.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review round 1 on the joined Owner read. The behaviour was accepted; three
comments claimed more than the code delivered, one spec pinned less than its doc
promised, and one pre-existing spec ranked epochs.
instanceIsLive said every reader of instance liveness uses it, which was false:
ReapStale spelled the complement by hand. The complement is now written as
NOT (instanceIsLive), so "stale" is exactly "not live", including how each side
treats a NULL last_seen, and the sentence is true. Inverting that predicate
fails 3 reaper specs, so the routing is held.
The Select("node_connections.*") in Owner was justified by a SELECT * hazard
that cannot occur: with a join present and nothing selected, gorm expands the
model's own columns table-qualified (callbacks.BuildQuerySQL), and the suite is
green with the Select removed. It stays, because the projection should be a
property of this query, and the comment now says that instead.
Owner gained the dialect guard Claim has. now() and make_interval are
PostgreSQL, so on the SQLite single-binary path it failed with "no such
function: now", which reads as a missing migration; that regression already
shipped once in phase 1. The refusal is deliberately not ErrNoConnection: a
deployment with no cluster has no answer about ownership, and reporting absence
would let a caller conclude the worker is not connected. A spec in the
non-PostgreSQL block holds all three properties.
The new specs aged rows by ten minutes, which any window between zero and ten
minutes satisfies, so nothing tied Owner's window to the one the sweeper uses.
They now age to just past InstanceLiveness, and a sibling ages to half of it and
must still resolve. Widening the window tenfold fails 2 specs, narrowing it
tenfold fails 1; before this both were silent.
The concurrent-claim spec asserted the stored epoch was the highest handed out,
and justified it with claims drawing their epoch after the row lock, which
contradicts Claim's own doc: the insert path draws nextval while the tuple is
built. It now asserts the stored epoch is one of the epochs handed out, and
ranks nothing.
OwnerRow's doc justified the function with a sweeper that does not call it.
ReapStale deletes orphans with a set difference; the callers are this package's
specs and one e2e assertion. It says that, and states plainly that a caller
needing to know who owns a node in order to dial it wants Owner.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ci): remove the e2e container before removing its image
`docker stop` returns as soon as the container exits, but the daemon
reaps a `--rm` container asynchronously after that. The `docker rmi
localai-tests` that follows teardown-e2e then loses the race against the
reaper and fails with "conflict: ... is using its referenced image", so
make exits 1 and the job goes red after every spec has passed.
This is why the E2E Backend Tests job fails at random across pull
requests. Runs 33435319093, 33435332991, 33412165884 and 33444669207 all
report "SUCCESS! -- 235 Passed | 0 Failed" and then die in teardown.
`docker rm -f` is synchronous, so the image reference is gone before
teardown-e2e returns. It also covers the case where no container is
running, which `docker stop` could not because it rejects an empty
argument list.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(ci): open a tmate session only when a PR asks for one
The tmate step runs on every failure and then holds the runner until
GitHub cancels the job at the 6 hour limit. A one second cleanup race in
the e2e teardown therefore costs a whole ubuntu-latest slot. The recent
run list is full of 6h, 7h and 12h cancelled runs for that reason.
The step now needs the `ci-debug` label on the pull request, so a
session opens when somebody wants to debug and never otherwise. The
30 minute step timeout caps the cost when the label is left behind.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
OwnerRow is a bare row read of node_connections. A connection row outlives the
replica that wrote it: a replica that dies stops heartbeating, but its rows
survive until a peer's sweep removes them, which is up to InstanceLiveness plus
one InstanceHeartbeat later. For that whole window the table names a process
that is gone. The next component phase 2 builds is the relaying dialer, and a
dialer reading OwnerRow would relay into a corpse for roughly 35 seconds after
every replica death, then report the worker as unreachable when it is in fact
absent, which is the distinction the phase 1 end-to-end specs pinned.
Owner is the resolving read: one statement joining instances, returning
ErrNoConnection when the row is missing OR its owner is not live. Both cases are
one answer on purpose, since both mean no replica here holds this tunnel; they
differ only in which sweep has run. It is one statement, not a row read followed
by an instance lookup, because between two statements the owner can die and the
caller would act on an owner the second read would have rejected.
OwnerRow stays, unjoined, for readers that need the row itself, and a spec holds
the two apart: with an aged-out owner, OwnerRow still names it and Owner
refuses, so neither can quietly become the other.
The liveness predicate is now one string, instanceIsLive, shared by Live and by
Owner's join. Two spellings of one fact drift, and this drift would show as a
relay to a replica one query calls dead and another calls alive. It is
table-qualified so it is unambiguous inside the join, and the cutoff stays on
the database clock, so replica clock skew cannot widen or narrow the window.
Both mutations were run. Dropping the liveness predicate from the join fails 3
specs, the aged-owner one among them. Replacing the database clock with a
Go-side time.Now() fails 1: the aged-owner specs still pass, because the two
clocks agree on one host, and only the recorded-SQL spec sees the literal
timestamp. That is why that spec exists.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
ReapStale deleted from instances then node_connections while Deregister took
them the other way round, both inside one transaction and both running
concurrently by design: a replica shuts down while a peer sweeps it. Opposite
orders let each hold the row the other waits for. PostgreSQL breaks the cycle by
aborting one side, so the cost today is a warning rather than lost data, but the
inversion costs nothing to remove.
Deregister now deletes the instance row first. That is the order ReapStale is
forced into anyway, since its connection delete asks which instance rows
survived, so the sweeper is the fixed side. Both functions say the order is
deliberate and shared, and name the other. A spec records the statements each
path issues and asserts they delete from the same two tables in the same order;
racing two transactions until they really deadlock would be flaky and could pass
for the wrong reason.
The rest is comment and spec accuracy, deferred from the phase 1 task reviews:
- co-location does not imply loopback. Compose's usual host=postgres resolves to
a bridge address and discovery works there; it is a DSN that NAMES localhost
that yields a loopback source address. Corrected in the DiscoverAdvertisedAddr
doc and in the spec comment that repeated it.
- unroutableReason labelled every scoped address "link-local", including the
class the check exists for, and formatted the IP with %s, which drops the
%iface, so the reported address was not the one being rejected. Split into two
cases, both rendered with their zone. CheckAdvertisedAddr passed zone "" and
net.ParseIP rejects fe80::1%eth0, so a scoped literal looked like a name and
collected no warning at all; the zone is now split off before parsing.
- Splice's "Both callers satisfy it" claimed callers that still do not exist.
It now names the two stream types the wake-on-Close property was verified
against and says a phase 2 caller over anything else has to check it.
- restored, short, why a socket-level ECONNRESET stays reported while a yamux
reset does not: the yamux endings are the teardown Splice's own Close
provokes, and whether an aborted request is routine is the relay's policy.
- the real-yamux spec's far.Read had no deadline, so a stall parked the suite
rather than failing it.
- gorilla's SetWriteDeadline is conn.go:796, not 787.
- ClusterPathPrefix is no longer derived from: the peer route spells its path
out, because core/services/cluster must not import core/http/auth. The comment
now points at the spec that holds them together instead of claiming a
derivation the move removed.
- the epoch spec asserted e2 > e1, an ordering Claim's doc tells callers not to
rely on. It asserts uniqueness, which is what the fence guarantees, and is
named for that. A sibling spec still described the epoch as incrementing in
SQL when it is drawn from a sequence.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review found the recurring class: assertions that a wrong implementation
also satisfies.
The "refuse promptly, never park the peer" guarantee was stated in three
places and tested in none. Removing the Close from the no-relay branch left
the whole cluster suite green, because the specs asserted only that some
error arrived and yamux reports a read deadline as ErrTimeout: a parked
stream satisfied that as well as a refused one. Both specs now require an
ENDING, EOF or a reset, inside a deadline short enough that parking is
unmistakable, and both go red when the Close is removed.
Deregistration existed only in a comment. Membership.Stop ended the loop and
left the row behind, so every clean rolling restart had peers dialling a
corpse for the full liveness window; the shutdown comment described the
opposite. Registry.Deregister deletes the row and the connections that
replica owned, in one transaction, for the reason the sweeper does both, and
an e2e spec pins departure inside a budget shorter than the liveness window
so it cannot pass on the sweeper doing the work. Before: the spec times out
with both replicas still live. After: 3.6s.
The configured advertised address bypassed every check discovery makes, so
the one value most likely to be copied between hosts, 127.0.0.1, was taken
verbatim and would make every peer dial itself. Both paths now share one
rejection rule: unparseable is refused, "this host" is warned about once and
honoured, because a single-host deployment uses it correctly.
Two comments claimed more than the code does. The sweeper said a stalled
replica recovers via re-register; only its instance row does, while the
connections another replica reaped stay gone and the sockets stay held here
- phase 2 must re-claim, on re-register, every connection a replica still
holds locally. And Owner became OwnerRow, documenting that the owner it
names may be dead for up to InstanceLiveness plus a heartbeat and that any
caller acting on it must join instances itself, so the deferred constraint
lives at the call site rather than in a report; the plain name is left free
for the joining version.
Minors: warn once when the peer link mounts with no registration token, so
an operator sees the cause rather than 401s; Stop no longer blocks forever
when Start was never called; corrected the NewRegistry migration doc and an
e2e comment that described a 6s window as "throughout".
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Tasks 1 to 5 built an instances table, a splice, both halves of a peer link
and an epoch fence, and nothing in the tree called any of it: no replica
registered, no route was mounted, no sweeper ran. Proving phase 1 end to
end therefore had to start by wiring it.
A frontend in distributed mode now publishes the address its peers dial,
heartbeats it, and sweeps replicas that stopped answering along with the
connection rows they owned, in one pass so the two can never disagree about
who is alive. It serves the peer link and owns the sessions peers dial in,
refusing streams on them until phase 2 installs a relay: a session nobody
accepts on does not fail a peer's Open, it hangs it.
The address is the one peers use, not the one the process binds, and it is
derived from the route to PostgreSQL. That derivation only holds while the
database is remote, so LOCALAI_DISTRIBUTED_ADVERTISE_ADDR sets it
explicitly and a replica that can determine neither warns and keeps
serving rather than failing to start.
Three e2e scenarios run against real local-ai processes, real PostgreSQL
and real dials: replicas publish addresses that can actually be connected
to; a sibling opens a stream over the peer link and is refused without the
cluster token; and a killed replica is reported unreachable, never absent,
loses the claim it held, and takes no worker with it. Each was verified by
mutation: eight injected defects, each failing the scenario that claims to
catch it.
Also moves RegisterClusterRoutes to core/http/routes beside every other
registrar, folds AutoMigrate and the epoch sequence into one
cluster.Migrate, and turns the peer route's auth-coverage spec into a real
assertion: it drives the request through the actual auth middleware
instead of comparing two string constants, which the old spec would have
passed even with the exemption deleted.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The NodeConnection model carried `default:now()`, which is PostgreSQL syntax
reaching the DDL, so AutoMigrate failed on the single-binary SQLite path and
took every SQLite caller of nodes.NewNodeRegistry down with it. Stamp the
database clock as an expression inside Claim instead, the way Register
already does, and leave the column plain.
CREATE SEQUENCE is Postgres-only for the same reason, so it is skipped on
another dialect, and Claim refuses that dialect outright: a fence that cannot
draw a token must say so rather than fail later as a missing function.
Also correct a claim the previous commit made in both the doc comment and its
message. An epoch is unique and never reissued, but it is not ordered: the
insert path draws its sequence value before taking the row lock, so a claim
that inserts after a release can be handed a lower number than one already
issued. Uniqueness is what Release needs, since it matches by equality;
callers must never compare epochs for order.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Release deletes the row, so a per-row `epoch + 1` restarted the numbering at 1
for the next claim. A replica could then be handed an epoch it already held:
claim w1 at epoch 1, lose the link silently, watch another replica claim and
release, reclaim and be handed 1 again, and its delayed cleanup for the first
dead link would match the live claim and delete it. The fence has to be
unique per node over time, not per row lifetime.
Every claim now draws nextval from a dedicated sequence on both the insert and
the conflict paths, so an epoch is never issued twice. The draw still happens
after the row lock on the conflict path, so the winning claim still holds the
highest epoch handed out.
Also drop last_seen. Nothing maintained it and it was always equal to
connected_at, but an indexed column named that way invites a second liveness
clock; whether the owner is alive is Instance.LastSeen, and whether a claim is
current is the epoch.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A worker whose link is silently broken reconnects to another replica while
the old owner's socket has not yet noticed. Without a fence both believe
they own it. Claim is a single atomic upsert returning the new epoch, and a
release must match both owner and epoch so a stale owner cannot delete a
live claim.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The peer link's WebSocket adapter and route constant lived in
core/http/endpoints/cluster, so the dialler in core/services/cluster had to
import an HTTP endpoints package to reach them. That pulled echo, core/http/auth
and core/config into a package whose doc says it is deliberately free of such
dependencies, and it made core/services/nodes reach an endpoints package
transitively. It also has no way forward: the worker-connect handler needs the
tunnel registry and the node token store, both of which are cycles from there.
Move WebsocketConn and PeerPath into core/services/cluster and let the endpoints
package import it, which is the direction the rest of core/http flows. The route
and the auth exemption still cannot drift apart, now asserted where both are
visible rather than by a const reference across the boundary, and the assertion
is stronger than the one it replaces: it pins the route under the prefix instead
of pinning the prefix's spelling.
Also guard the fresh-dial path with ctx.Err(), mirroring the cached path. A
caller with a 300ms deadline dialling a live, listening peer was told the peer
was unreachable, which would be enough for one impatient client to get a healthy
replica routed around once the relay consults these errors.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Distinguishes a peer missing from the registry from a peer that will not
answer: the second must never be readable as node absence, or a network
hiccup between replicas evicts healthy workers.
The distinction is a property of the error type rather than of the call
sites. The unreachable error formats its cause into its message and keeps
it out of its unwrap chain, so an ErrInstanceNotFound picked up on the
dial path cannot reach a caller's absence check.
One yamux session is cached per peer and re-dialled when OpenStream on it
fails, which is how both a dead transport and a graceful remote go-away
arrive. A reset of one stream never reaches the pool, so an abandoned
request cannot cost every other worker its link.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The prefix constant moves to core/http/auth beside the check that uses it, and
the endpoints package derives its route from there. Seven sibling endpoint
packages already import auth, so the previous direction would have deadlocked
the build as soon as this one registered in RouteFeatureRegistry, and it was
dragging echo, gorilla/websocket and yamux into unrelated service packages.
Four properties were argued in comments and held by nothing. Flipping the
empty-token check to fail open, making SetWriteDeadline a no-op, returning a
zero-length read for a zero-length message, and dropping the recover around the
callback all left the suite green. Each now fails a spec that asserts the
behaviour rather than the setter's return value.
SetWriteDeadline takes the write mutex because gorilla keeps that deadline in a
plain struct field applied at the next flush; SetReadDeadline must not take the
read mutex, since it goes straight to the net.Conn and would otherwise block
behind the read it exists to unblock.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A dead yamux session does not always arrive wrapped. Session.close publishes
shutdownErr and closes shutdownCh before it force-closes the streams, so a
Write or Close landing in that window gets the raw cause back instead
(session.go:507-510, 528-533), and for a peer that vanished the raw cause is a
bare io.EOF. The generic io.EOF clause then reported the dead session as a
clean completion.
Remove the clause. A clean read-side EOF never reached it anyway: io.Copy
consumes that and reports nil, and neither *yamux.Stream nor *net.TCPConn
takes a WriteTo/ReadFrom path that would hand one back. Every existing spec
still passes, the io.EOF entry in the normal-termination table included, which
is what showed the branch was dead for legitimate endings and live only for
the bug.
Add a spec driving a real yamux session end to end. Every mux shape until now
was a synthesized error, which is exactly why a race inside the real library
stayed invisible.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Upgrades to a WebSocket, wraps it as a yamux server session and hands it to
the caller. Rejects before upgrading so an unauthenticated dial sees a 401
rather than a WebSocket error, which is what the route-coverage test asserts.
The adapter keeps the reader of a partially consumed message across Read
calls. yamux reads through a 4 KiB bufio.Reader, so a small-payload test
cannot see a dropped message tail; the framing specs drive the adapter
directly with buffers smaller than the message.
An empty configured token authorizes nobody here, unlike the worker file
transfer server's check: this route is registered in every deployment, so
failing open would publish an unauthenticated mux.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Matching yamux errors with errors.Is was too broad. Session.close hands every
live stream ErrStreamReset wrapped around whatever killed the connection, so a
keepalive timeout, a broken TCP connection or a peer that simply vanished all
matched, and a relayed request that died reported a clean ending. Nothing
upstream would have retried or logged it.
Match the plain sentinels by identity, since only identity separates a stream
that was reset from the wrapped form that means the session died. Treat a
StreamError as a per-stream reset, and a GoAwayError as normal only when it
carries the no-error code, read off ErrRemoteGoAway because the constant is
unexported. ErrSessionShutdown needs no entry of its own; it is a GoAwayError
with that code.
Order matters as much as the matching: session death wraps its cause, which is
routinely io.EOF or a closed socket, so the mux checks run before the generic
endings. Reversing them alone puts a vanished peer back to nil.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
go-yamux/v5 matches none of its errors against net.ErrClosed, so the
classifier reported an ordinary teardown as a failure: when the session has
gone away, the FIN that Splice's own Close writes returns ErrSessionShutdown,
and a stream torn down under a live copy surfaces as ErrStreamClosed or a
reset. Splice owns that Close, so it owns the errors it produces; the
sentinels are named here rather than injected by the caller, which would make
a forgotten classifier reintroduce the same bug silently.
Cover the error half of the contract, which no in-memory pipe could reach: a
scripted stream now feeds Splice a genuine transport failure and each
closed-stream ending in turn. Replacing the tail of Splice with "return nil"
passed every previous spec.
Also assert that Splice does not return until the second direction has
finished, rename a spec that promised a leak check it never made, and correct
two comments that claimed more than the code did.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Returns on the first direction to finish and closes both sides so the other
unblocks; a sequential copy deadlocks on any protocol where the far side
speaks first. EOF and use-of-closed are normal termination, not errors.
The fourth spec covers a peer that stops reading mid-body, the case where a
copy is parked in Write rather than in Read. The other three tear down an
idle splice and pass even against a Splice that closes only one side.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
DiscoverAdvertisedAddr promised to return an error rather than a fallback
no peer can dial, but only rejected an unspecified address. With PostgreSQL
on the same host or pod as a replica, which is compose, single-node and any
sidecar layout, the route to it is loopback, so every replica advertised
127.0.0.1 and a peer dialling that reached itself. Loopback, link-local and
zoned source addresses are now rejected with an error naming the remedy, and
a port outside 1-65535 is rejected before it becomes an undialable address.
Liveness was also measured on each replica's own clock: Register and
Heartbeat stamped last_seen from the Go process, and Live compared those
rows against the reading replica's time.Now(). Skew therefore shrank or
stretched the window by writerBehind+readerAhead, evicting healthy peers or
keeping dead ones. Both sides now use the database clock, which is the one
clock every replica demonstrably shares.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
DS4 appends the opening thinking marker to tokenizer-templated prompts, so generated text begins directly with reasoning bytes. Starting DsmlParser in TEXT therefore puts the reasoning and closing marker in visible content.
Start the parser in THINK for structured chat requests with thinking enabled in both Predict and PredictStream. Keep the default TEXT state for raw prompts and reasoning-off requests, and add incremental regression coverage.
Assisted-by: Codex:gpt-5
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
stageDirectory and countStageableFiles already skip them, but
stageOptionDir did not - and it is the path sherpa-onnx voices take for
espeak-ng-data. The receiver writes "<file>.sha256" for every file it
accepts, so staging the sidecars made it write sidecars for those in
turn, one level deeper on every load.
Observed on a live node: "<file>.sha256" repeated eleven times, 5077
junk files out of 7832 in the models dir, and still growing. Staging
never finished, so vits-piper-it_IT-paola-sherpa stayed permanently
"staging on node" and every realtime warmup needing that voice failed
with the session then going silent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0142UfUh8HWxdim5JZqf8Tr6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Replicas need to find each other to relay worker traffic, and nothing in
the tree recorded a replica's address. The advertised address is discovered
by opening a UDP socket toward PostgreSQL and reading back the local
address, which yields the interface every replica demonstrably shares
without asking an operator to configure one.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The closure note in cluster/failure.go quoted a Gomega error that Gomega
does not emit. Describe the argument-count failure and the
Eventually().WithArguments() hint instead, so nobody greps for a string
that never appears.
The advisory-lock note in cluster_failover_test.go called the wedge
window unbounded. A SIGKILLed local child closes its socket at once, the
Postgres backend reads EOF and is reaped in milliseconds, so the
mechanism bounds the window tightly. Say bounded, and keep the low
probability but real framing, which was right.
The workflow comment attributed HealthCheckInterval to
core/services/nodes/health.go. It is declared in
core/config/distributed_config.go:64; health.go only carries the ticker
on the unexported checkInterval. Point a debugger at the right file.
Comments only, no behaviour change.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review of the whole branch found five comments that would send a reader to
the wrong place, plus three smaller inaccuracies. Nothing here changes
behaviour.
The KNOWN RACE note on both backend-log WebSocket handlers said the fix
needs an atomic snapshot-plus-subscribe "under the store lock". It does
not: BackendLogStore.mu guards only the buffers map, and AppendLine
enqueues and fans out under the per-buffer buf.mu. Whoever took the store
lock would ship and the race would survive, so both notes now name buf.mu
and say what s.mu does and does not exclude.
Two comments in the cluster harness quoted Eventually(c.FrontendAlive)
.Should(BeFalse()). FrontendAlive takes an index, so Gomega rejects that
with "requested 1 arguments but received 0". Both now quote the closure
form the specs actually use, and say why the closure is needed.
proveHealthCheckingIsAlive claimed to prove the health monitor ran for the
whole preceding window. It proves the monitor was alive at the end of it,
and inferring backwards needs any wedge to be sticky. In the
peer-replica-death spec that inverts: health checks are single-flighted by
a session-scoped pg_try_advisory_lock, the spec SIGKILLs the replica that
may hold it, and until Postgres reaps the session the survivor acquires
nothing and checks nothing silently. Consistently(healthy) can then pass
because nothing was checking, with the positive control still succeeding
once the lock frees. The doc now states what is proven, names that gap,
and says the assertion is a floor rather than a proof.
The Makefile still called DISTRIBUTED_TEST_FLAKES a retry count, which is
what seeded that error into the two docs just corrected against it, and
the workflow called the 15s window a reconcile tick when the mechanism is
HealthCheckInterval in the node health monitor.
Also: the cluster suite measured 509.1s / 509.8s / 512.3s, so about
8m30s and not the 8m39s/8m40s three files claimed; the dead-worker spec
title implied two independent detectors when both probes read one
advisory-lock-serialised verdict out of the same row; and the
sanitizeDBName length assertion used <= 50, which an empty string also
satisfies, where the invariant for an over-long input is exactly 50.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
--flake-attempts is total attempts, not retries: ginkgo v2.29.0 sets
maxAttempts = FlakeAttempts and loops attempt < maxAttempts, and the flag's
usage string reads "0 - failed tests are not retried". At 1 there is no retry
at all, so "retries a failing spec once" was false in CONTRIBUTING.md and
implied in .agents/building-and-testing.md. Both now say each spec runs once,
and cite the source so the next reader need not re-derive it.
Also restores the React-UI stub rationale, which is load-bearing because a spec
asserting on a UI asset passes locally against a real dist/ and is served the
stub in CI; explains why 213 and ~240 differ; records that the workflow also
triggers on master pushes, where paths-ignore does not apply; and completes the
LOCALAI_E2E_REQUIRE_BINARIES value table, including that any unparseable value
reads as ON.
In .agents/ci-caching.md the stale "13 of those 20" figure now carries its
qualifier inline rather than in the following sentence.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Two Make targets, a flake-budget variable and two environment variables
landed with no way to discover them. CONTRIBUTING.md now tells a contributor
how to run both suites, what each costs and which variables steer the cluster
one.
.agents/building-and-testing.md records the decisions that are easy to undo by
accident: suite-scoped containers, the shared NATS bus and what that means for
a new spec, BeforeSuite over SynchronizedBeforeSuite, the label split,
--fail-on-empty, the binary gate, the flake budget of 1, the coverage
exclusion, and why the cluster suite's long waits must not be shortened.
.agents/ci-caching.md lists tests-e2e-distributed.yml in its paths-ignore
inventory; the workflow already pointed readers there, so the cross-reference
was dangling.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Ginkgo exits 0 when a label filter matches nothing, so a refactor that
renamed or dropped Label("Cluster") would have left the job reporting
"Test Suite Passed" having started no cluster. LOCALAI_E2E_REQUIRE_BINARIES
does not cover that case: it only fires inside a spec that is already
running. Add --fail-on-empty to both distributed targets.
Drop -r from test-e2e-cluster while here. All six Cluster specs live in the
top-level package, and the cluster subpackage contributes nothing under this
filter by design, so recursing only widened the blast radius. test-e2e-
distributed keeps -r: it must reach the eight argument-validation specs in
that subpackage.
Raise the cluster job to 45 minutes, matching its sibling. The 20 minute
Ginkgo timeout bounds the suite alone; the job timeout must also cover setup,
which is the larger and more variable half here: cold-cache module download,
protoc and protogen-go, a full build of ./cmd/local-ai and a separate test
compile, realistically 8-12 minutes on a 4-vCPU runner. At 25 minutes the
runner would have hard-killed the job before Ginkgo could report which spec
hung, which is the red-with-no-evidence outcome that gets suites disabled.
Also move upload-artifact to @v7 with the rest of the repo, and note on the
react-ui stub step that it must go if a spec ever asserts on a UI asset,
since a developer box has a real dist/ and would not catch that locally.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add test-e2e-cluster and a second CI job that runs it. The cluster specs
spawn local-ai as real child processes and kill them, so they need a built
binary; keeping them in their own job means the fast in-process suite is not
held behind that build.
The binary is built with a stubbed core/http/react-ui/dist. A single
index.html satisfies the go:embed in core/http/app.go, and this suite drives
the HTTP API only, so the job skips a Node and Vite install entirely.
The job runs serial and pins --flake-attempts 1. Each Ginkgo process would
otherwise get its own PostgreSQL and NATS container while every spec spawns
two or three children, and a retry would hide exactly the nondeterminism the
suite exists to catch. Measured at 8m39s over three runs, hence a 25 minute
job timeout and a 20 minute Ginkgo timeout.
LOCALAI_E2E_LOG_DIR points inside the workspace so the per-process logs
upload as an artifact on failure; they are the only way to read a cluster
failure. LOCALAI_E2E_REQUIRE_BINARIES is set explicitly even though CI
already implies it, because a skipped cluster spec is indistinguishable from
a passing one and this job's whole value is that it cannot go green without
starting a cluster.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The two specs that assert a healthy worker stays healthy were pure negatives:
they say nothing happened. A cluster whose health checking had wedged, by
leaking the advisory lock the monitor takes at health.go:110, would freeze the
roster and satisfy both while observing a corpse. Kill the worker once the
window closes and require the roster to settle it to offline, so the preceding
Consistently is a statement about behaviour rather than about a stopped clock.
Applied to the cold-restart spec as well as the peer-death one: a restart is
exactly the event that could leave a replacement unable to check anything.
Document the hazard that can make an offline assertion hang. The staleness
branch skips a node already marked unhealthy (health.go:153-155), a skip meant
for nodes an operator took down, which also swallows the flap: an unhealthy mark
landing after the heartbeat goes stale means MarkOffline is never called and the
node stays unhealthy forever. Name the file and line at the assertion, and have
the failure message say so when the roster shows a node stuck there, so a
timeout sends the reader to LocalAI rather than to the harness.
Stop calling the two-replica registration spec a race. Start spawns workers
sequentially and the registrations land about a second apart; it is a
shared-roster identity test, and saying otherwise invites someone to trust it
for something it does not check.
WorkerRegistrar now bound-checks its index like every other index-taking method
here. It answered 0 for an out-of-range worker, and 0 is a real frontend index,
so the failure mode was a spec killing the wrong replica.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Four scenarios with no prior equivalent: killing a replica must not disturb a
worker that never depended on it, a cold-restarted replica must rehydrate the
roster from shared state and keep accepting the worker's heartbeats, a dead
worker must settle to offline on every replica, and two replicas registering a
worker each must converge on one roster.
The timings are measured, not assumed. Node liveness is heartbeat freshness, so
the only eviction path is StaleNodeThreshold (60s) plus one HealthCheckInterval
tick (15s), and neither is reachable from the CLI. A worker whose registrar was
killed was observed going offline at 74.2s. Every window here is sized to
outlast that, because an assertion that expires before the system could have
reacted proves nothing.
Two assertions are deliberately unlike the obvious form. Statuses are compared
for equality against a probe that returns a sentinel on error, rather than
asserting a name is absent from the healthy list: the list probe returns nil on
any error, and "does not contain" is satisfied by nil, so a 401 at the second
replica would have passed while observing nothing. And a killed worker is
required to settle to exactly offline, because it first flaps to unhealthy at
~8s and back to healthy at ~14s, which any not-healthy matcher would accept.
SpreadWorkerRegistrations is new, off by default, and exists so the racing
spec is a race: the harness otherwise points every worker at frontend 0, which
would have left that scenario asserting on two sequential writes through one
process. The default is unchanged because the baseline specs depend on it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add Q4 and Q8 GGUF builds with their shared vision projector. The
model is a recent refusal-removed Ornith derivative for alignment and
red-team research.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The previous round made a missing binary fail instead of skip, but only when a
workflow remembered to set LOCALAI_E2E_REQUIRE_BINARIES. That leaves the silent
pass one forgotten line away: the Cluster label partition is two specs, Ginkgo
exits 0 on skips, and a job that skips both reports "0 Passed | 2 Skipped" and
goes green having never started a cluster.
So the polarity is inverted. Binaries are required whenever CI is set, which
GitHub Actions always does, and the flag now exists to force the requirement
OFF rather than to be remembered ON. A local developer sees no change, since CI
is unset in an ordinary shell and a missing binary still skips with a message
naming the path and how to build it. off, no, n and disabled are honoured as
off; ParseBool rejects them, and reading a word that unambiguous as its
opposite would be a worse trap than the one this removes.
Also correct a claim the previous commit message got wrong. Comparing the
worker's registration id across the two replicas does not pin the topology:
NodeRegistry.Register looks a node up by name and preserves the existing id,
and both replicas read one Postgres, so registering the worker with every
frontend would yield identical ids too. The assertion is still worth keeping
for what it does catch, a replica answering from its own registry or database
instead of the shared one, and the comment now says that and nothing more.
The topology fact moves to where someone would break it: a note on
LOCALAI_REGISTER_TO recording that workers register with frontend 0 only, that
the cross-replica specs depend on it, and that nothing in those specs can
detect a change to it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The Cluster label partition is these two specs and nothing else, so a missing
binary skipped the entire job. Ginkgo exits 0 on skips, so a build step that
broke or moved its output would have left the job reporting "0 Passed |
2 Skipped" and going green without ever starting a cluster: the silent pass
this suite exists to make impossible. Skipping stays the local default, which
is the right courtesy for someone who has not run `make build`, but
LOCALAI_E2E_REQUIRE_BINARIES turns it into a failure that names the missing
path and the target that builds it. A value that is set but unparseable counts
as on, since reading it as off would restore the very skip it disables.
Failures also name themselves now. The roster poll kept returning a bare nil on
error, so a 401 at the second replica, a decode failure and "the worker never
registered" all presented identically as an empty list. It now retains the last
error and the last roster and reports whichever happened, through a lazily
evaluated Gomega description that costs nothing until something fails.
Finally, the two-frontend spec no longer depends on the harness to mean what it
says. It asserts an unauthenticated GET /api/nodes at frontend 1 is refused,
which observes the admin gate instead of assuming it, and it compares the
worker's registration id across the two replicas rather than its name. A future
harness that registered every worker with every frontend would have kept a
name-only assertion green while it quietly stopped proving anything about
shared state.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Tasks 4 to 6 built a harness that runs local-ai as real child processes, but
none of it had ever started a process: every spec so far returned inside
argument validation. These two specs are the first to run it against a real
binary, a real Postgres and a real NATS.
Two frontends against one database both see a worker that registered through
only one of them. Every failover spec assumes this, so it is asserted first.
One admin session is minted at frontend 0 and reused for both replicas rather
than registering per frontend. The auth routes share a five-per-minute-per-IP
limiter and all e2e traffic is 127.0.0.1, so a session per frontend would
exhaust the budget as soon as a spec needs a third one. Reuse is sound because
sessions live in the shared Postgres and the harness pins one HMAC secret
across replicas; frontend 1 answering /api/nodes with 200 on a cookie minted at
frontend 0 is what proves it.
The binaries are resolved before SetupInfra so a missing build skips without
first provisioning a database the skip would then have to tear down.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review round 1. Comments only, plus one guard.
The note on Process.alive claimed the exited check closed the zombie window.
It does not. The reaper closes exited only after Cmd.Wait returns, and Wait
marks the os.Process done before returning, so exited being closed implies
signal 0 already errors and the branch cannot fire earlier than the one it
precedes. The window between the child exiting and waitid collecting it stays
open in both versions, and the only real mitigation is for callers to poll
with Eventually rather than sample once. Keep the check as hygiene, say what
it actually does, and say it again on the exited field, so nobody reads the
old claim and drops the Eventually.
Record what the cold wipe destroys. The harness sets no LOCALAI_STORAGE_URL,
so the object store is a directory under DataPath, and quantization and
fine-tune outputs live there too. Postgres keeps the job row; the artifact it
points at does not survive the restart. A spec that asserts otherwise will
fail for a storage reason wearing a failover costume.
Tell callers to let a graceful stop finish before restarting: RestartFrontend
terminates with SIGKILL, so pairing it straight after StopFrontendGracefully
cuts the drain short and silently converts the rolling-update case into the
crash case.
Refuse to wipe when the cluster has no work dir. frontendDataDir is relative
when baseDir is empty, so a Cluster built by some future test helper without
one would have RemoveAll walking frontend-N/data inside the source tree. The
guard sits before terminate, so a refusal leaves the cluster as it was.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The point of running LocalAI as real child processes is to be able to take
one away. Add KillFrontend (SIGKILL, the lost replica), StopFrontendGracefully
(SIGTERM, the rolling update), KillWorker, RestartFrontend and FrontendAlive.
RestartFrontend pins the dead replica's original port. Workers read
LOCALAI_REGISTER_TO once at boot and never re-resolve it, so a replica that
returns on a fresh port is unreachable by exactly the workers that registered
with it and the failover under test never happens.
It also wipes the replica's data directory, so the process comes back with
empty local state and has to rehydrate node, session and job state from the
shared Postgres and NATS. Reusing the directory would model a pod with a
persistent volume and hide the class of bug these tests exist to find. That
is only safe because the harness pins LOCALAI_AUTH_HMAC_SECRET; otherwise the
wipe would take {DataPath}/.hmac_secret with it and every session minted
before the restart would 401 afterwards.
FrontendAlive consults the reaper's exited channel before signal 0: a child
that has died but has not yet been waited on is a zombie, and signal 0 to a
zombie succeeds, which would report a dead replica as alive.
The new specs cover argument validation only. Killing, stopping and
restarting a live process needs a built binary plus Postgres and NATS, so
those paths stay unexecuted until the failover suites land.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Session rows are keyed by an HMAC of the token under a secret generated
per instance into {DataPath}/.hmac_secret. The replicas shared that
secret only because they shared a working directory, and that directory
was the source tree. Give each frontend LOCALAI_DATA_PATH under its own
baseDir and pin LOCALAI_AUTH_HMAC_SECRET, so a session minted at one
replica resolves at every other one by construction.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The register handler answers 201 both for "user created, here is your
session" and for "this email already exists", so the status code cannot
tell a fresh registration from a repeat one. Key on the session cookie
instead and fall through to login when it is absent.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Restarting a frontend replica must not move it: workers read
LOCALAI_REGISTER_TO once at boot and never re-resolve it, so a replica that
returns on a fresh port is unreachable by the workers that registered with it.
startFrontend now takes the port, with <= 0 meaning "allocate".
Process logs are opened for append rather than truncated, so a restarted
process cannot erase the log of the instance that died, which is the log a
failover post-mortem needs. The post-SIGKILL wait is bounded, so one stuck
child no longer becomes a suite-wide timeout that names nothing. Stop is
nil-safe because Start returns a nil cluster after stopping itself.
Start's doc comment no longer claims to wait for worker registration; that
needs an authenticated admin session, so it now says callers must poll
/api/nodes themselves.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Runs local-ai as real child processes, one per frontend replica and one per
worker, against containerised infrastructure. The in-process suites cannot
express frontend-replica failure: there is no process to kill and no real HTTP
boundary between a worker and the frontend it registered with.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The path allowlist covered 13 of the 99 packages the suite reaches. Commit
1dc3aeef8 touched core/config, core/services/modeladmin and core/backend and
matched no entry, so it would have merged without running the very specs that
cover it. Use the paths-ignore denylist tests-e2e.yml already uses.
Disable the testcontainers reaper: the runner is ephemeral, so the reaper buys
nothing and its unpinned image was pulled mid-suite, defeating the pre-pull.
Drop continue-on-error, which no other workflow uses and which reports a failed
run as green. The job is advisory by staying out of branch protection instead.
Pin Go to 1.26.0 to match go.mod, and add the tmate-on-failure step.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The suite has never run in CI, so 239 specs across 32 files were verified only
by hand. Path-filtered to distributed code, advisory until it earns a track
record, and with flake retries at 1 rather than 5 so nondeterminism surfaces
instead of being retried away.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Three corrections from review of the previous commit.
The lock-order comment on SubscriberCount claimed no path takes s.mu and a
buffer lock together. Subscribe does exactly that, holding s.mu.RLock across
replica registrations that take buf.mu. State the rule that is actually true —
s.mu precedes any buffer lock, so counting after releasing it preserves the
order — and say what follows from it: the total is a sample, not a snapshot.
waitForLogSubscriber read as general-purpose but unblocks on the first
registered subscription. Subscribe attaches the exact-key buffer and each
replica buffer one at a time, so for a replicated model the count goes positive
while later replicas are still unattached and the race survives. Rename it
waitForSingleLogSubscriber, document that it holds only where Subscribe
resolves to one buffer, and assert on exactly 1: misuse then fails loudly on
the count rather than going quietly back to being flaky. Taking the expected
count as a parameter was the alternative, but that makes callers predict a
store-internal number and an under-count fails the same silent way as the
original bug.
The snapshot-then-subscribe race had no artifact outside a report, and review
found a second site carrying it. Mark both handlers identically, including the
point that swapping the two calls duplicates rather than drops and so is not
the fix. The race itself is left alone; this branch stays test infrastructure.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The WebSocket log handler writes its "initial" batch before it calls
Subscribe, so a line appended the instant that batch arrives lands in the
circular buffer with no subscriber to receive it. Three backend-logs specs
append exactly there and then wait out a 5s read deadline; once a gorilla
read hits its deadline the connection is unusable, so the spec cannot retry.
`--focus='Worker WebSocket log streaming' --repeat=25` failed on attempt 17
with nothing else running, which is far too often to wire into CI.
Add BackendLogStore.SubscriberCount, resolving a model ID by the same
exact-key and replica-prefix rules Subscribe uses, and have the specs poll it
until the handler has attached. Nothing in production calls it and no
assertion is weakened; the handler's own snapshot/subscribe window is left as
it is, being a production streaming question rather than a test one.
Verified with 60 repeats of the WebSocket specs and three consecutive
--randomize-all runs of the whole distributed suite, all at
--flake-attempts 1: 239 of 240 specs pass in about 80 seconds.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A failed CREATE DATABASE panics out of the assertion before closeDB runs,
leaking a pgx pool per attempt. With --flake-attempts 5 that exhausts
postgres:16-alpine's 100 connection slots, at which point the cleanup path's
own Expect fails the spec and one hiccup cascades across the suite. Scope the
admin handle so the panic unwinds through defer closeDB, and let cleanup use a
fallible tryAdminDB that reports rather than asserts.
Register DeferCleanup immediately after CREATE so a later failure cannot leave
the database behind, and warn on TestInfra that the container handles are now
suite-wide.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Starting a Postgres and a NATS container per spec cost roughly 48 minutes of
startup across the 213 specs behind SetupInfra, which is why this suite was
never wired into CI. Containers move to BeforeSuite and isolation comes from
CREATE DATABASE, which the dbName argument already described.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Install 404s because the gallery still points at mmproj-...-f16.gguf.
HF only ships ...-F16.gguf now, with a different sha256.
Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
Fixes#11673: on macOS the DMG launcher appeared to launch nothing. After
installing, the app sat in the menu bar with no window, nothing listening
on localhost:8080, and empty log files, because nothing ever started the
server unless the unrelated 'start on system boot' option was enabled.
- Start the LocalAI server automatically when the launcher opens and right
after a fresh install. The new auto_start_server config key defaults to
enabled and gets a settings checkbox; the legacy auto_start key was never
honored nor exposed, so every existing launcher.json carries an
unintentional false and is deliberately left behind.
- Fix the welcome window suppressing itself: its 'don't show this again'
checkbox was initialized with the inverted value, and SetChecked fired
the change callback which persisted ShowWelcome=false on the very first
showing.
- Surface auto-start failures through the systray startup-error dialog,
since there is no visible window during auto-start.
- Pass --app-version to fyne package so the app stops reporting itself as
version 0.0.0 in the About box.
- Document the first-launch flow (menu bar app, auto-start, WebUI URL) in
the macOS getting-started page.
- Repair two launcher specs that never ran in CI: a *bool matched against
BeTrue and a /tmp assertion that trips on Linux where the test tempdir
itself lives under /tmp.
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add three llama.cpp-compatible mixed quantizations from ISTA DASLab. These builds give Qwen3.8-27B users an 8.4 to 10.1 GB weight tier with the shared vision projector.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add Q4_K_M and Q8_0 MTP variants with the shared vision projector.
The publisher recommends these builds for faster Qwen3.8 generation.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add the 3B, 8B, and 30B safetensors checkpoints as one vLLM variant family so LocalAI can select the largest build that fits. Configure the parsers and sampling defaults recommended for Granite reasoning and tool calls.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The Makefile already had a hipblas branch, but no CI row built it and
the gallery's `amd:` mapping stayed commented out. On an AMD host the
capability lookup found no `amd` key and fell back to `default`, so
these users silently ran the CPU build.
Add the hipblas row to the backend matrix and the two gallery entries
it publishes, then point `amd:` at them.
Drop `-DGGML_HIPBLAS=ON` while here. `SD_HIPBLAS` sets `GGML_HIP`
itself, and `GGML_HIPBLAS` is the name ggml used before the rename, so
the flag only produced an unused-variable warning. Add gfx1151 to the
local target list to match the value the workflows pass in.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Tencent released three WeMM sizes with direct Sentence Transformers support. Add each safetensor repository so users can select the quality and resource tradeoff.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
OpenAI GA clients send multipart or raw SDP requests. They expect a bare
SDP answer. LocalAI only accepted its legacy JSON envelope, so signaling
failed before media setup.
Keep the JSON contract for existing clients. Accept both GA request
shapes and choose the matching response format.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add the Q4 and Q8 GGUF builds with the shared vision projector.\nThe variant pair lets LocalAI select the build that fits available memory.\n\nAssisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add complete vLLM and SGLang entries with their exact tool parsers. Preserve an explicit zero temperature in both backend adapters.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Node placement and replica rules could only name a model, so an operator
who pinned "llama3" to the GPU tier had to rewrite the rule whenever a
different model took over that job. An alias already gives a stable name
for whichever model serves it, and a rule on that name makes it a
deployment slot: repoint the alias and the placement follows.
A rule keeps the name the operator chose. Reads resolve that name through
the config loader to the model the rule governs, so the reconciler counts,
schedules and trims replicas of the target, and the router finds an
alias-keyed rule from the target it is already routing. An alias that
resolves to nothing governs nothing loadable, so the reconciler skips it
and the write paths refuse it.
A replica is shared by every name that resolves to it, so only one rule
can decide where it runs. The REST and MCP write paths reject a rule whose
target another rule already governs. A pair that arrives some other way,
such as a seed file or an alias repointed onto a model that already has a
rule, resolves in favour of the rule named after the model itself and then
the oldest, and the rest are listed as shadowed.
The eviction guard is the exception: it matches rules to replicas in raw
SQL inside a locking transaction and cannot resolve an alias. It reads a
stored target that the reconciler refreshes each tick, and falls back to
the rule's own name when that target is empty.
Assisted-by: Claude:claude-opus-5 golangci-lint eslint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ui): move node labels into the scheduling selector field
The scheduling page kept a node-label browser open above the rules
whether or not anyone was writing one, while the field that actually
needs labels, the rule's node selector, was two bare text inputs with no
hint of what the cluster reports.
The browser is gone. The selector's key input now completes against the
label keys the cluster uses, and the value input offers only the values
that key takes. The roster already loads for the page, so the
suggestions cost no request, and a roster that fails to load costs the
admin the hints and nothing else.
Suggestions stay suggestions: a key no node reports yet still commits as
typed, which is how an admin writes a rule before labelling the nodes
for it.
Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): size model fit against the cluster, not the frontend
The models page asked the frontend how much memory a model may occupy.
In distributed mode the frontend is usually a GPU-less pod while every
model runs on a worker, so a fleet of GPU nodes was told it could only
run the smallest CPU build. The variant picker's fits flag and its
auto-selection came from the same place, as did the hardware
recommendations.
The registry now reports the largest single healthy backend node. The
largest node, not the fleet total: a model loads into one node, so four
16GB workers are not a home for a 40GB model. An operator-set VRAM
budget caps a node's contribution, because the scheduler refuses a load
above that ceiling anyway, and a GPU node beats a CPU node holding more
system RAM.
GET /api/resources and GET /api/models carry this as an additional
cluster object. Their aggregate and ram fields keep reporting the
frontend's own hardware, which is what the resource monitor shows.
Variant selection judges backends against the union of the capabilities
present in the cluster, the way backend discovery already did.
Every path degrades to the local host: no cluster object in single-node
mode, and none when the registry cannot be read, so a hiccup narrows the
answer back to single-node behaviour rather than marking the whole
catalog too large.
The verdicts now name the node they belong to, since a model fits
somewhere or nowhere.
Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): follow upstream MTMD APIs
The dependency update adds MTMD initialization options to prompt and
bitmap helpers. The gRPC adapter now passes the server options through
each affected path.
The update also replaces the per-layer MoE regex helper. Preparation
probes both APIs because older forks still reuse this adapter.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
go-m1cpu v0.1.6 runs its cgo initialiser from a package init(), where
getFrequency() dereferences the CFTypeRef returned by
IORegistryEntryCreateCFProperty without a NULL check. On Apple M5 the
pmgr IORegistry node does not expose voltage-states5-sram /
voltage-states1-sram in the shape v0.1.6 expects, so the call returns
NULL and CFDataGetLength(NULL) faults before main() runs. Every command
dies, including local-ai --version.
The package is linked indirectly: cmd/local-ai reaches
gopsutil/v3/{process,disk}, which pull in gopsutil/v3/cpu on darwin,
which calls m1cpu.IsAppleSilicon() and m1cpu.PCoreHz().
v0.2.2 adds the missing NULL guard and moves the IORegistry probe out of
init() behind a lazy sync.Once. The exported Go API is unchanged and the
non-darwin stub is byte-identical, so gopsutil/v3 compiles against it
untouched and no other platform is affected.
Bumping gopsutil/v3 is not an alternative: v3.24.5 is the final v3
release, so the v3 line will never carry this fix.
Fixes#11735
Assisted-by: Claude:claude-opus-5
Signed-off-by: Szymon Podeszwa <2962046+sz-po@users.noreply.github.com>
* ⬆️ Update mudler/vllm.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(vllm-cpp): mirror ABI v23
The new engine pin reports ABI v23 and appends mmproj_path to
vllm_model_params. LocalAI still declares v21, so the build-time ABI
guard rejects every backend build.
Grow the Go mirror by the appended pointer and update its offset checks.
ABI v23 adds a video function but does not change the mirrored text
structs.
Assisted-by: Codex:gpt-5.6 [systematic-debugging]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
/api/tags appends :latest to untagged names, but chat and the other
model endpoints looked the tagged name up as-is and 404'd.
Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
* feat(gallery): add Ornith 1.5 397B variants
Add the official Q4_K_M and Q8_0 GGUF builds with their shared BF16 vision projector.
Assisted-by: Codex:gpt-5
* feat(gallery): resolve Ornith variant ordering\n\nKeep the 35B entries from master next to the 397B variants.\n\nAssisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): adapt to the common JSON API
The llama.cpp bump replaces its nlohmann JSON alias with common_json. Update the gRPC adapter for the new exception, iterator, conversion, and container APIs.
Assisted-by: Codex:gpt-5.6 [systematic-debugging]
* fix(turboquant): adapt the JSON exception type
The shared gRPC source now follows the upstream common_json API. The
TurboQuant fork still exposes nlohmann JSON and cannot compile the new
exception type.
Translate that exception in the fork-specific source patch so both
llama.cpp variants compile from the shared adapter.
Assisted-by: Codex:gpt-5.6 [systematic-debugging]
* fix(bonsai): adapt the JSON exception type
The shared gRPC source uses upstream's common_json wrapper. The Bonsai fork still exposes nlohmann JSON and cannot compile that exception type.\n\nTranslate the exception in the fork-specific preparation step and verify that repeated preparation stays idempotent.\n\nAssisted-by: Codex:gpt-5.6 [systematic-debugging]
* fix(llama-cpp): let prepare register gRPC
The score patch duplicated the gRPC CMake registration that prepare.sh already owns. Its stale context rejects the current upstream tools file on Darwin before compilation starts.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Those backends only forwarded the flag when it was "true", so "false"
never reached apply_chat_template and Qwen3 kept thinking on.
Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
The scheduler's liveness probe asks a worker a question over NATS and
reads "no responders" as proof the worker is gone. That is only sound
when every worker in the fleet subscribes to the subject asked.
It asked models.running, which arrived in 4.6. A 4.5 worker is alive and
serving, answers backend.list, and never subscribes to models.running,
so the probe condemned it on every scheduling attempt and marked it
unhealthy. A model pinned to such a node by its selector could then
never be placed at all: on this cluster an embedding model pinned to the
one Apple node was unschedulable for exactly this reason, while that
node's log showed it handling backend.list throughout.
Ask backend.list, which has been in the worker protocol far longer, and
treat a worker that answers anything as alive. Only a node that reports
no responders on every subject is absent, so adding a newer subject here
can never condemn an older worker.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
A model's revision is published by administration and checked against on
every inference request. Those were computed by separate code: the
request path resolves through the loader, while each publisher hashed
whatever ModelConfig it happened to hold. By then SetDefaults had folded
in the GGUF guess and app-level options, so the published value was one
no request would ever carry and the model became unroutable until the
row was deleted by hand.
Fixing the publishers one at a time did not hold. Three rounds each
found another: the startup resync, then a saved edit and a toggle, then
a rename and the peer-change path.
ModelConfigLoader.RevisionFor is now the only way to obtain a revision,
and the raw hash is unexported, so a caller outside this package cannot
hash a config it holds. A publisher and a request agree by construction
rather than by two implementations happening to match.
The request path no longer falls back to hashing its merged config
either: an unstamped config is routed without a revision rather than
with a wrong one.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
When no node the selector allows has a free slot, scheduling falls back
to evicting the least-recently-used idle model. That eviction searched
every healthy node, so it freed a slot on a node the selector forbids
and the model was then placed there: pinned to one class of hardware and
running on another.
An unrelated model pays for it. On this cluster an embedding model
pinned to Apple hardware could not reach its only matching node, so each
attempt evicted a large language model from an Nvidia node, failed to
start there anyway, and left the evicted model to reload. Repeated, that
reads as one replica bouncing between nodes.
Eviction is now restricted to the candidate set the selector produced.
With no selector the candidate set is nil and eviction stays global.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
The abandoned-load sweeper treated a replica row with no load job as
abandoned. Only the request path creates load jobs; the reconciler's own
scale-up loads a replica without one. So any scale-up that ran past the
five-minute grace period was deleted mid-transfer, which for a
multi-gigabyte checkpoint is every time. The replica never finished
anywhere, and the reconciler kept re-placing it, so it looked like one
replica hopping between nodes instead of a model reaching its replica
count.
A row with no job is now reclaimed only once its node stops being
healthy, which is the case the sweeper was written for: a worker that
dropped out mid-transfer. A job that failed or stopped heartbeating
still proves abandonment on its own. Every uncertain case leaves the
slot held.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
Only backend workers subscribe to backend.list. ListBackends asked every
node that was not pending, offline or draining, so an agent worker could
only answer "no responders", which the error handling reads as a node
that has gone away. Every poll of the backends view therefore marked
each agent node unhealthy, and its next heartbeat marked it healthy
again.
While unhealthy the node is not schedulable, so this also cost agent
capacity for as long as each flap lasted.
Skip non-backend workers, as the backend-op fan-out already does for the
same reason. A backend worker that does not answer is still marked
unhealthy: that one really is gone.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
The revision was computed after SetDefaults, which folds in things that
are not persisted configuration: the GGUF guess, the hardware defaults,
and app-level options such as threads.
The GGUF guess is the damaging one. It parses the model file to fill in
values like context size, and when that parse fails it falls back to a
different default. Whether a multi-gigabyte file on network storage
parses at a given moment is not a property of the configuration, so one
unchanged YAML produced two different revisions depending on when it was
read. The controller rejected every request carrying the other one, and
the model stayed unroutable until the stored value happened to match
again. This is why it never reproduced against a model directory with no
weights in it: the guess is skipped there and both values agree.
The app-level defaults are the same class of bug with a slower fuse:
changing threads in the settings UI changed every model's revision and
made every model unroutable.
The revision is now stamped when the file is parsed, before any defaults
are applied, so it is a function of the file alone.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
Two code paths computed a model's revision. Inference resolves the
config through the loader, which applies SetDefaults a second time.
Everything that publishes a revision hashed the stored config instead,
with SetDefaults applied once.
SetDefaults is not idempotent for every model: it re-runs the GGUF guess
and the hardware defaults, both of which read state the stored config
does not carry. Where the two disagree, a publisher wrote a revision no
request would ever carry, and the model became unroutable the moment it
was published. On this cluster the startup resync republished one such
value and every request for that model was then rejected against it.
The publishers now resolve the revision through the loader, exactly as a
request does, so there is one definition rather than two that agree only
when SetDefaults happens to be idempotent. This covers the startup
resync, a saved config edit, and enabling or disabling a model.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
quay.io redirects blob downloads to pre-signed S3/Akamai URLs that
expire after about 10 minutes. On a slow connection a multi-GiB
backend layer cannot finish inside that window, so the connection
drops mid-stream on every attempt. The retry added for #10577
restarted each attempt from byte zero, which replayed the same
failure until the budget ran out and the install failed with
"unexpected EOF".
A retry now keeps the bytes already on disk and re-requests the
blob with "Range: bytes=N-". Each request goes back to the
registry, so it gets a fresh redirect URL and auth token. The
retry budget only counts attempts that made no forward progress,
so a slow link that keeps advancing keeps downloading. A resumed
file is spliced from separate responses and bypasses the digest
check in layer.Compressed(), so the assembled file is re-verified
against the layer digest before it is trusted; on a mismatch the
download starts over through the verified reader.
Fixes#10577
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): add Qwen3.8 OBLITERATED variants
Add Q4_K_M and Q8_0 llama.cpp builds with the shared BF16 vision projector.
Assisted-by: Codex:gpt-5
* fix(tests): implement node liveness stub
NodeCommandSender now requires PingNode. The endpoint test stub must
implement it before the package can compile.
Assisted-by: Codex:gpt-5 [Codex]
* fix(distributed): restore node liveness tests
The router now probes models.running before it schedules work. The E2E
workers only mocked backend.install, so every test node appeared offline.
The endpoint test double also missed the new PingNode method and stopped
the Linux, Apple, and lint jobs during compilation.
Mock the existing worker reply in both distributed fixtures and keep the
endpoint test double aligned with NodeCommandSender.
Assisted-by: Codex:gpt-5 [golangci-lint]
* fix(tests): check node liveness replies
The liveness test subscriptions ignored setup and reply errors.
Errcheck rejected each branch that carried them.
Assisted-by: Codex:gpt-5 [golangci-lint]
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(distributed): restore node liveness tests
The router now probes models.running before it schedules work. The E2E
workers only mocked backend.install, so every test node appeared offline.
The endpoint test double also missed the new PingNode method and stopped
the Linux, Apple, and lint jobs during compilation.
Mock the existing worker reply in both distributed fixtures and keep the
endpoint test double aligned with NodeCommandSender.
Assisted-by: Codex:gpt-5 [golangci-lint]
* fix(tests): check node liveness replies
The liveness test subscriptions ignored setup and reply errors.
Errcheck rejected each branch that carried them.
Assisted-by: Codex:gpt-5 [golangci-lint]
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(ci): bound Discord release summaries
The release model can return more than Discord's 2,000-character
message limit. Discord then rejects the entire release notification.
Ask the model for a smaller response and truncate extracted content to
1,800 characters before the notification step. The smaller bound leaves
room below Discord's hard limit when model output varies.
Assisted-by: Codex:gpt-5
* fix(tests): implement node liveness stub
NodeCommandSender now requires PingNode. The endpoint test stub must
implement it before the package can compile.
Assisted-by: Codex:gpt-5 [Codex]
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The resync added in 3953448f6 ran before LoadModelConfigsFromPath, so it
read an empty loader, reconciled nothing and reported success. The
symptom was a stored revision that stayed stale across restarts while
the log showed no complaint, which is exactly what the resync was meant
to prevent.
Move the call after the configs are loaded, and refuse to treat an empty
loader as a clean run: reconciling zero models is indistinguishable from
reconciling correctly, and that is what hid the mis-ordered call.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
The controller pins a model's replicas to a stored revision and rejects
any request carrying a different one. Nothing ever re-derived that value
from the configuration on disk: it moved only on an edit, a gallery
install, or a peer's change broadcast. An inference request may only
establish a revision, never replace one.
So any other way for the two to diverge left the model permanently
unroutable. A configuration edited while a frontend was down lands
there, and so does a change in what the revision is computed over: an
upgrade that alters the hashed form leaves every stored revision
describing a configuration that no longer exists. The only recovery was
deleting the row by hand, which is not something a cluster should need.
Each frontend now reconciles the stored revisions against the loaded
configurations at startup and republishes the ones that disagree. Only
those: republishing quarantines every replica loaded under the old
revision, so doing it for a model that did not drift would unload a
healthy replica for nothing. A model with no stored revision has never
been served and is left for its first request to establish.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
A replica row in staging or loading holds its slot, because slot
allocation counts every state except unloading. Nothing ever reclaimed
such a row: every reconciler pass and the router's eviction query filter
state = "loaded", and the per-model probe skips rows without an address,
which is exactly what a row that never finished loading has.
So a worker that dropped out mid-transfer left a row that pinned the
only replica slot for that model on that node. Scheduling then found no
free slot and eviction found nothing it was allowed to evict, and the
request failed with "no replica slot on <node> and eviction failed: all
models busy". The state persisted until an operator intervened.
The reconciler now reclaims a row stuck before serving when no load job
is driving it. Ownership is decided by the job's LastProgress heartbeat,
not by elapsed time: staging a large checkpoint legitimately runs for a
long while without touching the replica row, so a deadline would either
be a model-size cliff or reclaim a healthy transfer. That heartbeat is
the same signal job takeover already trusts. Any error reading the job
leaves the slot held, because holding one for another pass costs a
scheduling opportunity while a wrong reclaim restarts a multi-gigabyte
transfer.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
A node's status comes from its HTTP heartbeat. Backend installs travel
over NATS. The two are independent, so a worker that dies stops
answering on the bus at once but stays healthy in the database until its
heartbeat ages out. Inside that window the scheduler picked a node it
could not reach, and the request failed with "no responders available"
rather than moving to a node that was up.
The scheduler now probes the node it selected and, when nothing answers,
marks it unhealthy and selects again. The demotion is what makes the
retry terminate: the next selection reads only healthy nodes. It also
tells the other frontends what this one learned, so the cluster does not
rediscover a dead worker one failed request at a time.
Only nats.ErrNoResponders counts as absent. A worker that answers slowly
stays eligible, because dropping it would cost capacity that is really
there. The probe reuses the models.running subject: a new subject would
go unanswered by workers that have not been upgraded, and every one of
them would then look dead.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
Picks up "surface option errors instead of discarding them". New records
the error from applying its options and Run returns it, so a process
whose state directory cannot be created reports the real cause instead
of failing later inside os.MkdirAll("") with an empty path.
LocalAI already resolves that directory itself, so this covers the other
process.New call sites rather than changing behaviour on the backend
start path.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5
process.New applies its options and discards the error they return. When
WithTemporaryStateDir could not create a directory, StateDir stayed
empty and every later option went unapplied, so the failure surfaced
from Run as "mkdir : no such file or directory" naming no path.
That message cost a full day of diagnosis on a worker whose volume was
full: the real errno was ENOSPC and nothing reported it.
The loader now creates the directory itself and returns the underlying
error with the path attached.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
A request that carries a file stages it to the worker, which writes it
under its staging directory. Nothing removed it afterwards. The frontend
expires ephemeral keys from object storage, but that sweep never covered
a worker's local disk, so every image, audio clip and video a worker
ever served stayed on it.
One worker had accumulated 175 request directories over three months.
The volume reached 100 percent, and from that point every backend start
failed because the process manager could not create a state directory.
The worker now sweeps its ephemeral staging directory on a timer and
once at startup, so files left by a crash are reclaimed too. Staged
model files live beside that directory and are not touched.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
"stale model config revision" reported only that two hashes differed.
It named neither, so an operator could not tell an edited configuration
from a revision that is not reproducible for one unchanged file, and the
failing value appears in no table.
The error now carries the revision the request brought and the one the
controller holds. It still wraps ErrStaleModelConfigRevision, so callers
that classify the error keep working.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
Add the 2B and 4B distilled models alongside the 9B model. Their compact GGUF builds make the Qwen3.8 reasoning distill practical on smaller hosts.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
LiquidAI now publishes official DSpark draft sidecars for its 2.6B target. Pair Q4 and Q8 targets with matching draft choices so LocalAI can use speculative decoding across different memory budgets.
Assisted-by: Codex:gpt-5 [systematic-debugging]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
syncKnownUsecasesFromString rebuilds KnownUsecaseStrings by ranging
GetAllModelConfigUsecases, which is a map. Go randomizes that order per
call, and the field is part of the serialized config, so one unchanged
YAML hashed to a different config revision on every load.
A model that derives a single usecase hid the problem. One that derives
several, such as a chat model with an mmproj, alternated between as many
revisions as there are orderings. The router treats a revision it did
not establish as a config change, so requests failed with "stale model
config revision" until the stored value happened to match again.
Sorting the list makes the revision a function of the file alone.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
The request middleware merges the caller's prediction parameters into
its copy of the model config. core/backend.ModelOptions then hashed
that copy, so the revision identified the request body rather than the
persisted configuration.
EstablishModelConfigRevision stores the first revision it sees and
requires an exact match afterwards. The first request after a restart
therefore pinned the model to its own temperature, top_p and stop
values, and every later request that sent different ones failed with
"stale model config revision". No config edit was involved.
The loader now stamps the revision when it materializes a config,
before any request override reaches it, and ModelOptions reads that
stamp. Model administration keeps hashing the same persisted config, so
both paths agree on one revision per configuration.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
* fix(vram): persist remote probe metadata
The startup warmer repeated remote size and GGUF metadata probes after every restart because both caches lived only in memory. Store successful HTTP probes for 24 hours so frequent restarts reuse the prior results.
Bound the cache, reject invalid records, and purge it when gallery data changes. Local model files continue to bypass persistence.
Assisted-by: Codex:gpt-5
* fix(vram): check temporary file cleanup
The lint gate rejects the unchecked cleanup call in the persistent cache writer.
Assisted-by: Codex:gpt-5.6 [golangci-lint]
* fix(vram): make persistent cache optional
Remote metadata probes can transfer enough data that operators need
control over disk reuse and startup warming. Gallery autoload now gates
both behaviors, and the runtime setting applies changes immediately.
Assisted-by: Codex:gpt-5
* fix(ui): expose gallery startup pre-warm
The existing gallery autoload setting also gates the startup metadata warmer. Name both effects in Settings so operators can find the requested boot control.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* docs: design configurable copy buffering
Document the context-aware copy buffer option and its validation plan.
Assisted-by: Codex:gpt-5
* docs: design durable distributed staging operations
Assisted-by: Codex:gpt-5
* docs: design distributed model config revisions
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
* feat(config): add stable model revisions
Hash typed model configuration and effective protobuf options deterministically for distributed revision comparisons.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
* feat(worker): acknowledge exact model stops
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
* feat(nodes): track model config revisions
Assisted-by: Codex:GPT-5 [apply_patch]
* fix(distributed): retry quarantined model cleanup
Stop quarantined replicas by exact process identity, retain failed cleanup as durable capped retries, and compare-and-delete only the claimed registry row. Process one sufficiently leased row at a time so multiple frontends cannot duplicate slow cleanup work.
Assisted-by: Codex:gpt-5
* fix(distributed): bind loads to config revisions
Assisted-by: Codex: GPT-5 [OpenAI Codex]
* fix(modeladmin): apply config revisions consistently
Route model edits, patches, state changes, deletion, and peer refreshes through the same revision lifecycle. Quarantine stale replicas before exact cleanup and report durable pending cleanup without failing successful config writes.
Assisted-by: Codex: GPT-5 [OpenAI Codex]
* feat(distributed): expose model config revision state
Document replica revision observability and durable cleanup behavior. Keep pending cleanup explicit in model mutation responses and verify endpoint contracts expose revision state without serialized load options.
Assisted-by: Codex:GPT-5 [OpenAI Codex]
* test(distributed): cover model revision convergence
Exercise cross-frontend quarantine, stale replay rejection, exact cleanup retry, worker re-registration, and current-generation replica convergence against the distributed PostgreSQL harness.
Assisted-by: Codex:gpt-5
* fix(distributed): pass config revision CI checks
Keep configured gallery sources out of authoritative runtime snapshots only after validating their real schema, and harden rollback snapshots against symlink races and non-regular files.
Assisted-by: Codex: GPT-5 [OpenAI Codex]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The Qwopus variant pair was added twice, which makes the later pair unreachable and fails the gallery invariant on master.
Keep the first pair that LocalAI already resolves and remove the dead duplicate declarations.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Keep the homepage focused on runtime capabilities and move engine details to their canonical directory. Make installation choices stable and explicit for users across supported hardware.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-bot <306113404+localai-org-bot@users.noreply.github.com>
* docs: design configurable copy buffering
Document the context-aware copy buffer option and its validation plan.
Assisted-by: Codex:gpt-5
* docs: design durable distributed staging operations
Assisted-by: Codex:gpt-5
* fix(distributed): merge durable staging operations
Use active model load jobs as the durable operations baseline and overlay replica-local staging progress without duplication. Preserve tracker-only operations when the registry cannot be read.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The gallery ranker reads serving features from tags instead of backend options. Mark two entries that enable draft MTP so ranking can identify them correctly.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix: GHSA-qwww-vcr4-c8h2 security vulnerability
Automated dependency upgrade by OrbisAI Security
Signed-off-by: anupamme <mediratta@gmail.com>
* fix: upgrade react-router-dom to 7.18.2 to fully remediate GHSA-qwww-vcr4-c8h2
The prior fix pinned react-router@7.18.2 directly but left react-router-dom
at ^7.18.1, which bun resolved to 7.18.1. That package bundles its own
react-router@7.18.1 sub-dep, leaving the vulnerable version in bun.lock
via the react-router-dom/react-router scoped resolution. Pinning
react-router-dom to 7.18.2 and regenerating the lockfile removes all
7.18.1 resolutions.
Assisted-by: Claude Code:claude-sonnet-4-6
Signed-off-by: Anupam Mediratta <mediratta@gmail.com>
---------
Signed-off-by: anupamme <mediratta@gmail.com>
Signed-off-by: Anupam Mediratta <mediratta@gmail.com>
* docs(blog): add the 4.9 release post and its demo clips
The 4.9 cycle changed how you authenticate, how chat handles a history
that no longer fits, and where models and backends live in the UI. The
release notes list every pull request; this post covers the three
changes that alter day-to-day use, and leads with the auth one because
it needs an action before upgrading.
Two clips are recorded from a real session against a local-ai built
from master with the live gallery loaded: model-lifecycle.mp4 walks
the unified models and backends pages, import-model.mp4 shows the
rebuilt import form. Both follow the clip conventions in
.agents/preparing-a-release.md: h264, no audio track, 1000x562, under
30 seconds, and named after the feature so they stay reusable.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Playwright]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(blog): anti-slop pass over the 4.9 post
Ran the post through the humanizer and no-ai-slop rules, calibrated
against what-landed-in-localai-4-8.md. That post is the one #11324
left unchanged, so it sets the rate for this series.
The draft ran denser than it on two constructions: "rather than" at
5.1 per thousand words against 3.5, and "instead of" at 3.1 against
1.6. Both are now at or below the 4.8 rate, 2.7 and 1.5, by rewriting
seven of them as plain statements.
Also cut: "keeping both cost a mode switch", a ledger metaphor for
something that is not money, which is the tell #11324 removed eight
times from the APEX post. "A follow-up fixed the thing that made that
awkward", an unearned framing plus a colon reveal. "This release adds
a different one: compress them", a second colon reveal. And
"byte-structurally identical", a second exactness idiom in a post that
already uses "byte-identical" where the precision carries weight.
Five paragraphs opened with "Two things" or "Two details", so three of
them start differently now. The summary listed three items, which is
the rule of three; it lists four, like the 4.8 summary.
Every figure, PR number, link and media reference is unchanged, checked
by diffing them out of both revisions. Hugo builds clean and the
rendered HTML has no em dashes.
Assisted-by: Claude Code:claude-opus-5 [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
2026-08-20 22:55:13 +02:00
678 changed files with 75557 additions and 8598 deletions
If your endpoint should be tracked for usage (token counts, request counts), add the `usageMiddleware` to its middleware chain. See `core/http/middleware/usage.go` and how it's applied in `routes/openai.go`.
## Control-plane database health metrics
In distributed mode the frontend registers three OpenTelemetry gauges over the
wired in `core/application/distributed.go`). They reach `/metrics` through the
same Prometheus exporter as the rest of the API metrics.
| Metric | Meaning | Page when |
|--------|---------|-----------|
| `localai_control_plane_oldest_xmin_age` | Transactions elapsed since the oldest snapshot any backend still holds | above a few million, and rising |
| `localai_control_plane_longest_transaction_seconds` | Age of the longest open transaction | above 3600 |
| `localai_control_plane_dead_tuple_ratio` | Dead tuples per live tuple, labelled by `table`, on `backend_nodes`, `node_models` and `gallery_operations` | sustained above ~10 on a small table |
A sustained high `localai_control_plane_oldest_xmin_age` is the one to page on.
While it grows, autovacuum can reclaim nothing anywhere in the database no
matter how often it runs, so the dead tuple ratio keeps climbing and a six-row
registry table can reach hundreds of megabytes. Tuning autovacuum does not help.
The fix is to find the transaction holding the horizon open and clear it:
An orphaned prepared transaction is cleared with `ROLLBACK PREPARED '<gid>'`,
and a stale slot with `pg_drop_replication_slot('<slot_name>')`. Check both
before concluding that a bloated table has some other cause.
Sampling is scrape-driven behind a 30 second cache, so scrape frequency does not
translate into database load. Failed and timed-out samples cost the same interval
as successful ones, so a database that is already struggling is not retried on
every scrape. A failed sample reports the last good values rather than failing the
scrape, because these gauges matter most when the database is struggling. Before
the first successful sample the gauges are absent rather than zero, since a zero
xmin age would read as a healthy horizon: alert on `absent()` too if you need to
distinguish "healthy" from "never sampled".
## Advertising surfaces — where to register a new capability
Beyond routing and auth, LocalAI publishes its capability surface in **four independent places**. When you add an endpoint — especially one introducing a net-new capability like a new media type or a new auth-gated feature — you must update every relevant surface. These aren't optional: missing them means the endpoint works but is invisible to clients, admins, and the UI.
- **Don't weaken the gate:** never hand-lower a baseline or widen a tolerance to turn a red gate green. The ratchet only moves up.
- If a change drops coverage, **add tests** (sort `coverage-summary.json` by line% ascending to find untested code) rather than editing the baseline. When coverage legitimately rises, commit the regenerated baseline (`make test-coverage-baseline` / `test-ui-coverage-baseline`).
- The Go gate is **strict — no tolerance**; `covermode=atomic` keeps it deterministic. The UI gate keeps a small tolerance only because its e2e coverage isn't.
## Distributed-mode test suites
Two suites cover distributed mode (frontend replicas, worker nodes, PostgreSQL), split by a Ginkgo label. Neither starts a message broker: a distributed deployment needs PostgreSQL and the frontends' own HTTP listener, and the `nats-io` modules are not in the build.
-`make test-e2e-distributed` runs `Distributed && !VLLMMultinode && !Cluster` over `./tests/e2e/distributed` recursively. Services are wired directly into the test binary. 231 specs in about 2m20s (measured 133.3s for the main suite plus the 8 argument-validation specs in the subpackage).
-`make test-e2e-cluster` runs `Cluster` and spawns real `local-ai` child processes through the `tests/e2e/distributed/cluster` helper package. It **builds `./local-ai` itself** (the `e2e-binary` target), so you do not run `make build` first and cannot forget to. 26 specs, 933.8s of specs in a 15m37s target run (four measurements; the suite grew from 6 specs and roughly 509s, then 21 specs and 802.5s, then 24 specs and 897 to 907s, so budget on the newest number).
Both jobs live in `.github/workflows/tests-e2e-distributed.yml`, with `timeout-minutes: 45` each. They trigger on pull requests *and* on every push to `master`; the `paths-ignore` filter (see [.agents/ci-caching.md](ci-caching.md)) sits on the pull-request trigger only, so a master push always runs both. They are advisory only because `master` carries no branch protection, which is a repository setting and not a YAML key: `continue-on-error: true` would flip the run's *conclusion* to success and hide the failure, so it is not used.
- **The container is suite-scoped, not spec-scoped.** `SetupInfra` used to start a PostgreSQL (~10s) and a broker (~3.5s) per spec. Across the 213 specs behind it that was roughly **48 minutes of pure container startup per run**, which is why this suite was never in CI. (213 rather than the ~240 above: the larger number is everything the label filter selects, the smaller one is just the specs that call `SetupInfra`.) The broker container is gone entirely, and the PostgreSQL one now starts once in `BeforeSuite` with each spec getting its own database via `CREATE DATABASE` (~67ms), which is what the `dbName` argument was always describing. Adding a spec needs no change: call `SetupInfra("some-name")` as before, the name is a prefix and a counter keeps it unique.
- **Consequence for new specs:** a fan-out carrier is *shared* within a Ginkgo process, so a wildcard subscriber can observe another spec's traffic. Filter assertions on an identifier your spec owns (a node ID, a job ID) instead of counting everything on `jobs.*.progress`, and verify the spec with `--randomize-all`.
- **`BeforeSuite`, not `SynchronizedBeforeSuite`.** Under `ginkgo -p` each process then gets its own container, keeping each process's carrier isolated. A single shared database across parallel processes would let specs on different processes see each other's messages on the same subject.
- **The label split.** The 8 argument-validation specs under `tests/e2e/distributed/cluster/` carry `Label("Distributed")` only, on purpose: they need no binary and no PostgreSQL, so they belong in the fast job. That is why `test-e2e-distributed` keeps `-r` (it must reach the subpackage) and `test-e2e-cluster` deliberately does **not** (the subpackage is out of its scope).
- **The binary is built, and a stale one is refused.** `test-e2e-cluster` used to take `./local-ai` as given and only check that a file by that name existed. An edit that was never rebuilt therefore ran the entire suite against an older build while reporting on the working tree, which surfaced once as a spec failing on caller line numbers that matched no line in any source file. Two things close it. The target depends on `e2e-binary`, which is `go build -o local-ai ./cmd/local-ai` and not `make build` (that one also builds the React UI, which this suite never touches and CI stubs). And `localAIBinary` in the harness **fails**, on a developer box as well as under CI, when the binary is older than the newest non-test Go source in the tree, which covers a run started with `LOCALAI_E2E_BINARY` or by invoking ginkgo directly. `_test.go` files are excluded from that scan on purpose: they compile into the ginkgo suite and never into `local-ai`, so counting them would fire on every edit to the spec being run. Set `LOCALAI_E2E_BINARY` to opt out of the build; there is no opt-out of the staleness check, because a rebuild is the remedy.
- **`--fail-on-empty` is load-bearing on both targets.** Ginkgo exits 0 when a label filter selects nothing, so without it a refactor that renames or drops `Label("Cluster")` leaves the target reporting "Test Suite Passed" having started no cluster at all. `LOCALAI_E2E_REQUIRE_BINARIES` does not cover this case: it only fires inside a spec that is actually running.
- **The binary gate.** `localAIBinary()` and `mockBackendBinary()`**fail** rather than skip when `CI` is set, or when `LOCALAI_E2E_REQUIRE_BINARIES` is truthy; `LOCALAI_E2E_REQUIRE_BINARIES=0` (also `off`, `no`, `n`, `disabled`, and anything `strconv.ParseBool` reads as false) forces skipping even under CI. **Any value that parses as neither reads as ON**, not off: setting the variable to something meaningless means someone meant to turn the gate on, and reading it as false would quietly restore the silent skip the flag exists to remove. The whole polarity is deliberate, because in CI a skipped cluster spec is indistinguishable from a passing one: Ginkgo exits 0 on skips. Locally a missing binary still just skips, since `CI` is unset in an ordinary shell.
- **Flake budget: no retries at all.** `--flake-attempts` is *total attempts*, not retries (ginkgo v2.29.0 `internal/group.go` sets `maxAttempts = FlakeAttempts` and loops `attempt < maxAttempts`; the flag's own usage string reads "0 - failed tests are not retried"). `DISTRIBUTED_TEST_FLAKES` defaults to **1**, so each spec runs once and a failure is a failure, and `test-e2e-cluster` pins `--flake-attempts 1` outright rather than reading the variable. The repo-wide `TEST_FLAKES=5` means up to five attempts, so up to four retries. These suites exist to surface nondeterminism, and a retry converts exactly that signal into a green run. Raise it locally when bisecting something unrelated, not in the Makefile.
- **Coverage:** `tests/e2e/distributed` is excluded from the coverage roots (`COVERAGE_E2E_ROOTS = ./tests/e2e`, run non-recursively), and so is the `cluster` helper package beneath it. Neither suite moves the baseline, so production code that these suites are the only cover for reads as **uncovered**. Unit tests for such code belong under `./core/...` with `testutil.SetupTestDB()`.
- **The cluster job builds against a stubbed React UI.** `core/http/react-ui/dist` is gitignored and built by Node, so the workflow writes a one-line `index.html` there to satisfy the `//go:embed react-ui/dist/*` in `core/http/app.go` and skips a full Node and Vite install. That holds only while the suite drives the HTTP API and never the UI, which has its own e2e suite. A spec that ever asserts on a UI asset would pass locally, where a real `dist/` exists, and be served the stub in CI: if you write one, the stub step has to go and the real build come back.
- **The topology that matters is two frontends and two workers.** `cluster_busless_test.go` runs the whole programme on it, and it is the only shape in which the OWNER path and the RELAY path are live at the same time: with the two workers' tunnels landing on different replicas, every request either goes to the replica holding that worker's tunnel or is relayed through the one that does. A one-worker cluster cannot tell those apart, and cannot state that killing a replica re-homes only ITS worker. Which worker served a request is read back from `node_models` rather than assumed, and the two models are pinned to one worker each through the admin API (`PUT /api/nodes/:id/labels` plus `POST /api/nodes/scheduling`), so a selector that failed open reddens the spec instead of hiding in it.
- **The fan-out carrier is proven under real binaries by `cluster_fanout_test.go`.** Both specs run two frontends and no workers against one PostgreSQL, publish at frontend 0 and read at frontend 1, and cost 7.0s together. They drive a gallery operation on purpose: `GET /models/jobs/<id>` answers from `galleryop`'s in-memory statuses map, which on a peer replica is filled by the `gallery.*.progress` subscriber and by nothing else (`Hydrate` runs once at startup, long before these operations exist), so a peer that answers was told over the carrier. Every other broadcast family has a durable table behind it that a peer would converge through anyway, and a spec on one of those cannot separate "the broadcast arrived" from "the row was read". The specs make that checkable rather than argued: the `gallery_operations` row is written when the gallery worker DEQUEUES an operation, so an operation still queued has none, and both specs assert zero rows while the peer is already answering with the operation's own bytes. Holding the queue is what `cluster.Options.Galleries` is for: it points every frontend at one gated index and turns the startup estimate warmer off, so the operation the spec submits is parked inside a fetch only the spec can complete. The second spec adds the oversized path: an element name of 9000 bytes puts the encoded notification past PostgreSQL's 8000-byte cap (exclusive), so the broadcast becomes a `bus_messages` row plus a notification carrying its id, and the spec asserts the row exists, decodes it, and compares its element name byte for byte with what frontend 1 answers. A one-byte control operation in the same run is asserted to leave NO row, which is what keeps "a row exists" a statement about size rather than one that is true of every operation.
- **Do not shorten the cluster suite's waits.** Three of its six specs sit at ~167s each because they wait out a 60s staleness threshold plus a 15s health-check tick. That wait is what stops the assertions from passing before the system could have reacted, which was a real false green earlier on. If the job has to get faster, the levers are CI concurrency or making the thresholds configurable, not shorter waits.
@@ -153,7 +153,7 @@ This is worth more than it looks. Measured over the week to 2026-07-30, **97% of
The volume is real: 13 gallery-only PRs merged that week with 10 open at once, and 78 of the 137 PRs opened were bot-generated.
`paths-ignore` on the PR trigger of `image-pr.yml` (7 jobs), `build-test.yaml` (3), `lint.yml` (2) and `tests-e2e.yml` (1) drops 13 of those 20. The excluded set:
`paths-ignore` on the PR trigger of `image-pr.yml` (7 jobs), `build-test.yaml` (3), `lint.yml` (2) and `tests-e2e.yml` (1) drops 13 of those 20, measured before `tests-e2e-distributed.yml` (2 jobs) landed. That workflow carries the same exclusion set for the same reason: its dependency graph is 99 packages, so an allowlist of paths would silently stop guarding the moment code moved, while a diff confined to the paths below provably cannot reach it. The excluded set:
| Path | Why no image or Go build can see it |
|---|---|
@@ -192,7 +192,7 @@ What still runs, and why it has to:
Two properties this relies on:
-`paths-ignore` skips a run only when **every** changed file matches, so a PR touching the gallery *and* Go code still runs everything. That is what makes the exclusion safe rather than a hole.
-`master` carries no branch protection and no rulesets, so a skipped workflow reports no status and nothing waits on it. If required status checks are ever introduced, these four entries must be excluded from the required set or PRs will hang on "Expected — Waiting for status to be reported".
-`master` carries no branch protection and no rulesets, so a skipped workflow reports no status and nothing waits on it. If required status checks are ever introduced, these five entries must be excluded from the required set or PRs will hang on "Expected — Waiting for status to be reported".
### `image.yml` on master push is gated too, by a job rather than a path filter
@@ -94,4 +94,4 @@ The user chose KISS. Every mutating tool has a safety rule (`prompts/10_safety.m
## Distributed mode
The in-memory MCP server runs only on the head node (where the chat handler runs). `inproc.Client` wraps services that are already distributed-aware (`GalleryService` coordinates with workers; `ListNodes` reads the NATS-populated registry). No NATS routing of MCP tools — the admin surface lives on the head, period.
The in-memory MCP server runs only on the head node (where the chat handler runs). `inproc.Client` wraps services that are already distributed-aware (`GalleryService` coordinates with workers; `ListNodes` reads the PostgreSQL-backed node registry). MCP tools are not routed anywhere: the admin surface lives on the head, period.
@@ -8,7 +8,7 @@ Human contributors: see [CONTRIBUTING.md](CONTRIBUTING.md) for the development w
LocalAI follows the Linux kernel project's [guidelines for AI coding assistants](https://docs.kernel.org/process/coding-assistants.html). Before submitting AI-assisted code, read [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md). Key rules:
- **No `Signed-off-by` from AI.** Only the human submitter may sign off on the Developer Certificate of Origin.
- **No `Signed-off-by` from AI.** Only the human submitter may sign off on the Developer Certificate of Origin. One exception: automation a maintainer operates signs off with *that maintainer's* identity, since no other human submitter exists to certify it. See [.agents/ai-coding-assistants.md](.agents/ai-coding-assistants.md).
- **No `Co-Authored-By: <AI>` trailers.** The human contributor owns the change.
- **Use an `Assisted-by:` trailer** to attribute AI involvement. Format: `Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2]`.
- **The human submitter is responsible** for reviewing, testing, and understanding every line of generated code.
@@ -218,7 +218,7 @@ LocalAI follows the **same guidelines as the Linux kernel project** for AI-assis
The full policy for this repository lives in [`.agents/ai-coding-assistants.md`](.agents/ai-coding-assistants.md). Summary:
- **AI agents MUST NOT add `Signed-off-by` tags.** Only humans can certify the Developer Certificate of Origin.
- **AI agents MUST NOT add `Signed-off-by` tags.** Only humans can certify the Developer Certificate of Origin. Automation operated by a maintainer is the one exception: it signs off with that maintainer's identity, because there is no other human submitter to certify it.
- **AI agents MUST NOT add `Co-Authored-By` trailers** attributing themselves as co-authors.
- **Attribute AI involvement with an `Assisted-by` trailer** in the commit message:
@@ -265,6 +265,37 @@ The e2e tests run LocalAI in a Docker container and exercise the API:
make test-e2e
```
### Running distributed-mode tests
Distributed mode (several frontend replicas, worker nodes and PostgreSQL) has two suites. Both bring up their PostgreSQL with testcontainers, so Docker has to be available. There is no message broker to bring up: a distributed deployment needs PostgreSQL and the frontends' own HTTP listener, and nothing else.
```bash
make test-e2e-distributed # in-process: services wired directly into the test binary
make test-e2e-cluster # process-level: real local-ai child processes
```
`make test-e2e-distributed` is the fast one (231 specs in roughly 2m20s). It starts one PostgreSQL for the whole run and gives each spec its own database. It runs each spec exactly once, with no retry: `DISTRIBUTED_TEST_FLAKES` defaults to 1 and feeds ginkgo's `--flake-attempts`, which counts *total attempts*, not retries. That is deliberately below the repo-wide `TEST_FLAKES=5`, because this suite exists to catch nondeterministic cluster behaviour and a retry hides exactly the failure it is meant to catch. Raise it locally when bisecting something unrelated.
`make test-e2e-cluster` runs `local-ai` as real child processes, one per frontend replica and one per worker, so a spec can kill a replica and assert what the survivors do. Budget about 13m30s (one measurement: 802.5s of specs in a 13m26s target run, which includes building the binary): several of its 21 specs wait out real staleness and health-check windows. An earlier record of 6 specs and roughly 509s is superseded. It needs a built binary and the mock backend:
```bash
make build build-mock-backend
make test-e2e-cluster
```
Two environment variables steer it:
| Variable | Purpose |
|---|---|
| `LOCALAI_E2E_BINARY` | path to the `local-ai` binary (default: `local-ai` in the repository root) |
| `LOCALAI_E2E_LOG_DIR` | directory for the per-process logs (default: a Ginkgo temp dir) |
Set `LOCALAI_E2E_LOG_DIR` when debugging. A cluster failure is unreadable without the individual frontend and worker logs, and Ginkgo only tells you which assertion failed.
A missing binary skips the cluster specs locally but fails them whenever `CI` is set, so a build problem cannot turn the CI job green without ever starting a cluster. `LOCALAI_E2E_REQUIRE_BINARIES=1` forces that failing behaviour anywhere; `LOCALAI_E2E_REQUIRE_BINARIES=0` forces the skip back on even under CI.
Both suites run in `.github/workflows/tests-e2e-distributed.yml`, on pull requests and on every push to `master`. The `paths-ignore` filter is on the pull-request trigger only, so a master push always runs them.
### React UI tests and coverage
The React UI (`core/http/react-ui/`) is covered by Playwright e2e specs, gated by a **monotonic line-coverage ratchet** (`make test-ui-coverage-check`, run in CI). The metric is non-deterministic — a fast local box reads higher than a slow CI runner for the same code — so a small tolerance is unavoidable.
cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os darwin -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)
cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os darwin -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME) --app-version $(LAUNCHER_APP_VERSION)
cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os linux -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)-linux && mv LocalAI.tar.xz ../../$(LAUNCHER_BINARY_NAME)-linux.tar.xz
cd cmd/launcher && go run fyne.io/tools/cmd/fyne@latest package -os linux -icon ../../core/http/static/logo.png --executable $(LAUNCHER_BINARY_NAME)-linux --app-version $(LAUNCHER_APP_VERSION)&& mv LocalAI.tar.xz ../../$(LAUNCHER_BINARY_NAME)-linux.tar.xz
returnfmt.Errorf("the distributed scheduler was built with no source of worker absence: it would place work on workers that have gone away and never demote one")
}
if!health.ReadsAbsence(){
returnfmt.Errorf("the node health monitor was built with no source of worker absence: it would report a worker whose tunnel is gone as healthy indefinitely")
returnnil,fmt.Errorf("the agent control client was built with no instance id: every MCP call would relay through a peer even for a worker whose tunnel this replica holds")
}
ifregistry==nil||conns==nil{
returnnil,fmt.Errorf("the agent control client was built with no way to find a connected agent worker")
}
ifcontrol==nil{
returnnil,fmt.Errorf("the agent control client was built with no control transport to reach an agent worker over")
returnnil,fmt.Errorf("wiring staging broadcasts: no broadcast carrier, so a staging transfer would show a progress bar only on the replica performing it")
}
iftracker==nil{
returnnil,nil
}
returntracker.SetBroadcaster(bus)
}
// wirePrefixCacheBroadcasts builds the cross-frontend prefix-cache layer on the
// carrier and subscribes it to peers, after refusing a configuration whose
// observations could not travel in a notification.
returnfmt.Errorf("sizing a prefix-cache observation: %w",err)
}
if!fits{
returnfmt.Errorf("the prefix-cache depth is too large to broadcast: an observation for %d blocks does not fit in one notification, so every request whose prefix changed would write a row and every replica would read it back on the inference path",cfg.MaxDepth)
returnnil,fmt.Errorf("the job dispatch loop was built with no instance id: its claims could not be told from ones a dead replica left")
}
ifregistry==nil||conns==nil{
returnnil,fmt.Errorf("the job dispatch loop was built with no way to find a connected agent worker")
}
ifbroadcast==nil{
returnnil,fmt.Errorf("the job dispatch loop was built with no broadcaster: every job would run with its progress and its result reaching no SSE stream in the deployment")
// Not fatal, and the cost is worth stating exactly rather than as
// "peers cannot reach it", because it is larger than that now.
//
// Without a row in the instances table this replica is not a live
// owner as far as Registry.Owner is concerned: that read joins a
// connection against a live instance, so a worker whose tunnel lands
// HERE is answered as unroutable at every OTHER replica, for as long
// as it stays here. This replica serves that worker perfectly well
// itself; nobody else can. On N replicas behind round robin that is
// (N-1)/N of the traffic for that worker.
//
// It does not refuse to START. Refusing would take out every existing
// single-host deployment, whose route to a local database is loopback
// and which has no peers to be unreachable by; the deployments this
// hurts are multi-replica ones, and telling those two apart at startup
// is a change with its own design and its own specs rather than a line
// here.
//
// What it does not get to do is stay quiet. One startup line scrolls
// away in seconds and the cost is paid for the whole life of the
// process, on a symptom (workers that 5xx from most of the fleet) whose
// obvious reading is "the worker is broken". So this is an ERROR, not a
// warning, and nagUnadvertisedReplica below repeats it for as long as
// the state lasts, naming the workers it is currently costing.
//
// It now costs the OTHER direction too, which is why the line says so.
// A peer link is authenticated by the dialling replica's own credential,
// published in the instances row this replica never writes, so this
// replica cannot dial a peer either: its own relayed requests are
// refused as an unproven identity rather than merely arriving nowhere.
xlog.Error("This replica is not registered in the cluster: no advertised address. Peers cannot reach it, any worker whose tunnel lands here will be unroutable from every other replica, and this replica cannot relay OUT either, because a peer link is authenticated by the credential an instance row publishes and this replica has no row",
returnfmt.Errorf("wiring departure evictions: no departure notifier, so a departed node would keep every per-node cache entry it has for the life of the process")
}
ifrouter==nil{
returnfmt.Errorf("wiring departure evictions: no router, so a departed node would keep its probe-freshness entries and its staging operations")
}
ifgallery==nil{
returnfmt.Errorf("wiring departure evictions: no gallery service, so a departed node would stay in every open operation's per-node breakdown")
}
ifcontrol==nil{
returnfmt.Errorf("wiring departure evictions: no control client, so a departed node would keep its cached HTTP client and that client's idle streams on a tunnel that is gone")
}
ifstager==nil{
returnfmt.Errorf("wiring departure evictions: no file stager, so a departed node would keep the cached HTTP client its transfers ran on")
}
// S1. Inside a nil check and not inside the prefix-cache-enabled block, so
// that "the disabled deployment registers nothing" is a fact a spec can
// hold rather than a property of where a line was written.
returnfmt.Errorf("distributed mode was initialized without a broadcast carrier: nothing could be published between replicas, and the PostgreSQL session it pins could not be closed on shutdown")
}
returnnil
}
// Broadcast is the ONE place a wiring site gets the deployment's fan-out
// carrier, and it exists so that "this family travels on the broadcast carrier
// and not on NATS" is decided once instead of at every adopter.
//
// It was five field reads before this: the fine-tune service, the quantization
// service, the agent-task setter (twice, on two startup paths), the per-user
// services manager and the Open Responses store. Every one of them takes a
// messaging.Broadcaster, and the struct used to carry a second field that
// satisfied it, so a site that reached for the wrong one compiled, started,
// published and was delivered onto a carrier almost nothing read. That second
// field is gone with the last family that needed a bus. Collapsing the choice
// to one function is what keeps it a fact a spec can pin.
//
// The return is the interface and not *pgbus.Bus on purpose: handing a nil
// *pgbus.Bus to an adopter would produce a non-nil interface wrapping a nil
// pointer, and every adopter reads a nil carrier as "standalone, do not
// broadcast". A typed nil would instead panic on the first Set. initDistributed
// already refuses to return a deployment with no carrier (see
// requireBroadcastCarrier), so the nil branch here is belt and braces for a
// logUnroutableWorkers says what the state costs RIGHT NOW.
//
// The two cases are kept apart because they call for different urgency and an
// operator can tell them apart at a glance. With no worker held this is a
// misconfiguration that has not been paid for yet; with workers held, every one
// of them is named, because "which worker is broken" is the question the
// symptom sends an operator to ask and the answer is that none of them is.
funclogUnroutableWorkers(held[]string){
iflen(held)==0{
xlog.Warn("This replica is still not registered in the cluster: no advertised address. No worker holds a tunnel here yet; the first that does will be unroutable from every other replica",
"knob","LOCALAI_DISTRIBUTED_ADVERTISE_ADDR")
return
}
xlog.Error("This replica is not registered in the cluster and holds worker tunnels: those workers are unroutable from every OTHER replica, and requests for their models fail there with no route. The workers are healthy; this replica is invisible",
// A nil check on the CONCRETE pointer, before it is widened. Once it is a
// messaging.Broadcaster a nil *pgbus.Bus is a non-nil interface holding a
// nil pointer, so every guard downstream reads it as a carrier that is
// present and every publish through it panics on a request instead.
ifbus==nil{
returnnil,nil,nil,fmt.Errorf("the job and agent fan-out bridges were built with no broadcast carrier: every job's progress and every agent's events would reach no SSE stream in the deployment")
}
ifcanceller==nil{
returnnil,nil,nil,fmt.Errorf("the agent event bridge was built with no way to cancel a worker-run agent: every cancel of one would be sent nowhere")
Loaded 100 of 678 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
