Exercise normal cluster shutdown through the real binary signal handlers and retain the loaded mock backend PID so the conformance scenario proves teardown leaves no orphan. Keep SIGKILL as a bounded fallback and leave the explicit crash primitives unchanged.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Register the new quantization, mock fixture, and protocol inventory tests through package-consistent Ginkgo suites. Keep table case names in Ginkgo diagnostics and label the protocol inventory for distributed CI selection.
Refresh the current-master fixture baseline using the final test structure.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
The cluster target identifies feature and machine-auth scenarios while
retaining one shared binary build. CI requires binaries, so neither
scenario can skip silently.
Record the current-master fixture baseline and measured local suite
budgets. Keep new conformance tests within the branch lint policy.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Keep pending agent workers alive with authenticated heartbeats while preserving approval as the tunnel and execution boundary. Exercise the real binary credential handoff through an authenticated inference and verify that credential remains non-admin.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Discover direct FileStagingClient backend overrides from every non-test source file in the nodes package, and locate that package without relying on runtime source paths so coverage inventory checks survive trimpath builds.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Reconcile the binary conformance inventory with the live backend interfaces, generated gRPC descriptor, staging overrides, and topology table so protocol growth cannot pass without an explicit coverage decision.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Separate hosts cannot use frontend paths directly. Stage the remaining path-bearing requests and the original model config.
Async quantization cleanup can span replicas and requests. Persist each cleanup phase so failures retry without fetching completed output twice.
Reject root aliases and untrusted export entries before they can mutate frontend or worker files.
The compiled cluster suite now exercises these paths through both public routes and the authenticated peer relay.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Quantization can outlive the frontend request that stages its model. Shared staging state lets another replica fetch the result and clean every remote artifact after a terminal update.
Unique output directories prevent concurrent exports and quantization jobs from sharing stale files.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Distributed workers cannot write to controller-local export and
quantization directories. Allocate model and data output directories on the
worker, then fetch their nested results through the active tunnel.
The binary conformance suite now proves all backend routes across a tunnel
owner and a peer relay with exact staged fixture bytes.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Keep inpainting request inputs in private ephemeral storage and publish only the completed image. Extend the real-binary feature matrix with canonical remeshing and typed protocol assertions, and preserve multipart diarization language hints.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Exercise the remaining public backend routes through real frontend and worker binaries, and keep fixture staging limits scoped to this conformance deployment. Fix inpainting artifacts so returned generated-image URLs resolve through the configured static mount.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Run one real frontend and worker against deterministic fixtures. The owner tunnel now covers every public backend route in this task.
Store requests exposed a missing config revision. Forward the persisted revision so distributed routing accepts store models.
The harness pins staging capacity because quota-backed CI filesystems can report misleading host totals.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Worker setup now uses a focused drawer without interrupting fleet
monitoring. Both generated worker commands retain their NATS settings.
Log views preserve their launch context and show one-based replica
labels. Bulk controls disclose selections outside the current view.
Assisted-by: Codex:gpt-5 Playwright ESLint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The failure fixture constructed a second registry with its restricted role. That role cannot create the cluster sequence, so the spec failed during migration instead of the slot lookup.
Keep migration on the admin handle and replace only the registry runtime handle after setup.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Treat an empty backend.stop response as the legacy success acknowledgement while preserving explicit worker failures and transport errors.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Allow automatic stale-node handling to advance unhealthy nodes to offline while preserving unhealthy status when automatic handling is disabled.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
The binary conformance design now covers a WebUI-authenticated deployment. It distinguishes browser sessions, shared registration credentials, and per-node tunnel credentials, including rejection and approval paths.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
The existing process tests prove chat routing and model transfer, but file-bearing backend operations rely on in-process coverage. Define the binary deployment shape, feature inventory, fixture contract, master baseline, and CI repair boundary before implementation.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
The disabled-state card renders one frontend command. Check its count and
required flags without waiting for a second command that does not exist.
Worker commands remain covered by the enabled-state test.
Assisted-by: Codex:gpt-6
Carry prefix-cache pressure and reported residency over PostgreSQL, keep truthful backend stop replies, and retain capacity-aware S3 staging release over the worker control tunnel after the rebase.
Adapt the overlapping master tests to the tunnel-native protocol and cover cross-replica cache events and departed-node cleanup.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Resolve the existing model directory before choosing a missing file. On
macOS, resolving the missing leaf fails and otherwise compares an aliased
path with a canonical allowed directory. Keep this test on the Upload
error branch instead of the containment rejection branch.
Assisted-by: Codex:GPT-6
Resolve allowed directories before comparing them with resolved files.
Otherwise staging rejects valid files under macOS temporary paths.
Cover aliased roots, sibling paths, and symlinks escaping the root.
Assisted-by: Codex:gpt-6
The general e2e target builds a container, not the host binary that
cluster specs require. Run only its own package; dedicated distributed
and cluster targets retain the full subpackage coverage.
Assisted-by: Codex:GPT-6
The mutex round the realtime transport double added a snapshot accessor for each
recorded slice. Only the event one has a caller, so make lint refuses the build:
realtime_doubles_test.go:64:25: func (*fakeTransport).recordedAudio is unused (unused)
No spec has ever read the audio log, before the mutex or after it, so the
accessor is deleted rather than nolinted and the struct comment says where the
next one comes from. audioLog stays written, because a double that silently
discarded what a coordinator sent it would be a different double.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Probing every candidate before handing it out is right, and it made sixteen
specs that were never about the kernel depend on it. They build a supervisor
directly, name the ports they expect literally, and those literals sit inside
Linux's default ephemeral range, so with the real probe each one asks this host
whether 50051 is bindable at that instant.
The first full -race run over ./core/... and ./pkg/... after the probe landed
went red on four of them, and holding 50051, 50052, 50060 and 50061 from
another process turns eleven red deterministically. Nothing was wrong with the
allocator in either case: something else on the machine held a port, which is
the situation the probe exists to survive.
So the specs that assert bookkeeping now inject a probe that always says yes,
and say why once. The two specs that are about the probe leave the field unset
and keep asking real sockets, which is what still fails if the probe is removed.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
None was introduced by this branch and all three are in test code, which is
what made them survive: every suite passed on every run and only the race
detector said otherwise. A known-failing -race run is worse than a noisy one,
because a real race raised by production code lands in the same report and is
read as one of these.
galleryop: gatedModelManager guarded the recorded names and not the gate
channel itself. A spec frees the parked worker by closing the gate and
installing a fresh one, on the spec goroutine, while the worker goroutine reads
the field to park on it. The channel is now read and replaced under the same
mutex, and cleanup closes idempotently.
pkg/model: two specs swapped xlog's package logger to capture output and
swapped it back on cleanup. xlog.SetLogger writes an unsynchronised global, so
the restore raced with the backend process watcher, which logs while a process
is stopping; the captured bytes.Buffer was written by that goroutine and read
by an Eventually at the same time. SetLogger is now called once for the whole
test binary, from init, before a goroutine exists to race with, and a spec
swaps the DESTINATION under a mutex through a routing slog.Handler. Per-spec
level filtering is preserved deliberately: one of these specs asserts that a
debug emission is filtered OUT and would pass vacuously against a handler that
recorded everything.
openai: fakeTransport appended to its event and audio logs from the response
and turn coordinators' goroutines while a spec ranged over them. Both are
behind a mutex and are read through snapshot accessors; the fields are renamed
so a raw read from another spec file does not compile.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The claim queue's attempts counter grew without bound and nothing read it. At
the default two-second poll a permanently undispatchable row cost about 43000
UPDATEs a day, and it cost more than writes: rows are claimed oldest first, so
the oldest stuck row was re-claimed ahead of every newer one on every tick and
held a dispatch slot while it failed. One poison row starved the queue behind
it.
No dead letter, and that is the decision rather than the omission. Read
settleClaim: the only outcome that releases a claim is one where NOTHING was
learned about the work. No agent worker was connected, the tunnel broke, a peer
could not be reached, the stream was refused before the request body left this
replica. Not one of those is a worker saying it ran the job and it failed, and
an attempt ceiling would turn "the fleet was away long enough" into a job
failure nobody reported, which is the collapse this whole design exists to
prevent pointed at work instead of at nodes. The one verdict available here,
that no build of any worker serves this kind, is already settled as an answer.
So the retry stays unbounded and the RATE does not. Each release stamps the row
with the earliest it may be claimed again, doubling from two seconds to a cap
of sixty, computed in the release statement from the row's own attempts count
and stamped on the DATABASE clock, because that is the clock competing replicas
order the queue on. Queued work becomes claimable again within one cap of the
fleet returning, and a stuck row no longer holds the head of the queue. A claim
released by the reap carries no delay at all: that work was never handed to
anyone, so there is nothing to back off from.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A frontend keeps two caches of one http.Client per worker: the control
client's, built on the first verb issued to a node, and the HTTP file stager's,
built on the first file staged to it. Both are keyed by node ID and neither was
ever pruned. Their own comments said so and named what a fix would need, a
signal that a node has left, which did not exist when they were written and
does now.
The map slot is the smaller half. Each entry holds an http.Transport whose idle
connections are streams on that worker's tunnel, kept until IdleConnTimeout
even after the tunnel is gone, so ForgetNode closes them rather than leaving
them to the collector. Both are registered on the deployment's one departure
notifier, as two subscribers and not one: a node can be in either cache without
being in the other, and a single hook would say only that some client was kept.
ForgetNode is on the FileStager interface rather than on the one implementation
that has state to drop, so a stager that grows a per-node map later cannot be
added without answering the question, and so registerDepartureEvictions can
take a FileStager and still fail to compile if the registration is deleted. The
S3 stager's is a documented no-op that deliberately does not forward to the
control client, which registers itself: forwarding would evict a cache it does
not own, twice per departure, and the second drop would not appear in the
subscriber names the wiring spec reads.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
HealthMonitor.misses holds one consecutive-failed-probe count per (node,
model, replica) and nothing ever removed an entry whose row had gone. It is
the only per-node state in a frontend that grows on model churn rather than on
fleet size, so a deployment that loads and unloads models for months
accumulates an integer per tuple it ever probed and gives none back.
There are four ways a row stops being visible to the pass, not one. A node
departs and the pass skips its probes; a node goes offline or unhealthy on a
stale heartbeat and the pass skips it entirely; an operator sets a node
draining; or the row is removed by an unload, a scale-down or an eviction, and
nothing tells this monitor.
So the bound is the pass itself, and not a subscription on the departure
notifier. The notifier evicts the caches a DEPARTURE invalidates and it keeps
that one meaning; this reads a different fact, that there is no longer a row
to count misses against, and covers all four cases with one rule. A row the
pass could not probe is marked seen before the probe, so an unreachable worker
still leaves its streak exactly as it was rather than having it forgiven; a
pass that could not list the fleet prunes nothing, since it observed nothing.
Forgetting only ever delays a reap by up to the miss threshold and can never
cause one.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The backend port allocator allocated from its own bookkeeping alone. That
bookkeeping records what this worker did with a port, and the collision it
cannot see is with something this worker never did: the default base port is
50051, inside Linux's default ephemeral range of 32768 to 60999, so the kernel
hands ports in this range to outbound connections and to anything that binds
port 0. A backend handed one of those dies on bind, and the frontend sees a
backend that will not start.
Every candidate is now probed by binding the exact address the backend will
listen on, in all four allocation branches: the key's own port, the free pool,
a grown port and a stolen one. Probing the free pool matters as much as
probing a grown port, because a port this worker released is exactly as
available to the kernel as one it never used.
A candidate that fails the probe is quarantined rather than blacklisted, since
whatever holds it is usually an ephemeral connection that gives it back, and
its affinity claim is dropped so an unbindable port does not stay reserved for
the key that last held it. Exhaustion now says how many candidates were
skipped, which is what tells an operator "something else is in my range" from
"my range is too narrow".
This does not remove the race and cannot: between the probe and the child's
bind the kernel can still give the port away. It removes the far larger window
in which the allocator hands out a port the kernel gave away minutes ago,
which was the whole of the observed one-in-three harness flake. The e2e
harness comment that recorded the missing check is corrected, and the docs say
how to move the range out of the ephemeral one entirely.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Skills and RAG collections had no cross-replica invalidation, and the two
builders that would have published one were deleted earlier in this branch
because nothing called them. Wiring one now would be wrong, not merely late.
Both features are derived entirely from the frontend's own state directory.
A skills.Service indexes <state dir>/skills, a collections backend enumerates
<state dir>/collections and holds one handle per collection it found there,
and no replica reads or writes another replica's copy of either. In
distributed mode PostgreSQL carries a skill's NAME and description in
skills_metadata, and nothing else: Get, Search, Export and the resource verbs
all read local files. So a peer told to drop a cache entry would rebuild it
from a directory that does not hold the change. For a postgres-engine
collection it would be worse than a no-op, since re-deriving one on a replica
with no local index file yields a collection that answers with an empty file
list against a populated vector store. What is missing is shared storage, not
a broadcast.
Recorded rather than left silent: the two cache fields say why nothing
invalidates them, a distributed frontend logs the limitation once at startup,
and the docs name the two deployments that avoid it. The new spec pins the
premise, so a change that moved either directory onto storage every replica
mounts reddens and the decision gets taken again.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
GET /api/cluster/peer authenticated with the deployment's shared
registration token and took the dialling replica's id from ?id= on trust.
Every worker holds that token, so anything holding it could open a peer
link as any replica: relay through it to every worker tunnel that replica
owns, displace a real replica's inbound link by declaring its id, and
point the roughly 31 GiB per-session receive window at one replica.
Validating the id against the instances table does not fix this, because
the attack declares a real replica's id. So the route now checks two
credentials and needs both. The shared token still says the dialler
belongs to this deployment; a new per-replica credential says which
replica it is.
The credential follows the per-node worker credential rather than
inventing a second mechanism: crypto/rand.Text, stored only as a hex
SHA-256, compared in constant time, with no fallback to the shared token.
It differs in the stronger direction. A worker's credential is minted by
the frontend and handed over once; a replica writes its own instances
row, so it mints its own secret, publishes only the hash in the same
statement that publishes its address, and never sends the plaintext
anywhere but the peer dial.
A peer that presents no credential is refused, not waved through. An old
replica and an attacker holding the shared token send the same request,
so accepting the first accepts the second; there is no safe downgrade
here, only a quiet one. The refusal is made loud instead, on both sides,
naming the upgrade rather than the network. On the documented
frontend-first order a new replica still dials an old one; an old replica
cannot dial a new one, which costs relayed requests that land on a
not-yet-restarted replica and surfaces as no route, never as absence.
A rejected peer gets its own sentinel, ErrPeerRejected, whose unwrap
chain carries ErrPeerUnreachable as well and no absence sentinel at all.
Keeping the older sentinel means no existing consumer changes behaviour;
the cause stays out of the chain, so absence cannot escape through it and
nothing can read an authorization failure as a worker that went away.
One consequence beyond the fix: a replica with no advertised address has
no instances row, so it now cannot dial out either. It was already
unreachable inward. The startup error and the docs say so.
Registry.Register, NewMembership, NewPeerPool, PeerHandler and
RegisterClusterRoutes all gained required arguments, so the identity
cannot be dropped without a compile failure.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Removing the broker left one thing carrying every broadcast family in the
product: PostgreSQL LISTEN/NOTIFY, in core/services/pgbus. It is covered
thoroughly in process by test-e2e-distributed, and it was covered nowhere at
all by real binaries: grepping the six Cluster spec files for pgbus,
bus_messages, LISTEN and NOTIFY returned zero hits. Registration, model staging
over the tunnel and inference through both the owner and the relay paths were
already proven by real processes; the carrier that now carries everything else
was not, so a deployment whose replicas each published to themselves and heard
nobody would have left every suite green.
Two specs, both on two frontends and no workers against one PostgreSQL,
publishing at frontend 0 and reading at frontend 1.
1. A gallery operation admitted at one replica, read out of the other, with the
queued state observed before the terminal one.
2. A broadcast of about 9.3 kilobytes, which PostgreSQL refuses as a
notification payload, making the round trip byte for byte through the
bus_messages spill table.
The family is a gallery operation for one property nothing else on this carrier
has: the answer a peer gives is held in memory ALONE. GET /models/jobs/<id>
reads galleryop's statuses map, which on a peer is filled by the
gallery.*.progress subscriber and by nothing else, because the only other
filler, Hydrate, runs once at startup and every operation here is created long
afterwards. Every other family has a durable table behind it that a peer would
converge through anyway, and a spec on one of those cannot separate "the
broadcast arrived" from "the row was read".
That is then made checkable rather than argued. The gallery_operations row is
written when the gallery worker DEQUEUES an operation, so an operation still
waiting in the queue has NO row, and both specs assert zero rows while the peer
is already answering with the operation's own bytes. Both also read the
instances table and require the reading replica to be a different live instance
from the publishing one, so "the other replica" cannot decay into a spelling of
"this replica".
Holding the queue is what cluster.Options.Galleries is for. The gallery worker
runs one operation at a time on an unbuffered channel, so an install parked
inside a gated index fetch parks everything behind it; without that the
admission broadcast and the terminal one are separated by two database round
trips and no HTTP poller could see between them. The option also turns the
startup estimate warmer off, because a second fetcher filling the process-wide
index cache would leave the operation never blocking and the spec passing on an
ordering nothing enforced.
The spill spec is written against a failure this branch has shipped three
times: a size-limit spec that cannot fail. The oversized body is an ordinary
element name that the real consumer decodes and surfaces, so it is not a body
the decoder would have refused at any size. The size is ABSOLUTE at 9000 bytes
rather than derived from the cap, and a one-byte control operation in the same
run is required to leave no spill row, so moving the 8000-byte cap in either
direction reddens the spec. pgbus.FitsInline, which shares its encoder and its
comparison with Publish, is asked about both payloads and must answer
differently. The spilled row is then decoded and its element name compared byte
for byte against what frontend 1 answers.
The terminal assertion in spec 1 does not re-check the element name: a terminal
status does not carry one, because updateError in galleryop.Start builds a
fresh OpStatus holding only the error. It asserts the two fields that status
does carry, in the relation that one place writes them.
Attacks run, each alone, each reverted, each behaving as predicted. Neutering
the pg_notify in pgbus.Publish so every replica knows only what it did itself
reddens both specs at frontend 1, which answers 500 for an operation it was
never told about; the bus_messages row assertion still passes under it, which
is right, since the row is written before the notification. Releasing the queue
gate reddens spec 1 at the gallery_operations count, because the operation is
dequeued and the row appears. Shrinking the oversized name to 100 bytes reddens
spec 2 at FitsInline; inverting that guard so the run reaches the row check
reddens it there instead, with no bus_messages row written, which is what makes
the row a statement about size.
test-e2e-cluster is 26 specs in 933.8 seconds of Ginkgo time, 15m37s wall. The
two additions cost 7.0 seconds together, 5.0s and 2.0s: they start no workers,
so they pay for no registration, and what they wait on is a broadcast rather
than a threshold. test-e2e-distributed is unchanged at 223 plus 8 specs, 130.6
seconds. The budget comment and .agents/building-and-testing.md move from 24
specs at 897 to 907 seconds to 26 at 933.8.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Tasks 1 to 17 are proven by unit and integration specs and by two e2e passes
taken mid-flight. This is the pass that boots the real binaries with every
carrier in place and none of the old one, and it does so on the topology the
feature was built for rather than on the one-worker shape the rest of the
cluster suite uses.
Two frontends and two workers is the configuration that matters. With each
worker's tunnel landing on a different replica, the owner path and the relay
path are live at the same instant against one roster, one scheduler and one
health monitor, so a routing mistake has somewhere to show up instead of
hiding. It is also the only shape in which "killing a replica re-homes only ITS
worker" can be stated at all.
Three scenarios, all 2x2:
1. Both workers served from both replicas. No broker as a property of the
ARTIFACT (debug/buildinfo reports no github.com/nats-io module, with the
module count asserted non-zero so a stripped binary cannot pass vacuously),
no broker in either worker's live /proc environment, and no advertised
address on either worker. One completion over the owner path and one over
the relay, plus the mirror image through the other replica, plus four
control-plane listings covering both paths for both workers.
2. The replica owning worker 0's tunnel is killed with that tunnel blocked.
Leg 1 asserts nothing and only waits for the killed instance to leave the
live set, because before that it still reads as a live owner and the
scenario is not yet about absence. Leg 2 then holds a window inside the
reconnect grace requiring that nothing acted on the absence. Leg 3 requires
the re-home and inference again. Worker 1 keeps serving throughout.
3. The suite-wide negative control. Both tunnel dials refused while
registration and heartbeats flow, both workers refused at both replicas
naming the routing fact and not a departure, nothing reaped and both
heartbeats fresh. Then ONE tunnel is restored and exactly one worker
recovers while the other stays refused.
Which worker served is read back from node_models rather than assumed: the two
models are pinned to one worker each through PUT /api/nodes/:id/labels and
POST /api/nodes/scheduling, and every assertion requires the model to be on the
expected node AND absent from the other. That the relay hit a non-owner is read
from the production Owner query before the request and re-read after it.
Attacks run, each alone, each reverted, each behaving as predicted:
hand a worker a broker URL reddens scenario 1's environment leg; point the
module check at gorm.io reddens its artifact leg; start one worker instead of
two reddens all three at the topology guard; delete the relay in
WorkerDialer.Dial reddens scenario 1 on exactly the request sent to the
non-owner while 2 and 3 stay green; a one-nanosecond reconnect grace reddens
scenario 2's leg 2 on the demotion while 1 and 3 stay green; lifting both
blocks at scenario 3's differential reddens its "still unreachable" half. The
brief's "restore the NatsURL validation" attack cannot be applied:
DistributedConfig has no such field left to validate.
Label-orphan arithmetic, counting non-skipped It nodes from --dry-run:
all 256, dist 231, cluster 24, vllm 1, and 231 + 24 + 1 = 256, so no spec is
orphaned by the label filters.
Three test-e2e-cluster runs: 897.0s, 897.8s and 906.6s of Ginkgo time, 24 specs,
15 minutes wall. The only failure across the three was a pre-existing spec dying
at cluster.Start with frontend-1 exiting status 2, which passed in the other two
and is reported as a port-allocation flake rather than a regression.
test-e2e-distributed is 223 plus 8 specs in 131.7s. The budget comment and
.agents/building-and-testing.md move from 21 specs at 800 to 830 seconds to 24
specs at 897 to 907.
The harness gains ProcessEnviron, which reads /proc for any of the three process
families; WorkerEnviron and FrontendEnviron become wrappers rather than being
deleted, so the specs that call them are not re-aimed for a rename.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
Distributed mode has not dialled a message broker since the control plane
moved onto the workers' own outward tunnels and every fan-out family moved
onto PostgreSQL LISTEN/NOTIFY. What was left was the dependency itself, and
the code that existed only to feed it.
Dropped from go.mod: nats-io/jwt/v2, nats-io/nats.go, nats-io/nkeys,
nats-io/nuid and testcontainers-go/modules/nats, along with the fourteen
indirect requires that only the NATS testcontainer pulled in. go.sum carries
no nats line either, so the removal is not the partial kind where the require
goes and the checksum stays.
Deleted with them: pkg/natsauth in full, the broker client's remaining
options and TLS files, the per-node JWT minting on both the register and the
approve path, and the natsauth.Config parameter threaded through the node
routes. The credential manager is renamed and stripped rather than deleted,
because it still holds the tunnel token that every re-registration rotates.
The bus flags stay accepted and ignored, and are now hidden, on every command
that had them, so an existing unit file, compose file or Helm values file
still starts on the day of the upgrade. What is not kept is the validation
that REQUIRED one: a distributed frontend started with no bus URL is no
longer fatal. The TLS paths lose type:"existingfile" deliberately, so a
certificate deleted along with the broker cannot fail a startup.
One operator-visible behaviour change: --nats-require-auth no longer makes an
agent worker wait through admin approval. Ask for that wait with
--distributed-require-auth, which already implied it. It is documented in the
migration section and pinned from both sides.
A deployment now needs PostgreSQL and the frontends' own HTTP listener, and
nothing else.
coverage-baseline.txt moves from 54.2 to 62.0.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Every carrier had already moved and no process opened a bus connection, but
the surface an operator reads still described a deployment with a broker in
it: a compose service, a 220-line credential-generation script, two CI steps
pulling a container nothing started, two flag tables offering --nats-url, an
architecture diagram with a NATS box wired to the workers, a join-command
generator in the Nodes page that emitted --nats-url for agent workers, and a
test suite that stood a NATS server up for specs that no longer used it.
That is the one way this programme could still fail invisibly. Every test
passes, every binary works, and every production deployment goes on running
and paying for infrastructure that carries nothing.
Nothing in this repository starts a NATS server any more. The compose file is
four services, the docs say to shut the broker down and what to keep, and the
e2e suite runs on one PostgreSQL container.
The three LOCALAI_NATS_*_TIMEOUT env vars are KEPT, and are now documented
twice as being kept. They were never broker settings: each names a control-RPC
budget the frontend applies to a worker, still read and still enforced. They
carry the prefix only because they arrived with the bus, and renaming them
would break every existing deployment for cosmetics.
The agent worker's join command was the last surface still emitting the flag,
two tasks after the agent worker stopped dialling. The Playwright spec that
covered it asserted the opposite of what is now true, so it is inverted rather
than deleted, and it reads the rendered command string rather than the
component's variables: the variables are what the fix removes, so a spec
reading them would have stopped compiling instead of failing, and a compile
error is not evidence about what an operator is shown.
nats_jwt_test.go and its helpers are deleted. They pinned a real server
ENFORCING the minted permissions. The CONTENT of those allow lists is still
pinned, untouched, by pkg/natsauth's own suites, including the spec that
refuses to let the agent lists go empty, since an empty allow list in NATS
means unrestricted. The enforcement half is retired rather than moved:
enforcement is a property of a connection, and nothing opens one.
The suite's own NATS container goes with them, which the brief left for the
next task. Removing the pre-pull while BeforeSuite still ran the image would
have defeated the step rather than cleaned it up, and this change removes the
last reader of TestInfra.NC. agent_native_executor_test.go and
mcp_ci_job_test.go are moved onto infra.Bus() instead of deleted: they were
the last two specs building a bridge and a dispatcher on a client nobody uses,
which is exactly the drift TestInfra.Bus's own comment warns about.
cluster.Options.NatsURL is now fed a deliberately dead address rather than a
live container's. Frontends and agent workers still receive LOCALAI_NATS_URL,
because that is the coverage for the promise that an existing command line
still starts; sourcing it from a running server would have let a regression
that actually dialled it pass. The control in cluster_control_test.go keeps
its assertion and loses its explanation, which claimed the deployment had a
bus and no longer could.
One latent spec race surfaced and is fixed: the background-run spec waited for
a COUNT of events and then read a snapshot for the terminal status, which is
the last event of a run and therefore always arrives after the count is met.
Its immediate twin had already been fixed this way. Nothing in production
changed.
pkg/natsauth keeps its files. It is reachable from production only through the
natsauth.Config parameter thread, and that thread is the next task's.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Task 4 gave agent workers tunnels and deliberately left the NodeType skip in
HealthMonitor.tunnelDeparted, with a spec asserting that an agent node whose
presence reader answers PresenceGone is NOT marked unhealthy. That spec was
scaffolding. It was true while an agent worker took its jobs and its verbs over
the message bus: a departure row for one said nothing about whether it could
work, and an early bug in the new tunnel client could otherwise have demoted a
fleet of healthy agent workers.
There is no bus. An agent worker is reachable through its tunnel and through
nothing else, so a departed agent tunnel means exactly what a departed backend
tunnel means: no live replica holds it, the departure has outlived the reconnect
grace, and that is a routing fact the scheduler and a reaper may act on. The
skip would now hide the only symptom an unreachable agent worker has. This is
the deliberate removal Task 4's M6 predicted, and task-4-report.md is where that
mutation already stands recorded red against the spec this commit deletes.
The skip existed at ONE site. router_liveness.go has none: its candidates come
from queries that already filter node_type = 'backend'. The two skips in
managers_distributed.go stay, because an agent worker still runs no backend
processes, so it has no backend to list and no backend op to apply.
Two node types can depart now, which is why the second half exists. Before this,
one type could depart and every per-node cache a departure left stale was
dropped from wherever its owner happened to notice, so a reader could not tell
which caches a demotion invalidated by reading the demotion path. Departure gets
ONE notification point. DepartureNotifier is edge triggered, because the monitor
runs on a ticker and a departed node stays departed; its subscribers are NAMED,
because what has to be caught is a forgotten cache and a count can say only that
one of four is missing; and NewHealthMonitor takes it as a required positional
argument, so a caller that does not pass one fails to compile.
Four caches subscribe: prefix-cache affinity in every model, probe freshness at
every address, in-flight staging operations, and the per-node breakdown of every
open gallery operation. The prefix-cache one is registered only when
prefix-cache routing is enabled, so --distributed-prefix-cache=false stays a
true no-op. The notification carries the node's name as well as its id, because
the staging tracker keys on the name and the other two key on the id, and a
subscriber should not have to read the registry from inside an eviction hook.
A departure notification is an act on absence, so it fires only on the routing
fact. A tunnel lost inside the grace, a worker that never dialled, a presence
query that failed and a stale heartbeat all announce nothing, asserted per node
type. The stale-heartbeat branch is excluded on purpose: it already marks the
node offline, which deletes its rows and runs the registry's replica-removed
hooks, so firing there too would double-evict and make the notification mean two
different things at its subscribers.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
agent.<name>.cancel was the last family on a message bus, and the only
reason an agent worker dialled one. Its subscriber is the worker running
the execution, and a worker has no database, so the family could not move
to the PostgreSQL fan-out carrier: a cancel published there would reach no
worker while reporting that it had been sent.
It is a control verb now. An agent worker mounts workerctl.PathAgentCancel
on the loopback control plane behind its tunnel and applies the cancel to
the same registry the executor registers a run on. The frontend issues it
through nodes.AgentControlClient.CancelAgentRun.
That call is a FAN-OUT and not a pick, because nothing records which worker
holds a given execution: the claim row names the claiming replica, and it
is deleted when the run ends. Every agent worker a live replica can reach
is asked over its own tunnel, relayed by the peer mesh when a peer holds
it, and each worker answers only for itself.
The answers stay apart, which is why this family was held back. A cancel a
worker made is nil. A cancel some worker could not be asked is
ErrAgentCancelUndelivered, which is neither a refusal nor a missing run. A
cancel every reachable worker declined to own is ErrAgentRunNotOnAnyWorker.
A deployment with no agent worker is ErrNoAgentWorker. Neither new sentinel
wraps ErrWorkerUnroutable and neither is a worker answer, so nothing is
reaped, demoted or evicted because of a cancel.
A worker in the ABSENT CONNECTION condition, one whose tunnel was lost
inside the reconnect grace, counts as undelivered. It is not retried in the
call and not queued: a retry would spend a budget the caller did not
choose, and a queue would need durable state whose only consumer is a run
whose control stream went with the tunnel. A worker whose departure has
outlived the grace is the one routing fact a caller may act on and is
excluded, or a single retired agent node would make every cancel
undelivered for ever.
The fan-out reads a different node set from the pick. A draining worker
takes no new work but is still finishing what it holds, so it is offered
the cancel; a pending one is refused by the tunnel route on every dial and
is not.
With that, nothing in LocalAI connects to NATS. The agent worker's dial,
its credential ladder and its refresh loop are gone, and so is the
frontend's cancel carrier. LOCALAI_NATS_URL is accepted and ignored
everywhere, and distributed mode no longer requires it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Nothing in the tree publishes, subscribes, queue-subscribes or requests through
the MessagingClient interface any more, so it is deleted rather than shrunk to
Broadcaster: two exported names for one method set in one package is an
invitation for the next author to pick whichever the surrounding file already
imported.
$ grep -rn 'messaging\.MessagingClient' --include='*.go' .
core/services/syncstate/syncstate.go:54: // It is messaging.Broadcaster rather than messaging.MessagingClient because
(one hit, a comment; no live referent. The naive grep in the plan also matches
prose and the local test type names fakeMessagingClient and
countingMessagingClient, so it can never be empty.)
*messaging.Client is shrunk to exactly Broadcaster plus its own lifecycle.
QueueSubscribe, QueueSubscribeReply, SubscribeReply, Request, Conn and the
package helpers QueueSubscribeJSON and RequestJSON go with it; none had a
production caller. Deleting the methods rather than only the call sites is what
makes putting a family back on this carrier a build error instead of a line that
compiles, publishes successfully, and is delivered onto a carrier the deployment
is being taken off. Conn is in that list because while it existed every other
name was one c.Conn().X() away; the flush-and-verdict that its real consumers
needed is now ConfirmRoundTrip, which keeps the NATS JWT permission specs armed.
The client, its options and its TLS plumbing are NOT deleted, and both processes
stay on the bus. agent.<name>.cancel is the one fan-out family that could not
move: its only subscriber is the agent worker, which has no database and cannot
join the PostgreSQL carrier at all, so a cancel published there would reach no
worker and be reported as sent. The frontend passes the client to
newFanoutBridges as its cancelCarrier and the worker subscribes on it, so
--nats-url stays required on agent-worker. Both go with the tunnel cancel verb.
The struct field is renamed Nats -> CancelCarrier to say what it is for, and
agentpool loses the messaging.Publisher it held only to be non-nil: it never
published on it, and it was gating whether a frontend runs agents distributed or
in an in-process pool. Retiring the bus would have flipped every replica back to
the in-process pool silently. The gate now reads the agent store, which is the
dependency the mode actually requires.
Also deletes four subject builders with no production publisher
(SubjectFineTuneProgress, SubjectFineTuneCancel, SubjectCacheInvalidateSkills,
SubjectCacheInvalidateCollection), the queue and request/reply halves of the
shared test double, and the e2e specs that were their only callers. Every
surviving subject is now pinned to its exact literal, because a subject is a
cross-version wire format and a rename that looks internal stops half a fleet
hearing the other half.
Docs: distributed-mode.md and cli-reference.md no longer claim NATS carries the
agent-worker job subjects, the frontend's cross-replica events, or an agent
worker's real work.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Gallery progress and cancel, the operation cache's start and end, the model
and backend cache invalidations, staging progress, and the prefix cache's
observations and invalidations now travel on the LISTEN/NOTIFY carrier. No
subject is published or subscribed on messaging.Client anywhere in the tree,
which is what makes retiring that package a deletion rather than a migration:
$ grep -rn 'natsClient\.Publish\|nats\.Publish\|\.Nats\.Publish\|QueueSubscribe\|SubscribeReply\|\.Request(' \
--include='*.go' core/ pkg/ | grep -v _test \
| grep -v 'c\.Request()\|ctx\.Request()\|Request()\.Context' \
| grep -v 'core/services/testutil/fakebus.go'
core/services/messaging/client.go:168,170,172,227,234,236,250,252,254,268,269,287
core/services/messaging/interfaces.go:21,22,23
Every remaining hit is inside core/services/messaging itself. The production
reads of the NATS client are now three, all of them the documented agent-worker
exception: Close on shutdown, the agent pool's publisher, and the agent-cancel
carrier passed to newFanoutBridges.
Prefix-cache observations publish like every other family rather than through a
method that refuses a message too large for a notification. The plan proposed
such a refusal on the reasoning that a long prompt makes a chain of thousands of
entries; ExtractChain caps a chain at Config.MaxDepth blocks, MaxDepth is a
constant with no operator knob, and the chain reaching Sync.Observe has one
source, the router's own extraction hook. A worst-case observation is a few
kilobytes against an 8000-byte cap, so the hot-path spill the refusal was
designed to avoid cannot occur, and shipping it would have added the programme's
only deliberate message drop to guard a condition that cannot arise. pgbus gains
FitsInline instead, a predicate that shares one size decision with Publish and
decides nothing, and core/application refuses at startup to wire a prefix cache
whose configured depth would put every observation over the cap.
The carrier choice is no longer stated at four sites. StagingTracker.SetPublisher
and SubscribeBroadcasts become one SetBroadcaster, so a tracker that publishes
where its peers are not listening cannot be spelled; prefixcache.Sync gains
SubscribeBroadcasts, which reads the carrier it publishes on; and the gallery
service and the operation cache are wired by methods on DistributedServices that
name no carrier at all, so the NATS client beside it cannot be handed over.
OpCache.SetMessagingClient and GalleryService.SetNATSClient are renamed to
SetBroadcaster so a missed call site fails to compile.
Two pre-existing defects that the two-real-carrier specs surfaced are fixed. A
progress tick published before a cancel and delivered after it cleared Cancelled
and left the operation reading as still running on that replica; mergeStatus now
drops a stale tick rather than merging it. GetStatus and GetAllStatus handed out
the stored OpStatus pointer while the broadcast subscribers mutated it in place,
so an /api/operations response could be marshalled mid-write; both now copy.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Five of the six families whose subscriber is an open HTTP response rather
than a process-lifetime cache now travel on pgbus: jobs.<id>.progress,
jobs.<id>.result, jobs.<id>.cancel, agent.<name>.events.<user> and
responses.<id>.cancel. Both ends of each move together, so there is no
state where a publisher is on one carrier and its subscriber on the other.
agent.<name>.cancel does NOT move, and the plan was wrong about why. Its
only subscriber in the tree is the agent worker, which has no database and
so cannot join the PostgreSQL carrier at all. Publishing that cancel on
pgbus would have lost every cancel of a worker-run agent while returning
nil, which reports a cancel that reached nobody as a cancel that was sent.
EventBridge now names its cancel carrier separately, a frontend replica
sets it to the carrier the worker reads, and it stays there until a cancel
rides the worker's tunnel like every other verb addressed to a worker.
The carrier drops at 256 rather than blocking, which is not safe on its own
for a result: a lost result has no successor message. It is not the only
path. The claiming replica persists the terminal line before it releases
the claim, and an open progress stream re-reads the job row once after
subscribing and then periodically, so a dropped terminal broadcast costs
promptness and never the answer.
Both per-request subscriptions close in a defer instead of on one return
path, and pgbus grows Subscribers() so the leak they would otherwise cause
can be asserted. It has no other symptom: only the first subscriber of a
channel issues a LISTEN, so a leaked filter just adds one closure per
notification for every stream the replica has ever served. Subscribe now
issues its LISTEN before it registers, which makes that count a readiness
signal rather than a figure to compare against itself.
Two rules that were stated at several sites and pinned at none are now one
each. The re-broadcaster is built beside the dispatcher and the bridge and
handed to the dispatch loop, so no line is left that can point it at a
carrier nobody subscribes to while every spec stays green. The set of
statuses a job never leaves is one exported set that the SSE bridge and the
store both read. The last hand-written subject filter in production code
became messaging.SubjectAgentEventsWildcard.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
syncstate.Config held one carrier field typed as the NATS client, so a
pgbus.Bus could not be handed to a SyncedMap at all: it satisfies
messaging.Broadcaster and not MessagingClient. The durable re-hydration
path built for the responses map therefore had a NATS-only consumer and
nothing in the build said so.
The field becomes Bus messaging.Broadcaster, SubscribeJSON moves to its
own file and relaxes its parameter to Broadcaster, and the four adopters
fan out over PostgreSQL LISTEN/NOTIFY: fine-tune jobs, quantization jobs,
agent tasks with their per-tenant children, and Open Responses metadata.
A new spec proves it on a real database, over two Bus instances on two
pinned listener connections: a Set and a Delete carry, a payload past the
8000-byte notification cap comes back byte identical through the spill
row, two families sharing one LISTEN channel stay separate, and a
terminated listener re-hydrates a row written while it was gone.
The five sites that each chose a carrier for an adopter are collapsed
into one DistributedServices.Broadcast() accessor. Five field reads were
five chances to leave one family on NATS with nothing failing, because
messaging.Client satisfies Broadcaster too. The accessor also refuses to
hand out a nil pgbus.Bus wrapped in a non-nil interface, which every
adopter would read as "broadcast" and dereference on the first Set.
SetTaskSyncNATS and SetJobSyncNATS are renamed to SetTaskSyncBus and
SetJobSyncBus so a missed wiring site fails to compile.
The response metadata table gains a retention of its own, defaulting to
24 hours. It inherited the Open Responses store TTL, which defaults to 0
meaning no expiration. Zero is defensible for a map that dies with the
process and is not for a table: the table grew for the life of the
deployment and a restarting replica re-hydrated every response the
cluster had ever created. A row that names its own expiry is still judged
on that column alone, and "this row is dead" now has one SQL spelling
that PurgeExpired deletes by and ListUnexpired is the negation of, so a
hydrate cannot resurrect what a sweep has already retired.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Every AgentJobService built its tasks SyncedMap with the name "agent.tasks",
and there is one service per user. So every tenant published on and subscribed
to the same subject, state.agent-tasks.delta, and SyncedMap.apply scopes
nothing: a task tenant A created was written into tenant B's in-memory map on
every replica, and ListTasks reads that map. Nothing repaired it short of a
process restart.
The subject now carries the tenant in a token of its own,
state.<name>.<tenant>.delta. Four tokens where the unscoped builder makes
three, deliberately: SubjectMatches compares token count before anything else,
so a tenant's subject and the cluster-wide one cannot cross-match, and neither
can two tenants. Putting the tenant inside the name token would not do that,
because the sanitizer folds '.' to '-' and the only filter that could then span
tenants is state.*.delta, which spans every other family too.
The rule is stated once. subscribeFilters calls publishSubject rather than
restating the subject, so a map cannot end up publishing scoped and subscribing
unscoped, which would leak exactly as before while every publish assertion
passed. The one case that decides on its own is the cluster-wide administrative
view: it hydrates from every tenant's rows, so it also takes the per-tenant
wildcard, or it would be stale the moment any tenant wrote. A tenant hydrates
from its own rows and applies only its own deltas. PerTenant defaults to false,
so finetune, quantization and the responses store keep the subject they have.
The second half of the same defect was the delete. taskStoreAdapter.Delete
called DeleteTask(id) and JobStore deleted by primary key with no user
predicate, reachable from DELETE /api/agent/tasks/:id, which takes the id off
the URL. A tenant who learned another tenant's task id destroyed that tenant's
row. The user id now travels with the id and lands as a user_id predicate.
Empty stays the administrative any-owner scope, the same thing an empty id
already means for ListTasks and ListJobs. A foreign delete removes nothing and
returns no error: not yours and not there are the same answer to the caller,
and neither is a store failure.
SetUserID rebuilds the tasks map for the same reason SetTaskSyncNATS does.
GetJobs happens to set the user id first, nothing enforced it, and with the
order reversed the map would be built with an empty tenant and put that user's
tasks back on the cluster-wide subject.
Both halves predate this programme; they are surfaced here rather than caused.
Neither is fully closed for a deployment with the agent pool off, where the
task routes are still served by one cluster-wide service that every
authenticated caller shares; that is a separate gap and it is documented.
testutil.FakeBus grew a real defect this was the first change to trip:
Unsubscribe matched on the filter string, so with two subscribers on one
filter, closing one deafened the other. Subscriptions now carry an id.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Ettore Di Giacinto