Master added Animate3D, negative_prompt, and context_size after this
branch diverged. The old suite did not exercise those paths, and Kokoros
no longer implemented the generated service trait.
Extend binary conformance across the tunnel owner and peer relay. Allow
long development versions so rebased binaries can register in PostgreSQL.
Clear the security findings introduced by the branch's new code.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
The frame length conversion is already bounded by the guard three lines
above it, which caps a payload at 1024 bytes and so cannot truncate a
uint16. The agent pick spreads work across workers that are equivalent by
construction, and predicting it buys an attacker nothing.
Both carry a #nosec naming the rule and the reason rather than a silent
suppression, matching the rest of the tree.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Tighten the permissions that can be tightened and justify the ones that
cannot. The e2e harness creates its directories 0750 and writes its model
config and process logs 0600; the copied binary keeps 0755 because the
suite executes it.
Everything else is a gosec rule firing on code that is doing what it is
supposed to: jitter drawn from math/rand for a reconnect backoff, a fixed
credential for a throwaway cluster, and subprocesses launched from the
binary the suite just built. Each carries a #nosec naming the rule and
the reason, in the form the tree already uses.
Scanned before and after: 13 findings in the packages this branch added,
now 0. The one that remains in core/services/worker is byte-identical on
master and is not this branch's to silence.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Upgrade the GGUF parser to reject string and array lengths larger than the remaining remote file before allocating them. This prevents malformed or inconsistent range responses from terminating LocalAI during background gallery warming.
Assisted-by: Codex:GPT-5 [apply_patch] [golangci-lint]
Record the authoritative Cluster label count and describe the restart helper's bounded graceful shutdown path accurately.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Exercise normal cluster shutdown through the real binary signal handlers and retain the loaded mock backend PID so the conformance scenario proves teardown leaves no orphan. Keep SIGKILL as a bounded fallback and leave the explicit crash primitives unchanged.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Register the new quantization, mock fixture, and protocol inventory tests through package-consistent Ginkgo suites. Keep table case names in Ginkgo diagnostics and label the protocol inventory for distributed CI selection.
Refresh the current-master fixture baseline using the final test structure.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
The cluster target identifies feature and machine-auth scenarios while
retaining one shared binary build. CI requires binaries, so neither
scenario can skip silently.
Record the current-master fixture baseline and measured local suite
budgets. Keep new conformance tests within the branch lint policy.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Keep pending agent workers alive with authenticated heartbeats while preserving approval as the tunnel and execution boundary. Exercise the real binary credential handoff through an authenticated inference and verify that credential remains non-admin.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Discover direct FileStagingClient backend overrides from every non-test source file in the nodes package, and locate that package without relying on runtime source paths so coverage inventory checks survive trimpath builds.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Reconcile the binary conformance inventory with the live backend interfaces, generated gRPC descriptor, staging overrides, and topology table so protocol growth cannot pass without an explicit coverage decision.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Separate hosts cannot use frontend paths directly. Stage the remaining path-bearing requests and the original model config.
Async quantization cleanup can span replicas and requests. Persist each cleanup phase so failures retry without fetching completed output twice.
Reject root aliases and untrusted export entries before they can mutate frontend or worker files.
The compiled cluster suite now exercises these paths through both public routes and the authenticated peer relay.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Quantization can outlive the frontend request that stages its model. Shared staging state lets another replica fetch the result and clean every remote artifact after a terminal update.
Unique output directories prevent concurrent exports and quantization jobs from sharing stale files.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Distributed workers cannot write to controller-local export and
quantization directories. Allocate model and data output directories on the
worker, then fetch their nested results through the active tunnel.
The binary conformance suite now proves all backend routes across a tunnel
owner and a peer relay with exact staged fixture bytes.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Keep inpainting request inputs in private ephemeral storage and publish only the completed image. Extend the real-binary feature matrix with canonical remeshing and typed protocol assertions, and preserve multipart diarization language hints.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Exercise the remaining public backend routes through real frontend and worker binaries, and keep fixture staging limits scoped to this conformance deployment. Fix inpainting artifacts so returned generated-image URLs resolve through the configured static mount.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Run one real frontend and worker against deterministic fixtures. The owner tunnel now covers every public backend route in this task.
Store requests exposed a missing config revision. Forward the persisted revision so distributed routing accepts store models.
The harness pins staging capacity because quota-backed CI filesystems can report misleading host totals.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Worker setup now uses a focused drawer without interrupting fleet
monitoring. Both generated worker commands retain their NATS settings.
Log views preserve their launch context and show one-based replica
labels. Bulk controls disclose selections outside the current view.
Assisted-by: Codex:gpt-5 Playwright ESLint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The failure fixture constructed a second registry with its restricted role. That role cannot create the cluster sequence, so the spec failed during migration instead of the slot lookup.
Keep migration on the admin handle and replace only the registry runtime handle after setup.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Treat an empty backend.stop response as the legacy success acknowledgement while preserving explicit worker failures and transport errors.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Allow automatic stale-node handling to advance unhealthy nodes to offline while preserving unhealthy status when automatic handling is disabled.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
The binary conformance design now covers a WebUI-authenticated deployment. It distinguishes browser sessions, shared registration credentials, and per-node tunnel credentials, including rejection and approval paths.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
The existing process tests prove chat routing and model transfer, but file-bearing backend operations rely on in-process coverage. Define the binary deployment shape, feature inventory, fixture contract, master baseline, and CI repair boundary before implementation.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
The disabled-state card renders one frontend command. Check its count and
required flags without waiting for a second command that does not exist.
Worker commands remain covered by the enabled-state test.
Assisted-by: Codex:gpt-6
Carry prefix-cache pressure and reported residency over PostgreSQL, keep truthful backend stop replies, and retain capacity-aware S3 staging release over the worker control tunnel after the rebase.
Adapt the overlapping master tests to the tunnel-native protocol and cover cross-replica cache events and departed-node cleanup.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Resolve the existing model directory before choosing a missing file. On
macOS, resolving the missing leaf fails and otherwise compares an aliased
path with a canonical allowed directory. Keep this test on the Upload
error branch instead of the containment rejection branch.
Assisted-by: Codex:GPT-6
Resolve allowed directories before comparing them with resolved files.
Otherwise staging rejects valid files under macOS temporary paths.
Cover aliased roots, sibling paths, and symlinks escaping the root.
Assisted-by: Codex:gpt-6
The general e2e target builds a container, not the host binary that
cluster specs require. Run only its own package; dedicated distributed
and cluster targets retain the full subpackage coverage.
Assisted-by: Codex:GPT-6
The mutex round the realtime transport double added a snapshot accessor for each
recorded slice. Only the event one has a caller, so make lint refuses the build:
realtime_doubles_test.go:64:25: func (*fakeTransport).recordedAudio is unused (unused)
No spec has ever read the audio log, before the mutex or after it, so the
accessor is deleted rather than nolinted and the struct comment says where the
next one comes from. audioLog stays written, because a double that silently
discarded what a coordinator sent it would be a different double.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Probing every candidate before handing it out is right, and it made sixteen
specs that were never about the kernel depend on it. They build a supervisor
directly, name the ports they expect literally, and those literals sit inside
Linux's default ephemeral range, so with the real probe each one asks this host
whether 50051 is bindable at that instant.
The first full -race run over ./core/... and ./pkg/... after the probe landed
went red on four of them, and holding 50051, 50052, 50060 and 50061 from
another process turns eleven red deterministically. Nothing was wrong with the
allocator in either case: something else on the machine held a port, which is
the situation the probe exists to survive.
So the specs that assert bookkeeping now inject a probe that always says yes,
and say why once. The two specs that are about the probe leave the field unset
and keep asking real sockets, which is what still fails if the probe is removed.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
None was introduced by this branch and all three are in test code, which is
what made them survive: every suite passed on every run and only the race
detector said otherwise. A known-failing -race run is worse than a noisy one,
because a real race raised by production code lands in the same report and is
read as one of these.
galleryop: gatedModelManager guarded the recorded names and not the gate
channel itself. A spec frees the parked worker by closing the gate and
installing a fresh one, on the spec goroutine, while the worker goroutine reads
the field to park on it. The channel is now read and replaced under the same
mutex, and cleanup closes idempotently.
pkg/model: two specs swapped xlog's package logger to capture output and
swapped it back on cleanup. xlog.SetLogger writes an unsynchronised global, so
the restore raced with the backend process watcher, which logs while a process
is stopping; the captured bytes.Buffer was written by that goroutine and read
by an Eventually at the same time. SetLogger is now called once for the whole
test binary, from init, before a goroutine exists to race with, and a spec
swaps the DESTINATION under a mutex through a routing slog.Handler. Per-spec
level filtering is preserved deliberately: one of these specs asserts that a
debug emission is filtered OUT and would pass vacuously against a handler that
recorded everything.
openai: fakeTransport appended to its event and audio logs from the response
and turn coordinators' goroutines while a spec ranged over them. Both are
behind a mutex and are read through snapshot accessors; the fields are renamed
so a raw read from another spec file does not compile.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The claim queue's attempts counter grew without bound and nothing read it. At
the default two-second poll a permanently undispatchable row cost about 43000
UPDATEs a day, and it cost more than writes: rows are claimed oldest first, so
the oldest stuck row was re-claimed ahead of every newer one on every tick and
held a dispatch slot while it failed. One poison row starved the queue behind
it.
No dead letter, and that is the decision rather than the omission. Read
settleClaim: the only outcome that releases a claim is one where NOTHING was
learned about the work. No agent worker was connected, the tunnel broke, a peer
could not be reached, the stream was refused before the request body left this
replica. Not one of those is a worker saying it ran the job and it failed, and
an attempt ceiling would turn "the fleet was away long enough" into a job
failure nobody reported, which is the collapse this whole design exists to
prevent pointed at work instead of at nodes. The one verdict available here,
that no build of any worker serves this kind, is already settled as an answer.
So the retry stays unbounded and the RATE does not. Each release stamps the row
with the earliest it may be claimed again, doubling from two seconds to a cap
of sixty, computed in the release statement from the row's own attempts count
and stamped on the DATABASE clock, because that is the clock competing replicas
order the queue on. Queued work becomes claimable again within one cap of the
fleet returning, and a stuck row no longer holds the head of the queue. A claim
released by the reap carries no delay at all: that work was never handed to
anyone, so there is nothing to back off from.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A frontend keeps two caches of one http.Client per worker: the control
client's, built on the first verb issued to a node, and the HTTP file stager's,
built on the first file staged to it. Both are keyed by node ID and neither was
ever pruned. Their own comments said so and named what a fix would need, a
signal that a node has left, which did not exist when they were written and
does now.
The map slot is the smaller half. Each entry holds an http.Transport whose idle
connections are streams on that worker's tunnel, kept until IdleConnTimeout
even after the tunnel is gone, so ForgetNode closes them rather than leaving
them to the collector. Both are registered on the deployment's one departure
notifier, as two subscribers and not one: a node can be in either cache without
being in the other, and a single hook would say only that some client was kept.
ForgetNode is on the FileStager interface rather than on the one implementation
that has state to drop, so a stager that grows a per-node map later cannot be
added without answering the question, and so registerDepartureEvictions can
take a FileStager and still fail to compile if the registration is deleted. The
S3 stager's is a documented no-op that deliberately does not forward to the
control client, which registers itself: forwarding would evict a cache it does
not own, twice per departure, and the second drop would not appear in the
subscriber names the wiring spec reads.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
HealthMonitor.misses holds one consecutive-failed-probe count per (node,
model, replica) and nothing ever removed an entry whose row had gone. It is
the only per-node state in a frontend that grows on model churn rather than on
fleet size, so a deployment that loads and unloads models for months
accumulates an integer per tuple it ever probed and gives none back.
There are four ways a row stops being visible to the pass, not one. A node
departs and the pass skips its probes; a node goes offline or unhealthy on a
stale heartbeat and the pass skips it entirely; an operator sets a node
draining; or the row is removed by an unload, a scale-down or an eviction, and
nothing tells this monitor.
So the bound is the pass itself, and not a subscription on the departure
notifier. The notifier evicts the caches a DEPARTURE invalidates and it keeps
that one meaning; this reads a different fact, that there is no longer a row
to count misses against, and covers all four cases with one rule. A row the
pass could not probe is marked seen before the probe, so an unreachable worker
still leaves its streak exactly as it was rather than having it forgiven; a
pass that could not list the fleet prunes nothing, since it observed nothing.
Forgetting only ever delays a reap by up to the miss threshold and can never
cause one.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The backend port allocator allocated from its own bookkeeping alone. That
bookkeeping records what this worker did with a port, and the collision it
cannot see is with something this worker never did: the default base port is
50051, inside Linux's default ephemeral range of 32768 to 60999, so the kernel
hands ports in this range to outbound connections and to anything that binds
port 0. A backend handed one of those dies on bind, and the frontend sees a
backend that will not start.
Every candidate is now probed by binding the exact address the backend will
listen on, in all four allocation branches: the key's own port, the free pool,
a grown port and a stolen one. Probing the free pool matters as much as
probing a grown port, because a port this worker released is exactly as
available to the kernel as one it never used.
A candidate that fails the probe is quarantined rather than blacklisted, since
whatever holds it is usually an ephemeral connection that gives it back, and
its affinity claim is dropped so an unbindable port does not stay reserved for
the key that last held it. Exhaustion now says how many candidates were
skipped, which is what tells an operator "something else is in my range" from
"my range is too narrow".
This does not remove the race and cannot: between the probe and the child's
bind the kernel can still give the port away. It removes the far larger window
in which the allocator hands out a port the kernel gave away minutes ago,
which was the whole of the observed one-in-three harness flake. The e2e
harness comment that recorded the missing check is corrected, and the docs say
how to move the range out of the ephemeral one entirely.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Skills and RAG collections had no cross-replica invalidation, and the two
builders that would have published one were deleted earlier in this branch
because nothing called them. Wiring one now would be wrong, not merely late.
Both features are derived entirely from the frontend's own state directory.
A skills.Service indexes <state dir>/skills, a collections backend enumerates
<state dir>/collections and holds one handle per collection it found there,
and no replica reads or writes another replica's copy of either. In
distributed mode PostgreSQL carries a skill's NAME and description in
skills_metadata, and nothing else: Get, Search, Export and the resource verbs
all read local files. So a peer told to drop a cache entry would rebuild it
from a directory that does not hold the change. For a postgres-engine
collection it would be worse than a no-op, since re-deriving one on a replica
with no local index file yields a collection that answers with an empty file
list against a populated vector store. What is missing is shared storage, not
a broadcast.
Recorded rather than left silent: the two cache fields say why nothing
invalidates them, a distributed frontend logs the limitation once at startup,
and the docs name the two deployments that avoid it. The new spec pins the
premise, so a change that moved either directory onto storage every replica
mounts reddens and the decision gets taken again.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
GET /api/cluster/peer authenticated with the deployment's shared
registration token and took the dialling replica's id from ?id= on trust.
Every worker holds that token, so anything holding it could open a peer
link as any replica: relay through it to every worker tunnel that replica
owns, displace a real replica's inbound link by declaring its id, and
point the roughly 31 GiB per-session receive window at one replica.
Validating the id against the instances table does not fix this, because
the attack declares a real replica's id. So the route now checks two
credentials and needs both. The shared token still says the dialler
belongs to this deployment; a new per-replica credential says which
replica it is.
The credential follows the per-node worker credential rather than
inventing a second mechanism: crypto/rand.Text, stored only as a hex
SHA-256, compared in constant time, with no fallback to the shared token.
It differs in the stronger direction. A worker's credential is minted by
the frontend and handed over once; a replica writes its own instances
row, so it mints its own secret, publishes only the hash in the same
statement that publishes its address, and never sends the plaintext
anywhere but the peer dial.
A peer that presents no credential is refused, not waved through. An old
replica and an attacker holding the shared token send the same request,
so accepting the first accepts the second; there is no safe downgrade
here, only a quiet one. The refusal is made loud instead, on both sides,
naming the upgrade rather than the network. On the documented
frontend-first order a new replica still dials an old one; an old replica
cannot dial a new one, which costs relayed requests that land on a
not-yet-restarted replica and surfaces as no route, never as absence.
A rejected peer gets its own sentinel, ErrPeerRejected, whose unwrap
chain carries ErrPeerUnreachable as well and no absence sentinel at all.
Keeping the older sentinel means no existing consumer changes behaviour;
the cause stays out of the chain, so absence cannot escape through it and
nothing can read an authorization failure as a worker that went away.
One consequence beyond the fix: a replica with no advertised address has
no instances row, so it now cannot dial out either. It was already
unreachable inward. The startup error and the docs say so.
Registry.Register, NewMembership, NewPeerPool, PeerHandler and
RegisterClusterRoutes all gained required arguments, so the identity
cannot be dropped without a compile failure.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Removing the broker left one thing carrying every broadcast family in the
product: PostgreSQL LISTEN/NOTIFY, in core/services/pgbus. It is covered
thoroughly in process by test-e2e-distributed, and it was covered nowhere at
all by real binaries: grepping the six Cluster spec files for pgbus,
bus_messages, LISTEN and NOTIFY returned zero hits. Registration, model staging
over the tunnel and inference through both the owner and the relay paths were
already proven by real processes; the carrier that now carries everything else
was not, so a deployment whose replicas each published to themselves and heard
nobody would have left every suite green.
Two specs, both on two frontends and no workers against one PostgreSQL,
publishing at frontend 0 and reading at frontend 1.
1. A gallery operation admitted at one replica, read out of the other, with the
queued state observed before the terminal one.
2. A broadcast of about 9.3 kilobytes, which PostgreSQL refuses as a
notification payload, making the round trip byte for byte through the
bus_messages spill table.
The family is a gallery operation for one property nothing else on this carrier
has: the answer a peer gives is held in memory ALONE. GET /models/jobs/<id>
reads galleryop's statuses map, which on a peer is filled by the
gallery.*.progress subscriber and by nothing else, because the only other
filler, Hydrate, runs once at startup and every operation here is created long
afterwards. Every other family has a durable table behind it that a peer would
converge through anyway, and a spec on one of those cannot separate "the
broadcast arrived" from "the row was read".
That is then made checkable rather than argued. The gallery_operations row is
written when the gallery worker DEQUEUES an operation, so an operation still
waiting in the queue has NO row, and both specs assert zero rows while the peer
is already answering with the operation's own bytes. Both also read the
instances table and require the reading replica to be a different live instance
from the publishing one, so "the other replica" cannot decay into a spelling of
"this replica".
Holding the queue is what cluster.Options.Galleries is for. The gallery worker
runs one operation at a time on an unbuffered channel, so an install parked
inside a gated index fetch parks everything behind it; without that the
admission broadcast and the terminal one are separated by two database round
trips and no HTTP poller could see between them. The option also turns the
startup estimate warmer off, because a second fetcher filling the process-wide
index cache would leave the operation never blocking and the spec passing on an
ordering nothing enforced.
The spill spec is written against a failure this branch has shipped three
times: a size-limit spec that cannot fail. The oversized body is an ordinary
element name that the real consumer decodes and surfaces, so it is not a body
the decoder would have refused at any size. The size is ABSOLUTE at 9000 bytes
rather than derived from the cap, and a one-byte control operation in the same
run is required to leave no spill row, so moving the 8000-byte cap in either
direction reddens the spec. pgbus.FitsInline, which shares its encoder and its
comparison with Publish, is asked about both payloads and must answer
differently. The spilled row is then decoded and its element name compared byte
for byte against what frontend 1 answers.
The terminal assertion in spec 1 does not re-check the element name: a terminal
status does not carry one, because updateError in galleryop.Start builds a
fresh OpStatus holding only the error. It asserts the two fields that status
does carry, in the relation that one place writes them.
Attacks run, each alone, each reverted, each behaving as predicted. Neutering
the pg_notify in pgbus.Publish so every replica knows only what it did itself
reddens both specs at frontend 1, which answers 500 for an operation it was
never told about; the bus_messages row assertion still passes under it, which
is right, since the row is written before the notification. Releasing the queue
gate reddens spec 1 at the gallery_operations count, because the operation is
dequeued and the row appears. Shrinking the oversized name to 100 bytes reddens
spec 2 at FitsInline; inverting that guard so the run reaches the row check
reddens it there instead, with no bus_messages row written, which is what makes
the row a statement about size.
test-e2e-cluster is 26 specs in 933.8 seconds of Ginkgo time, 15m37s wall. The
two additions cost 7.0 seconds together, 5.0s and 2.0s: they start no workers,
so they pay for no registration, and what they wait on is a broadcast rather
than a threshold. test-e2e-distributed is unchanged at 223 plus 8 specs, 130.6
seconds. The budget comment and .agents/building-and-testing.md move from 24
specs at 897 to 907 seconds to 26 at 933.8.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Tasks 1 to 17 are proven by unit and integration specs and by two e2e passes
taken mid-flight. This is the pass that boots the real binaries with every
carrier in place and none of the old one, and it does so on the topology the
feature was built for rather than on the one-worker shape the rest of the
cluster suite uses.
Two frontends and two workers is the configuration that matters. With each
worker's tunnel landing on a different replica, the owner path and the relay
path are live at the same instant against one roster, one scheduler and one
health monitor, so a routing mistake has somewhere to show up instead of
hiding. It is also the only shape in which "killing a replica re-homes only ITS
worker" can be stated at all.
Three scenarios, all 2x2:
1. Both workers served from both replicas. No broker as a property of the
ARTIFACT (debug/buildinfo reports no github.com/nats-io module, with the
module count asserted non-zero so a stripped binary cannot pass vacuously),
no broker in either worker's live /proc environment, and no advertised
address on either worker. One completion over the owner path and one over
the relay, plus the mirror image through the other replica, plus four
control-plane listings covering both paths for both workers.
2. The replica owning worker 0's tunnel is killed with that tunnel blocked.
Leg 1 asserts nothing and only waits for the killed instance to leave the
live set, because before that it still reads as a live owner and the
scenario is not yet about absence. Leg 2 then holds a window inside the
reconnect grace requiring that nothing acted on the absence. Leg 3 requires
the re-home and inference again. Worker 1 keeps serving throughout.
3. The suite-wide negative control. Both tunnel dials refused while
registration and heartbeats flow, both workers refused at both replicas
naming the routing fact and not a departure, nothing reaped and both
heartbeats fresh. Then ONE tunnel is restored and exactly one worker
recovers while the other stays refused.
Which worker served is read back from node_models rather than assumed: the two
models are pinned to one worker each through PUT /api/nodes/:id/labels and
POST /api/nodes/scheduling, and every assertion requires the model to be on the
expected node AND absent from the other. That the relay hit a non-owner is read
from the production Owner query before the request and re-read after it.
Attacks run, each alone, each reverted, each behaving as predicted:
hand a worker a broker URL reddens scenario 1's environment leg; point the
module check at gorm.io reddens its artifact leg; start one worker instead of
two reddens all three at the topology guard; delete the relay in
WorkerDialer.Dial reddens scenario 1 on exactly the request sent to the
non-owner while 2 and 3 stay green; a one-nanosecond reconnect grace reddens
scenario 2's leg 2 on the demotion while 1 and 3 stay green; lifting both
blocks at scenario 3's differential reddens its "still unreachable" half. The
brief's "restore the NatsURL validation" attack cannot be applied:
DistributedConfig has no such field left to validate.
Label-orphan arithmetic, counting non-skipped It nodes from --dry-run:
all 256, dist 231, cluster 24, vllm 1, and 231 + 24 + 1 = 256, so no spec is
orphaned by the label filters.
Three test-e2e-cluster runs: 897.0s, 897.8s and 906.6s of Ginkgo time, 24 specs,
15 minutes wall. The only failure across the three was a pre-existing spec dying
at cluster.Start with frontend-1 exiting status 2, which passed in the other two
and is reported as a port-allocation flake rather than a regression.
test-e2e-distributed is 223 plus 8 specs in 131.7s. The budget comment and
.agents/building-and-testing.md move from 21 specs at 800 to 830 seconds to 24
specs at 897 to 907.
The harness gains ProcessEnviron, which reads /proc for any of the three process
families; WorkerEnviron and FrontendEnviron become wrappers rather than being
deleted, so the specs that call them are not re-aimed for a rename.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
Distributed mode has not dialled a message broker since the control plane
moved onto the workers' own outward tunnels and every fan-out family moved
onto PostgreSQL LISTEN/NOTIFY. What was left was the dependency itself, and
the code that existed only to feed it.
Dropped from go.mod: nats-io/jwt/v2, nats-io/nats.go, nats-io/nkeys,
nats-io/nuid and testcontainers-go/modules/nats, along with the fourteen
indirect requires that only the NATS testcontainer pulled in. go.sum carries
no nats line either, so the removal is not the partial kind where the require
goes and the checksum stays.
Deleted with them: pkg/natsauth in full, the broker client's remaining
options and TLS files, the per-node JWT minting on both the register and the
approve path, and the natsauth.Config parameter threaded through the node
routes. The credential manager is renamed and stripped rather than deleted,
because it still holds the tunnel token that every re-registration rotates.
The bus flags stay accepted and ignored, and are now hidden, on every command
that had them, so an existing unit file, compose file or Helm values file
still starts on the day of the upgrade. What is not kept is the validation
that REQUIRED one: a distributed frontend started with no bus URL is no
longer fatal. The TLS paths lose type:"existingfile" deliberately, so a
certificate deleted along with the broker cannot fail a startup.
One operator-visible behaviour change: --nats-require-auth no longer makes an
agent worker wait through admin approval. Ask for that wait with
--distributed-require-auth, which already implied it. It is documented in the
migration section and pinned from both sides.
A deployment now needs PostgreSQL and the frontends' own HTTP listener, and
nothing else.
coverage-baseline.txt moves from 54.2 to 62.0.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Every carrier had already moved and no process opened a bus connection, but
the surface an operator reads still described a deployment with a broker in
it: a compose service, a 220-line credential-generation script, two CI steps
pulling a container nothing started, two flag tables offering --nats-url, an
architecture diagram with a NATS box wired to the workers, a join-command
generator in the Nodes page that emitted --nats-url for agent workers, and a
test suite that stood a NATS server up for specs that no longer used it.
That is the one way this programme could still fail invisibly. Every test
passes, every binary works, and every production deployment goes on running
and paying for infrastructure that carries nothing.
Nothing in this repository starts a NATS server any more. The compose file is
four services, the docs say to shut the broker down and what to keep, and the
e2e suite runs on one PostgreSQL container.
The three LOCALAI_NATS_*_TIMEOUT env vars are KEPT, and are now documented
twice as being kept. They were never broker settings: each names a control-RPC
budget the frontend applies to a worker, still read and still enforced. They
carry the prefix only because they arrived with the bus, and renaming them
would break every existing deployment for cosmetics.
The agent worker's join command was the last surface still emitting the flag,
two tasks after the agent worker stopped dialling. The Playwright spec that
covered it asserted the opposite of what is now true, so it is inverted rather
than deleted, and it reads the rendered command string rather than the
component's variables: the variables are what the fix removes, so a spec
reading them would have stopped compiling instead of failing, and a compile
error is not evidence about what an operator is shown.
nats_jwt_test.go and its helpers are deleted. They pinned a real server
ENFORCING the minted permissions. The CONTENT of those allow lists is still
pinned, untouched, by pkg/natsauth's own suites, including the spec that
refuses to let the agent lists go empty, since an empty allow list in NATS
means unrestricted. The enforcement half is retired rather than moved:
enforcement is a property of a connection, and nothing opens one.
The suite's own NATS container goes with them, which the brief left for the
next task. Removing the pre-pull while BeforeSuite still ran the image would
have defeated the step rather than cleaned it up, and this change removes the
last reader of TestInfra.NC. agent_native_executor_test.go and
mcp_ci_job_test.go are moved onto infra.Bus() instead of deleted: they were
the last two specs building a bridge and a dispatcher on a client nobody uses,
which is exactly the drift TestInfra.Bus's own comment warns about.
cluster.Options.NatsURL is now fed a deliberately dead address rather than a
live container's. Frontends and agent workers still receive LOCALAI_NATS_URL,
because that is the coverage for the promise that an existing command line
still starts; sourcing it from a running server would have let a regression
that actually dialled it pass. The control in cluster_control_test.go keeps
its assertion and loses its explanation, which claimed the deployment had a
bus and no longer could.
One latent spec race surfaced and is fixed: the background-run spec waited for
a COUNT of events and then read a snapshot for the terminal status, which is
the last event of a run and therefore always arrives after the count is met.
Its immediate twin had already been fixed this way. Nothing in production
changed.
pkg/natsauth keeps its files. It is reachable from production only through the
natsauth.Config parameter thread, and that thread is the next task's.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Task 4 gave agent workers tunnels and deliberately left the NodeType skip in
HealthMonitor.tunnelDeparted, with a spec asserting that an agent node whose
presence reader answers PresenceGone is NOT marked unhealthy. That spec was
scaffolding. It was true while an agent worker took its jobs and its verbs over
the message bus: a departure row for one said nothing about whether it could
work, and an early bug in the new tunnel client could otherwise have demoted a
fleet of healthy agent workers.
There is no bus. An agent worker is reachable through its tunnel and through
nothing else, so a departed agent tunnel means exactly what a departed backend
tunnel means: no live replica holds it, the departure has outlived the reconnect
grace, and that is a routing fact the scheduler and a reaper may act on. The
skip would now hide the only symptom an unreachable agent worker has. This is
the deliberate removal Task 4's M6 predicted, and task-4-report.md is where that
mutation already stands recorded red against the spec this commit deletes.
The skip existed at ONE site. router_liveness.go has none: its candidates come
from queries that already filter node_type = 'backend'. The two skips in
managers_distributed.go stay, because an agent worker still runs no backend
processes, so it has no backend to list and no backend op to apply.
Two node types can depart now, which is why the second half exists. Before this,
one type could depart and every per-node cache a departure left stale was
dropped from wherever its owner happened to notice, so a reader could not tell
which caches a demotion invalidated by reading the demotion path. Departure gets
ONE notification point. DepartureNotifier is edge triggered, because the monitor
runs on a ticker and a departed node stays departed; its subscribers are NAMED,
because what has to be caught is a forgotten cache and a count can say only that
one of four is missing; and NewHealthMonitor takes it as a required positional
argument, so a caller that does not pass one fails to compile.
Four caches subscribe: prefix-cache affinity in every model, probe freshness at
every address, in-flight staging operations, and the per-node breakdown of every
open gallery operation. The prefix-cache one is registered only when
prefix-cache routing is enabled, so --distributed-prefix-cache=false stays a
true no-op. The notification carries the node's name as well as its id, because
the staging tracker keys on the name and the other two key on the id, and a
subscriber should not have to read the registry from inside an eviction hook.
A departure notification is an act on absence, so it fires only on the routing
fact. A tunnel lost inside the grace, a worker that never dialled, a presence
query that failed and a stale heartbeat all announce nothing, asserted per node
type. The stale-heartbeat branch is excluded on purpose: it already marks the
node offline, which deletes its rows and runs the registry's replica-removed
hooks, so firing there too would double-evict and make the notification mean two
different things at its subscribers.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Ettore Di Giacinto