Code scanning reports alerts on every line of a touched file. Mark the
gRPC auth env var name and the mock backend's staged-path reads as
reviewed, and log the error when evicting a model after its connection
fails instead of dropping it.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
errcheck flagged tests/e2e/realtime_ws_test.go:307 as new on this branch
(the localai-proxy realtime failover spec) once the e2e lint scope widened.
Same house style the maintainer bot used for the other close results in
this file (19f6a18b8): wrap in a deferred closure so the error is explicitly
discarded rather than silently ignored.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
19f6a18b8 moved credential-env resolution behind an explicit lookup
(ApplicationConfig.ProxyAPIKeyEnvLookup / config.WithProxyAPIKeyEnvLookup),
wired only at the CLI boundary (core/cli/run.go). The e2e suite builds its
Application in-process without that option, so the failover prober could
never resolve a remote target's api_key_env, remote liveness never passed,
and "fails over ... and fails back" hung waiting for chain-remote to
recover. Pass config.WithProxyAPIKeyEnvLookup(os.Getenv) there too, same as
the CLI. worker/federated commands don't serve proxy/failover configs and
tests/e2e-ui never sets api_key_env, so neither needs the lookup.
Also make the misconfiguration itself easier to diagnose: the prober now
logs a one-time xlog.Warn per api_key_env when a remote target sets it but
no lookup is configured, instead of only surfacing it as a per-probe
"is unset" error indistinguishable from a genuinely empty env var.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The e2e suite now registers the localai-proxy binary and points proxy
models back at the test server itself, so a request leaves LocalAI
through the backend, returns over REST and is answered by a mock model.
Chat, embeddings, TTS and transcription through the proxy return the
upstream model's answer; a chain whose proxy target's upstream model
fails to load serves from the local target; and a realtime pipeline
whose LLM stage is a chain on a remote target completes a turn, then
switches to the local target with a localai.model.failover trip event
when a gate in front of the upstream starts answering 503.
The docs describe the localai-proxy backend next to cloud-proxy and add
a per-stage remote LocalAI example to the failover page.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Pass the API key environment lookup through ApplicationConfig to satisfy
core configuration lint. Keep credential resolution dynamic and exclude
the callback from serialization.
Handle the five close results reported by errcheck.
Assisted-by: Codex:gpt-6 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
make build-cloud-proxy-backend writes it next to the mock backend,
which is ignored; the cloud-proxy binary showed up as untracked.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A chain request reached a cloud-proxy target with the client's model,
the chain name, whenever the target set no upstream_model: passthrough
forwards the body's model and translate falls back to it. The upstream
answered 404, which neither retries nor trips, while the liveness
probe, which checks the target's own name, kept passing.
PrepareTarget now sets the upstream model of a remote target to
proxy.upstream_model or the target name, the same name the probe uses.
The request pipeline and realtime chain stages both call it.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A stage that names a chain is resolved on every call, so a switch keeps
the session and its conversation. Clients get localai.model.failover
events at session start and on every switch.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A missing file is not a reliable signal: models download on first use,
some backends need no file, and dotted names like Phi-3.5-mini look like
paths. Marking such a fallback down removed the retry a chain exists
for. Cold targets are now judged only by real requests.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Master added Animate3D, negative_prompt, and context_size after this
branch diverged. The old suite did not exercise those paths, and Kokoros
no longer implemented the generated service trait.
Extend binary conformance across the tunnel owner and peer relay. Allow
long development versions so rebased binaries can register in PostgreSQL.
Clear the security findings introduced by the branch's new code.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Tighten the permissions that can be tightened and justify the ones that
cannot. The e2e harness creates its directories 0750 and writes its model
config and process logs 0600; the copied binary keeps 0755 because the
suite executes it.
Everything else is a gosec rule firing on code that is doing what it is
supposed to: jitter drawn from math/rand for a reconnect backoff, a fixed
credential for a throwaway cluster, and subprocesses launched from the
binary the suite just built. Each carries a #nosec naming the rule and
the reason, in the form the tree already uses.
Scanned before and after: 13 findings in the packages this branch added,
now 0. The one that remains in core/services/worker is byte-identical on
master and is not this branch's to silence.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Record the authoritative Cluster label count and describe the restart helper's bounded graceful shutdown path accurately.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Exercise normal cluster shutdown through the real binary signal handlers and retain the loaded mock backend PID so the conformance scenario proves teardown leaves no orphan. Keep SIGKILL as a bounded fallback and leave the explicit crash primitives unchanged.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Register the new quantization, mock fixture, and protocol inventory tests through package-consistent Ginkgo suites. Keep table case names in Ginkgo diagnostics and label the protocol inventory for distributed CI selection.
Refresh the current-master fixture baseline using the final test structure.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
The cluster target identifies feature and machine-auth scenarios while
retaining one shared binary build. CI requires binaries, so neither
scenario can skip silently.
Record the current-master fixture baseline and measured local suite
budgets. Keep new conformance tests within the branch lint policy.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Keep pending agent workers alive with authenticated heartbeats while preserving approval as the tunnel and execution boundary. Exercise the real binary credential handoff through an authenticated inference and verify that credential remains non-admin.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Discover direct FileStagingClient backend overrides from every non-test source file in the nodes package, and locate that package without relying on runtime source paths so coverage inventory checks survive trimpath builds.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Reconcile the binary conformance inventory with the live backend interfaces, generated gRPC descriptor, staging overrides, and topology table so protocol growth cannot pass without an explicit coverage decision.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Separate hosts cannot use frontend paths directly. Stage the remaining path-bearing requests and the original model config.
Async quantization cleanup can span replicas and requests. Persist each cleanup phase so failures retry without fetching completed output twice.
Reject root aliases and untrusted export entries before they can mutate frontend or worker files.
The compiled cluster suite now exercises these paths through both public routes and the authenticated peer relay.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Quantization can outlive the frontend request that stages its model. Shared staging state lets another replica fetch the result and clean every remote artifact after a terminal update.
Unique output directories prevent concurrent exports and quantization jobs from sharing stale files.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Distributed workers cannot write to controller-local export and
quantization directories. Allocate model and data output directories on the
worker, then fetch their nested results through the active tunnel.
The binary conformance suite now proves all backend routes across a tunnel
owner and a peer relay with exact staged fixture bytes.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Keep inpainting request inputs in private ephemeral storage and publish only the completed image. Extend the real-binary feature matrix with canonical remeshing and typed protocol assertions, and preserve multipart diarization language hints.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Exercise the remaining public backend routes through real frontend and worker binaries, and keep fixture staging limits scoped to this conformance deployment. Fix inpainting artifacts so returned generated-image URLs resolve through the configured static mount.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Run one real frontend and worker against deterministic fixtures. The owner tunnel now covers every public backend route in this task.
Store requests exposed a missing config revision. Forward the persisted revision so distributed routing accepts store models.
The harness pins staging capacity because quota-backed CI filesystems can report misleading host totals.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
The failure fixture constructed a second registry with its restricted role. That role cannot create the cluster sequence, so the spec failed during migration instead of the slot lookup.
Keep migration on the admin handle and replace only the registry runtime handle after setup.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
The backend port allocator allocated from its own bookkeeping alone. That
bookkeeping records what this worker did with a port, and the collision it
cannot see is with something this worker never did: the default base port is
50051, inside Linux's default ephemeral range of 32768 to 60999, so the kernel
hands ports in this range to outbound connections and to anything that binds
port 0. A backend handed one of those dies on bind, and the frontend sees a
backend that will not start.
Every candidate is now probed by binding the exact address the backend will
listen on, in all four allocation branches: the key's own port, the free pool,
a grown port and a stolen one. Probing the free pool matters as much as
probing a grown port, because a port this worker released is exactly as
available to the kernel as one it never used.
A candidate that fails the probe is quarantined rather than blacklisted, since
whatever holds it is usually an ephemeral connection that gives it back, and
its affinity claim is dropped so an unbindable port does not stay reserved for
the key that last held it. Exhaustion now says how many candidates were
skipped, which is what tells an operator "something else is in my range" from
"my range is too narrow".
This does not remove the race and cannot: between the probe and the child's
bind the kernel can still give the port away. It removes the far larger window
in which the allocator hands out a port the kernel gave away minutes ago,
which was the whole of the observed one-in-three harness flake. The e2e
harness comment that recorded the missing check is corrected, and the docs say
how to move the range out of the ephemeral one entirely.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
GET /api/cluster/peer authenticated with the deployment's shared
registration token and took the dialling replica's id from ?id= on trust.
Every worker holds that token, so anything holding it could open a peer
link as any replica: relay through it to every worker tunnel that replica
owns, displace a real replica's inbound link by declaring its id, and
point the roughly 31 GiB per-session receive window at one replica.
Validating the id against the instances table does not fix this, because
the attack declares a real replica's id. So the route now checks two
credentials and needs both. The shared token still says the dialler
belongs to this deployment; a new per-replica credential says which
replica it is.
The credential follows the per-node worker credential rather than
inventing a second mechanism: crypto/rand.Text, stored only as a hex
SHA-256, compared in constant time, with no fallback to the shared token.
It differs in the stronger direction. A worker's credential is minted by
the frontend and handed over once; a replica writes its own instances
row, so it mints its own secret, publishes only the hash in the same
statement that publishes its address, and never sends the plaintext
anywhere but the peer dial.
A peer that presents no credential is refused, not waved through. An old
replica and an attacker holding the shared token send the same request,
so accepting the first accepts the second; there is no safe downgrade
here, only a quiet one. The refusal is made loud instead, on both sides,
naming the upgrade rather than the network. On the documented
frontend-first order a new replica still dials an old one; an old replica
cannot dial a new one, which costs relayed requests that land on a
not-yet-restarted replica and surfaces as no route, never as absence.
A rejected peer gets its own sentinel, ErrPeerRejected, whose unwrap
chain carries ErrPeerUnreachable as well and no absence sentinel at all.
Keeping the older sentinel means no existing consumer changes behaviour;
the cause stays out of the chain, so absence cannot escape through it and
nothing can read an authorization failure as a worker that went away.
One consequence beyond the fix: a replica with no advertised address has
no instances row, so it now cannot dial out either. It was already
unreachable inward. The startup error and the docs say so.
Registry.Register, NewMembership, NewPeerPool, PeerHandler and
RegisterClusterRoutes all gained required arguments, so the identity
cannot be dropped without a compile failure.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Removing the broker left one thing carrying every broadcast family in the
product: PostgreSQL LISTEN/NOTIFY, in core/services/pgbus. It is covered
thoroughly in process by test-e2e-distributed, and it was covered nowhere at
all by real binaries: grepping the six Cluster spec files for pgbus,
bus_messages, LISTEN and NOTIFY returned zero hits. Registration, model staging
over the tunnel and inference through both the owner and the relay paths were
already proven by real processes; the carrier that now carries everything else
was not, so a deployment whose replicas each published to themselves and heard
nobody would have left every suite green.
Two specs, both on two frontends and no workers against one PostgreSQL,
publishing at frontend 0 and reading at frontend 1.
1. A gallery operation admitted at one replica, read out of the other, with the
queued state observed before the terminal one.
2. A broadcast of about 9.3 kilobytes, which PostgreSQL refuses as a
notification payload, making the round trip byte for byte through the
bus_messages spill table.
The family is a gallery operation for one property nothing else on this carrier
has: the answer a peer gives is held in memory ALONE. GET /models/jobs/<id>
reads galleryop's statuses map, which on a peer is filled by the
gallery.*.progress subscriber and by nothing else, because the only other
filler, Hydrate, runs once at startup and every operation here is created long
afterwards. Every other family has a durable table behind it that a peer would
converge through anyway, and a spec on one of those cannot separate "the
broadcast arrived" from "the row was read".
That is then made checkable rather than argued. The gallery_operations row is
written when the gallery worker DEQUEUES an operation, so an operation still
waiting in the queue has NO row, and both specs assert zero rows while the peer
is already answering with the operation's own bytes. Both also read the
instances table and require the reading replica to be a different live instance
from the publishing one, so "the other replica" cannot decay into a spelling of
"this replica".
Holding the queue is what cluster.Options.Galleries is for. The gallery worker
runs one operation at a time on an unbuffered channel, so an install parked
inside a gated index fetch parks everything behind it; without that the
admission broadcast and the terminal one are separated by two database round
trips and no HTTP poller could see between them. The option also turns the
startup estimate warmer off, because a second fetcher filling the process-wide
index cache would leave the operation never blocking and the spec passing on an
ordering nothing enforced.
The spill spec is written against a failure this branch has shipped three
times: a size-limit spec that cannot fail. The oversized body is an ordinary
element name that the real consumer decodes and surfaces, so it is not a body
the decoder would have refused at any size. The size is ABSOLUTE at 9000 bytes
rather than derived from the cap, and a one-byte control operation in the same
run is required to leave no spill row, so moving the 8000-byte cap in either
direction reddens the spec. pgbus.FitsInline, which shares its encoder and its
comparison with Publish, is asked about both payloads and must answer
differently. The spilled row is then decoded and its element name compared byte
for byte against what frontend 1 answers.
The terminal assertion in spec 1 does not re-check the element name: a terminal
status does not carry one, because updateError in galleryop.Start builds a
fresh OpStatus holding only the error. It asserts the two fields that status
does carry, in the relation that one place writes them.
Attacks run, each alone, each reverted, each behaving as predicted. Neutering
the pg_notify in pgbus.Publish so every replica knows only what it did itself
reddens both specs at frontend 1, which answers 500 for an operation it was
never told about; the bus_messages row assertion still passes under it, which
is right, since the row is written before the notification. Releasing the queue
gate reddens spec 1 at the gallery_operations count, because the operation is
dequeued and the row appears. Shrinking the oversized name to 100 bytes reddens
spec 2 at FitsInline; inverting that guard so the run reaches the row check
reddens it there instead, with no bus_messages row written, which is what makes
the row a statement about size.
test-e2e-cluster is 26 specs in 933.8 seconds of Ginkgo time, 15m37s wall. The
two additions cost 7.0 seconds together, 5.0s and 2.0s: they start no workers,
so they pay for no registration, and what they wait on is a broadcast rather
than a threshold. test-e2e-distributed is unchanged at 223 plus 8 specs, 130.6
seconds. The budget comment and .agents/building-and-testing.md move from 24
specs at 897 to 907 seconds to 26 at 933.8.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Tasks 1 to 17 are proven by unit and integration specs and by two e2e passes
taken mid-flight. This is the pass that boots the real binaries with every
carrier in place and none of the old one, and it does so on the topology the
feature was built for rather than on the one-worker shape the rest of the
cluster suite uses.
Two frontends and two workers is the configuration that matters. With each
worker's tunnel landing on a different replica, the owner path and the relay
path are live at the same instant against one roster, one scheduler and one
health monitor, so a routing mistake has somewhere to show up instead of
hiding. It is also the only shape in which "killing a replica re-homes only ITS
worker" can be stated at all.
Three scenarios, all 2x2:
1. Both workers served from both replicas. No broker as a property of the
ARTIFACT (debug/buildinfo reports no github.com/nats-io module, with the
module count asserted non-zero so a stripped binary cannot pass vacuously),
no broker in either worker's live /proc environment, and no advertised
address on either worker. One completion over the owner path and one over
the relay, plus the mirror image through the other replica, plus four
control-plane listings covering both paths for both workers.
2. The replica owning worker 0's tunnel is killed with that tunnel blocked.
Leg 1 asserts nothing and only waits for the killed instance to leave the
live set, because before that it still reads as a live owner and the
scenario is not yet about absence. Leg 2 then holds a window inside the
reconnect grace requiring that nothing acted on the absence. Leg 3 requires
the re-home and inference again. Worker 1 keeps serving throughout.
3. The suite-wide negative control. Both tunnel dials refused while
registration and heartbeats flow, both workers refused at both replicas
naming the routing fact and not a departure, nothing reaped and both
heartbeats fresh. Then ONE tunnel is restored and exactly one worker
recovers while the other stays refused.
Which worker served is read back from node_models rather than assumed: the two
models are pinned to one worker each through PUT /api/nodes/:id/labels and
POST /api/nodes/scheduling, and every assertion requires the model to be on the
expected node AND absent from the other. That the relay hit a non-owner is read
from the production Owner query before the request and re-read after it.
Attacks run, each alone, each reverted, each behaving as predicted:
hand a worker a broker URL reddens scenario 1's environment leg; point the
module check at gorm.io reddens its artifact leg; start one worker instead of
two reddens all three at the topology guard; delete the relay in
WorkerDialer.Dial reddens scenario 1 on exactly the request sent to the
non-owner while 2 and 3 stay green; a one-nanosecond reconnect grace reddens
scenario 2's leg 2 on the demotion while 1 and 3 stay green; lifting both
blocks at scenario 3's differential reddens its "still unreachable" half. The
brief's "restore the NatsURL validation" attack cannot be applied:
DistributedConfig has no such field left to validate.
Label-orphan arithmetic, counting non-skipped It nodes from --dry-run:
all 256, dist 231, cluster 24, vllm 1, and 231 + 24 + 1 = 256, so no spec is
orphaned by the label filters.
Three test-e2e-cluster runs: 897.0s, 897.8s and 906.6s of Ginkgo time, 24 specs,
15 minutes wall. The only failure across the three was a pre-existing spec dying
at cluster.Start with frontend-1 exiting status 2, which passed in the other two
and is reported as a port-allocation flake rather than a regression.
test-e2e-distributed is 223 plus 8 specs in 131.7s. The budget comment and
.agents/building-and-testing.md move from 21 specs at 800 to 830 seconds to 24
specs at 897 to 907.
The harness gains ProcessEnviron, which reads /proc for any of the three process
families; WorkerEnviron and FrontendEnviron become wrappers rather than being
deleted, so the specs that call them are not re-aimed for a rename.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
Distributed mode has not dialled a message broker since the control plane
moved onto the workers' own outward tunnels and every fan-out family moved
onto PostgreSQL LISTEN/NOTIFY. What was left was the dependency itself, and
the code that existed only to feed it.
Dropped from go.mod: nats-io/jwt/v2, nats-io/nats.go, nats-io/nkeys,
nats-io/nuid and testcontainers-go/modules/nats, along with the fourteen
indirect requires that only the NATS testcontainer pulled in. go.sum carries
no nats line either, so the removal is not the partial kind where the require
goes and the checksum stays.
Deleted with them: pkg/natsauth in full, the broker client's remaining
options and TLS files, the per-node JWT minting on both the register and the
approve path, and the natsauth.Config parameter threaded through the node
routes. The credential manager is renamed and stripped rather than deleted,
because it still holds the tunnel token that every re-registration rotates.
The bus flags stay accepted and ignored, and are now hidden, on every command
that had them, so an existing unit file, compose file or Helm values file
still starts on the day of the upgrade. What is not kept is the validation
that REQUIRED one: a distributed frontend started with no bus URL is no
longer fatal. The TLS paths lose type:"existingfile" deliberately, so a
certificate deleted along with the broker cannot fail a startup.
One operator-visible behaviour change: --nats-require-auth no longer makes an
agent worker wait through admin approval. Ask for that wait with
--distributed-require-auth, which already implied it. It is documented in the
migration section and pinned from both sides.
A deployment now needs PostgreSQL and the frontends' own HTTP listener, and
nothing else.
coverage-baseline.txt moves from 54.2 to 62.0.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Every carrier had already moved and no process opened a bus connection, but
the surface an operator reads still described a deployment with a broker in
it: a compose service, a 220-line credential-generation script, two CI steps
pulling a container nothing started, two flag tables offering --nats-url, an
architecture diagram with a NATS box wired to the workers, a join-command
generator in the Nodes page that emitted --nats-url for agent workers, and a
test suite that stood a NATS server up for specs that no longer used it.
That is the one way this programme could still fail invisibly. Every test
passes, every binary works, and every production deployment goes on running
and paying for infrastructure that carries nothing.
Nothing in this repository starts a NATS server any more. The compose file is
four services, the docs say to shut the broker down and what to keep, and the
e2e suite runs on one PostgreSQL container.
The three LOCALAI_NATS_*_TIMEOUT env vars are KEPT, and are now documented
twice as being kept. They were never broker settings: each names a control-RPC
budget the frontend applies to a worker, still read and still enforced. They
carry the prefix only because they arrived with the bus, and renaming them
would break every existing deployment for cosmetics.
The agent worker's join command was the last surface still emitting the flag,
two tasks after the agent worker stopped dialling. The Playwright spec that
covered it asserted the opposite of what is now true, so it is inverted rather
than deleted, and it reads the rendered command string rather than the
component's variables: the variables are what the fix removes, so a spec
reading them would have stopped compiling instead of failing, and a compile
error is not evidence about what an operator is shown.
nats_jwt_test.go and its helpers are deleted. They pinned a real server
ENFORCING the minted permissions. The CONTENT of those allow lists is still
pinned, untouched, by pkg/natsauth's own suites, including the spec that
refuses to let the agent lists go empty, since an empty allow list in NATS
means unrestricted. The enforcement half is retired rather than moved:
enforcement is a property of a connection, and nothing opens one.
The suite's own NATS container goes with them, which the brief left for the
next task. Removing the pre-pull while BeforeSuite still ran the image would
have defeated the step rather than cleaned it up, and this change removes the
last reader of TestInfra.NC. agent_native_executor_test.go and
mcp_ci_job_test.go are moved onto infra.Bus() instead of deleted: they were
the last two specs building a bridge and a dispatcher on a client nobody uses,
which is exactly the drift TestInfra.Bus's own comment warns about.
cluster.Options.NatsURL is now fed a deliberately dead address rather than a
live container's. Frontends and agent workers still receive LOCALAI_NATS_URL,
because that is the coverage for the promise that an existing command line
still starts; sourcing it from a running server would have let a regression
that actually dialled it pass. The control in cluster_control_test.go keeps
its assertion and loses its explanation, which claimed the deployment had a
bus and no longer could.
One latent spec race surfaced and is fixed: the background-run spec waited for
a COUNT of events and then read a snapshot for the terminal status, which is
the last event of a run and therefore always arrives after the count is met.
Its immediate twin had already been fixed this way. Nothing in production
changed.
pkg/natsauth keeps its files. It is reachable from production only through the
natsauth.Config parameter thread, and that thread is the next task's.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Task 4 gave agent workers tunnels and deliberately left the NodeType skip in
HealthMonitor.tunnelDeparted, with a spec asserting that an agent node whose
presence reader answers PresenceGone is NOT marked unhealthy. That spec was
scaffolding. It was true while an agent worker took its jobs and its verbs over
the message bus: a departure row for one said nothing about whether it could
work, and an early bug in the new tunnel client could otherwise have demoted a
fleet of healthy agent workers.
There is no bus. An agent worker is reachable through its tunnel and through
nothing else, so a departed agent tunnel means exactly what a departed backend
tunnel means: no live replica holds it, the departure has outlived the reconnect
grace, and that is a routing fact the scheduler and a reaper may act on. The
skip would now hide the only symptom an unreachable agent worker has. This is
the deliberate removal Task 4's M6 predicted, and task-4-report.md is where that
mutation already stands recorded red against the spec this commit deletes.
The skip existed at ONE site. router_liveness.go has none: its candidates come
from queries that already filter node_type = 'backend'. The two skips in
managers_distributed.go stay, because an agent worker still runs no backend
processes, so it has no backend to list and no backend op to apply.
Two node types can depart now, which is why the second half exists. Before this,
one type could depart and every per-node cache a departure left stale was
dropped from wherever its owner happened to notice, so a reader could not tell
which caches a demotion invalidated by reading the demotion path. Departure gets
ONE notification point. DepartureNotifier is edge triggered, because the monitor
runs on a ticker and a departed node stays departed; its subscribers are NAMED,
because what has to be caught is a forgotten cache and a count can say only that
one of four is missing; and NewHealthMonitor takes it as a required positional
argument, so a caller that does not pass one fails to compile.
Four caches subscribe: prefix-cache affinity in every model, probe freshness at
every address, in-flight staging operations, and the per-node breakdown of every
open gallery operation. The prefix-cache one is registered only when
prefix-cache routing is enabled, so --distributed-prefix-cache=false stays a
true no-op. The notification carries the node's name as well as its id, because
the staging tracker keys on the name and the other two key on the id, and a
subscriber should not have to read the registry from inside an eviction hook.
A departure notification is an act on absence, so it fires only on the routing
fact. A tunnel lost inside the grace, a worker that never dialled, a presence
query that failed and a stale heartbeat all announce nothing, asserted per node
type. The stale-heartbeat branch is excluded on purpose: it already marks the
node offline, which deletes its rows and runs the registry's replica-removed
hooks, so firing there too would double-evict and make the notification mean two
different things at its subscribers.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
agent.<name>.cancel was the last family on a message bus, and the only
reason an agent worker dialled one. Its subscriber is the worker running
the execution, and a worker has no database, so the family could not move
to the PostgreSQL fan-out carrier: a cancel published there would reach no
worker while reporting that it had been sent.
It is a control verb now. An agent worker mounts workerctl.PathAgentCancel
on the loopback control plane behind its tunnel and applies the cancel to
the same registry the executor registers a run on. The frontend issues it
through nodes.AgentControlClient.CancelAgentRun.
That call is a FAN-OUT and not a pick, because nothing records which worker
holds a given execution: the claim row names the claiming replica, and it
is deleted when the run ends. Every agent worker a live replica can reach
is asked over its own tunnel, relayed by the peer mesh when a peer holds
it, and each worker answers only for itself.
The answers stay apart, which is why this family was held back. A cancel a
worker made is nil. A cancel some worker could not be asked is
ErrAgentCancelUndelivered, which is neither a refusal nor a missing run. A
cancel every reachable worker declined to own is ErrAgentRunNotOnAnyWorker.
A deployment with no agent worker is ErrNoAgentWorker. Neither new sentinel
wraps ErrWorkerUnroutable and neither is a worker answer, so nothing is
reaped, demoted or evicted because of a cancel.
A worker in the ABSENT CONNECTION condition, one whose tunnel was lost
inside the reconnect grace, counts as undelivered. It is not retried in the
call and not queued: a retry would spend a budget the caller did not
choose, and a queue would need durable state whose only consumer is a run
whose control stream went with the tunnel. A worker whose departure has
outlived the grace is the one routing fact a caller may act on and is
excluded, or a single retired agent node would make every cancel
undelivered for ever.
The fan-out reads a different node set from the pick. A draining worker
takes no new work but is still finishing what it holds, so it is offered
the cancel; a pending one is refused by the tunnel route on every dial and
is not.
With that, nothing in LocalAI connects to NATS. The agent worker's dial,
its credential ladder and its refresh loop are gone, and so is the
frontend's cancel carrier. LOCALAI_NATS_URL is accepted and ignored
everywhere, and distributed mode no longer requires it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Nothing in the tree publishes, subscribes, queue-subscribes or requests through
the MessagingClient interface any more, so it is deleted rather than shrunk to
Broadcaster: two exported names for one method set in one package is an
invitation for the next author to pick whichever the surrounding file already
imported.
$ grep -rn 'messaging\.MessagingClient' --include='*.go' .
core/services/syncstate/syncstate.go:54: // It is messaging.Broadcaster rather than messaging.MessagingClient because
(one hit, a comment; no live referent. The naive grep in the plan also matches
prose and the local test type names fakeMessagingClient and
countingMessagingClient, so it can never be empty.)
*messaging.Client is shrunk to exactly Broadcaster plus its own lifecycle.
QueueSubscribe, QueueSubscribeReply, SubscribeReply, Request, Conn and the
package helpers QueueSubscribeJSON and RequestJSON go with it; none had a
production caller. Deleting the methods rather than only the call sites is what
makes putting a family back on this carrier a build error instead of a line that
compiles, publishes successfully, and is delivered onto a carrier the deployment
is being taken off. Conn is in that list because while it existed every other
name was one c.Conn().X() away; the flush-and-verdict that its real consumers
needed is now ConfirmRoundTrip, which keeps the NATS JWT permission specs armed.
The client, its options and its TLS plumbing are NOT deleted, and both processes
stay on the bus. agent.<name>.cancel is the one fan-out family that could not
move: its only subscriber is the agent worker, which has no database and cannot
join the PostgreSQL carrier at all, so a cancel published there would reach no
worker and be reported as sent. The frontend passes the client to
newFanoutBridges as its cancelCarrier and the worker subscribes on it, so
--nats-url stays required on agent-worker. Both go with the tunnel cancel verb.
The struct field is renamed Nats -> CancelCarrier to say what it is for, and
agentpool loses the messaging.Publisher it held only to be non-nil: it never
published on it, and it was gating whether a frontend runs agents distributed or
in an in-process pool. Retiring the bus would have flipped every replica back to
the in-process pool silently. The gate now reads the agent store, which is the
dependency the mode actually requires.
Also deletes four subject builders with no production publisher
(SubjectFineTuneProgress, SubjectFineTuneCancel, SubjectCacheInvalidateSkills,
SubjectCacheInvalidateCollection), the queue and request/reply halves of the
shared test double, and the e2e specs that were their only callers. Every
surviving subject is now pinned to its exact literal, because a subject is a
cross-version wire format and a rename that looks internal stops half a fleet
hearing the other half.
Docs: distributed-mode.md and cli-reference.md no longer claim NATS carries the
agent-worker job subjects, the frontend's cross-replica events, or an agent
worker's real work.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Gallery progress and cancel, the operation cache's start and end, the model
and backend cache invalidations, staging progress, and the prefix cache's
observations and invalidations now travel on the LISTEN/NOTIFY carrier. No
subject is published or subscribed on messaging.Client anywhere in the tree,
which is what makes retiring that package a deletion rather than a migration:
$ grep -rn 'natsClient\.Publish\|nats\.Publish\|\.Nats\.Publish\|QueueSubscribe\|SubscribeReply\|\.Request(' \
--include='*.go' core/ pkg/ | grep -v _test \
| grep -v 'c\.Request()\|ctx\.Request()\|Request()\.Context' \
| grep -v 'core/services/testutil/fakebus.go'
core/services/messaging/client.go:168,170,172,227,234,236,250,252,254,268,269,287
core/services/messaging/interfaces.go:21,22,23
Every remaining hit is inside core/services/messaging itself. The production
reads of the NATS client are now three, all of them the documented agent-worker
exception: Close on shutdown, the agent pool's publisher, and the agent-cancel
carrier passed to newFanoutBridges.
Prefix-cache observations publish like every other family rather than through a
method that refuses a message too large for a notification. The plan proposed
such a refusal on the reasoning that a long prompt makes a chain of thousands of
entries; ExtractChain caps a chain at Config.MaxDepth blocks, MaxDepth is a
constant with no operator knob, and the chain reaching Sync.Observe has one
source, the router's own extraction hook. A worst-case observation is a few
kilobytes against an 8000-byte cap, so the hot-path spill the refusal was
designed to avoid cannot occur, and shipping it would have added the programme's
only deliberate message drop to guard a condition that cannot arise. pgbus gains
FitsInline instead, a predicate that shares one size decision with Publish and
decides nothing, and core/application refuses at startup to wire a prefix cache
whose configured depth would put every observation over the cap.
The carrier choice is no longer stated at four sites. StagingTracker.SetPublisher
and SubscribeBroadcasts become one SetBroadcaster, so a tracker that publishes
where its peers are not listening cannot be spelled; prefixcache.Sync gains
SubscribeBroadcasts, which reads the carrier it publishes on; and the gallery
service and the operation cache are wired by methods on DistributedServices that
name no carrier at all, so the NATS client beside it cannot be handed over.
OpCache.SetMessagingClient and GalleryService.SetNATSClient are renamed to
SetBroadcaster so a missed call site fails to compile.
Two pre-existing defects that the two-real-carrier specs surfaced are fixed. A
progress tick published before a cancel and delivered after it cleared Cancelled
and left the operation reading as still running on that replica; mergeStatus now
drops a stale tick rather than merging it. GetStatus and GetAllStatus handed out
the stored OpStatus pointer while the broadcast subscribers mutated it in place,
so an /api/operations response could be marshalled mid-write; both now copy.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Five of the six families whose subscriber is an open HTTP response rather
than a process-lifetime cache now travel on pgbus: jobs.<id>.progress,
jobs.<id>.result, jobs.<id>.cancel, agent.<name>.events.<user> and
responses.<id>.cancel. Both ends of each move together, so there is no
state where a publisher is on one carrier and its subscriber on the other.
agent.<name>.cancel does NOT move, and the plan was wrong about why. Its
only subscriber in the tree is the agent worker, which has no database and
so cannot join the PostgreSQL carrier at all. Publishing that cancel on
pgbus would have lost every cancel of a worker-run agent while returning
nil, which reports a cancel that reached nobody as a cancel that was sent.
EventBridge now names its cancel carrier separately, a frontend replica
sets it to the carrier the worker reads, and it stays there until a cancel
rides the worker's tunnel like every other verb addressed to a worker.
The carrier drops at 256 rather than blocking, which is not safe on its own
for a result: a lost result has no successor message. It is not the only
path. The claiming replica persists the terminal line before it releases
the claim, and an open progress stream re-reads the job row once after
subscribing and then periodically, so a dropped terminal broadcast costs
promptness and never the answer.
Both per-request subscriptions close in a defer instead of on one return
path, and pgbus grows Subscribers() so the leak they would otherwise cause
can be asserted. It has no other symptom: only the first subscriber of a
channel issues a LISTEN, so a leaked filter just adds one closure per
notification for every stream the replica has ever served. Subscribe now
issues its LISTEN before it registers, which makes that count a readiness
signal rather than a figure to compare against itself.
Two rules that were stated at several sites and pinned at none are now one
each. The re-broadcaster is built beside the dispatcher and the bridge and
handed to the dispatch loop, so no line is left that can point it at a
carrier nobody subscribes to while every spec stays green. The set of
statuses a job never leaves is one exported set that the SSE bridge and the
store both read. The last hand-written subject filter in production code
became messaging.SubjectAgentEventsWildcard.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
syncstate.Config held one carrier field typed as the NATS client, so a
pgbus.Bus could not be handed to a SyncedMap at all: it satisfies
messaging.Broadcaster and not MessagingClient. The durable re-hydration
path built for the responses map therefore had a NATS-only consumer and
nothing in the build said so.
The field becomes Bus messaging.Broadcaster, SubscribeJSON moves to its
own file and relaxes its parameter to Broadcaster, and the four adopters
fan out over PostgreSQL LISTEN/NOTIFY: fine-tune jobs, quantization jobs,
agent tasks with their per-tenant children, and Open Responses metadata.
A new spec proves it on a real database, over two Bus instances on two
pinned listener connections: a Set and a Delete carry, a payload past the
8000-byte notification cap comes back byte identical through the spill
row, two families sharing one LISTEN channel stay separate, and a
terminated listener re-hydrates a row written while it was gone.
The five sites that each chose a carrier for an adopter are collapsed
into one DistributedServices.Broadcast() accessor. Five field reads were
five chances to leave one family on NATS with nothing failing, because
messaging.Client satisfies Broadcaster too. The accessor also refuses to
hand out a nil pgbus.Bus wrapped in a non-nil interface, which every
adopter would read as "broadcast" and dereference on the first Set.
SetTaskSyncNATS and SetJobSyncNATS are renamed to SetTaskSyncBus and
SetJobSyncBus so a missed wiring site fails to compile.
The response metadata table gains a retention of its own, defaulting to
24 hours. It inherited the Open Responses store TTL, which defaults to 0
meaning no expiration. Zero is defensible for a map that dies with the
process and is not for a table: the table grew for the life of the
deployment and a restarting replica re-hydrated every response the
cluster had ever created. A row that names its own expiry is still judged
on that column alone, and "this row is dead" now has one SQL spelling
that PurgeExpired deletes by and ListUnexpired is the negation of, so a
hydrate cannot resurrect what a sweep has already retired.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The responses.metadata SyncedMap had no durable Store, so its reconnect
re-hydrate replaced nothing. That was survivable while responses converged
through deltas on a broker that mostly stayed up. It is not survivable on a
carrier whose listener is one pinned PostgreSQL session: every response created
while the subscription was down stays invisible on that replica forever, and the
symptom is a 404 from one replica and a 200 from another for the same
response_id.
State that must survive a gap now lives in a response_metadata table, and the
notification only says it changed. The map writes through on a Set and reads the
table on hydrate, on reconnect and on reconcile, so the gap closes instead of
becoming permanent.
The row carries the whole projection as JSON rather than one column per field. A
column-per-field schema would be a second definition of what a peer may act on,
and the two would drift the first time syncedResponse gained a field: the map
would broadcast the new field and hydrate without it, so a replica that had
reconnected would serve a different response body from one that had not, with
nothing failing anywhere. Only PayloadJSON is ever decoded; owner_replica and
owner are indexed copies for an operator reading the table by hand.
A missing row and an unreachable database are different facts. Every store and
adapter method returns a driver failure as an error and never as an empty
result, and syncstate replaces nothing when its source errors, so an outage
leaves the map holding what it had rather than blanking it into a cluster-wide
404.
Liveness is the database's clock, spelled expires_at IS NULL OR expires_at >
now(), because every replica hydrating from this table must agree on which rows
are live and a Go-side cutoff makes that a property of whichever process asked.
The test container shares the host clock, so no behavioural spec can tell the
two apart; the statement shape is pinned instead. The constructor refuses a
non-PostgreSQL handle, because an unguarded now() on the single-binary path
reads as a missing migration.
A ticker sweeps expired rows every five minutes on each replica, and Close waits
for it rather than racing it. Note that the sweep removes nothing while
LOCALAI_OPEN_RESPONSES_STORE_TTL is 0, which is the default: with no TTL nothing
ever expires and the table grows for the life of the deployment. The docs say so
plainly.
EnableDistributed takes the store positionally and last, so a call site that
forgets it fails to compile rather than silently restoring the deltas-only map
this change exists to replace. A nil store there is refused by name: it is
reached only from the distributed branch of route registration, so it is a
wiring bug and not a deployment shape.
What still never leaves the owning replica is unchanged: the resume buffer and
the CancelFunc. The write-through is one row per response state change, not one
per generated token.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The three NATS queue groups jobs.new, jobs.mcp-ci.new and agent.execute are
gone. Dispatching work is now a row in a work_claims table, taken by one
frontend replica with SELECT ... FOR UPDATE SKIP LOCKED and driven on an agent
worker as a streaming control RPC over that worker's tunnel. Exactly-one
delivery among competing consumers is a database problem, not a broker feature.
An agent worker has no database, so it never claims; it executes what the
claiming replica hands it.
A claim must not outlive the replica that took it. The reap releases a claim
whose owner is no longer a live replica in the instances table, on the database
clock, and never asks how long the claim has been held. A job that legitimately
runs for an hour on a heartbeating replica is left alone, while a claim whose
owner stopped heartbeating becomes claimable again within one liveness window.
A replica with no advertised address has no instances row at all, so it refuses
to claim rather than have its work reaped out from under it mid-run.
The settle rule is stated once, in settleClaim, and every exit path calls it. A
transport failure releases the claim and never completes or discards it; only a
decoded reply line completes it. That line is deliberately not
cluster.IsWorkerAnswer, which accepts the stream refusals a worker's tunnel
writes before any request body reaches its control server: completing on those
would discard work that never ran.
The terminal line is persisted before the claim is completed, so a store that
refuses leaves the claim standing rather than leaving the job running for ever.
That is the dropped-result defect fixed structurally rather than by retry.
This also surfaces a pre-existing gap rather than causing one: no worker has
ever served plain task jobs, and publishing them into an empty queue group left
them running with no trace. Such a claim is now failed with a reason.
Removes QueueWorkers, --agent-subject and --agent-queue, and narrows an agent
worker's minted JWT by agent.execute and jobs.mcp-ci.new.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Two things made every end-to-end result on this branch unreliable.
make test-e2e-cluster did not rebuild ./local-ai. It only checked that a
file by that name existed, so an edit that was never rebuilt ran the whole
suite against an older build while reporting on the working tree, and a
missing binary skipped every spec and exited 0 with "Test Suite Passed".
The target now depends on a new e2e-binary target, which is the plain go
build CI already used rather than make build, since that one also builds
the React UI this suite never touches. The harness carries the other half:
localAIBinary now FAILS, locally as well as under CI, when the binary is
older than the newest non-test Go source in the tree, which covers a run
started with LOCALAI_E2E_BINARY or by invoking ginkgo directly. Test files
are excluded from that scan because they compile into the ginkgo suite and
never into local-ai. The CI job drops its own build step and the env var so
that one place owns the build and it happens after protogen-go.
test-e2e-distributed was audited for the same hole and has none: ginkgo
compiles that suite from the working tree on every run and it execs no
prebuilt binary. build-mock-backend already rebuilt unconditionally.
"Worker tunnel under load" bounded the worst probe inside the bulk transfer
window against the worst probe under the empty-load window. A max over n
samples is a biased estimator when the two n differ, and here they always
do: the bulk window is by construction longer and draws several times as
many chances at an unrelated scheduling outlier. Anything loading the box
widens that gap, so the spec reddened on what else was running: 255ms
against a 161ms budget with make lint beside it, 57ms alone. It now bounds
the probe COMPLETION RATE instead, which is the statistic a serialised
session actually moves and a mean over dozens of samples in both terms, so
a uniform slowdown cancels in the ratio. Measured with make lint running:
0.97 direct and 0.60 relayed against a floor of 0.125.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
nodes.<id>.backend.stop was the last worker-facing NATS subject, and it existed
only because ONE publisher had not moved. An agent worker already mounted
workerctl.PathBackendStop on the tunnel it holds, and a backend worker already
took its stop there, so RemoteUnloaderAdapter branched on NodeType to pick a
carrier for a verb both kinds of worker served the same way.
The branch is gone, and with it nodeTypeOf and its NodeTypeBackend default,
which removes one of the ten NodeType branches left to sweep. The adapter loses
its messaging.MessagingClient outright rather than keeping an unused field: it
now holds no publisher, so re-routing any verb back onto the bus is a change to
the struct and to every caller of the constructor, and does not compile until
all of them agree. messaging.SubjectNodeBackendStop and subjectNodePrefix are
deleted, the agent worker's subscription with them.
pkg/natsauth drops the per-node backend.stop grant from the agent SUB list. That
is a narrowing of eleven entries to ten, never to nothing: NATS reads an EMPTY
allow list as unrestricted, so the coverage spec asserts both that the retired
subject is no longer covered and that the queue subjects an agent worker lives
on still are. The e2e half proves it against a real enforcing server: one spec
subscribes successfully on an agent-minted JWT, the next is refused the retired
subject on a JWT minted the same way.
Both halves of the old split were pinned, so both pins are re-aimed rather than
deleted, and the two node types are asserted separately rather than as one
parameterised case, because only two cases can show that the two used to differ.
Three assertions that the adapter published nothing are deleted instead: with no
publisher to hold, no change could ever redden them.
The CLI's handler set moves into agentWorkerControlHandlers so a spec can stand
it up and post to it. That wiring was a bare literal no spec pinned, and
deleting the subscription made it the ONLY carrier for backend.stop: a dropped
field would have been a 404 the frontend reads as a worker too old to serve the
verb, and nothing in the repo would have noticed.
Mutations: the agent branch restored off the control route reddens two specs;
the backend branch restored, separately, reddens five; PathBackendDelete in
place of PathBackendStop reddens nine across both node types; dropping the CLI
wiring line reddens the new wiring table; re-adding the allow-list entry reddens
the unit spec and the JWT e2e spec; and restoring the publisher for real does
not compile.
Four comments this change falsified are fixed, in core/cli, pkg/model and the
distributed-mode docs, which now say both kinds of worker serve
POST /v1/control/backend/stop and what each does with it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Two specs in the distributed e2e suite have been red since 1cf847f29, which
stopped workers advertising an address and removed every fall-back to the
node's own endpoint. After that commit a replica row must name the loopback
endpoint of its own backend process: DistributedModelStore.Range refuses to
list a replica whose backend process is unnamed, and SmartRouter treats an
unnamed warm row as naming no process and cold-loads instead.
1cf847f29 updated the unit specs under core/services/nodes for the new
contract but not tests/e2e/distributed, and the phase's closing verification
ran test-e2e-cluster rather than test-e2e-distributed, so nothing reported it.
The task brief named six later commits as candidates and called 671785621
known good; both are wrong, and 671785621 fails these two specs as well.
The fix is the scripted input, not the guard. Both specs wrote an empty
address, which is a row this release cannot produce, since installBackendOnNode
refuses an install that names no address.
The dedup spec is repaired rather than merely un-reddened. With an empty
address its DB row was dropped by the unnamed-replica guard before Range ever
consulted the seen-set, so deleting the dedup check left it green: it asserted
nothing. With the endpoint named, removing that check reddens it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
localai-org-maint-bot