mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-12 22:33:54 -04:00
fcb93b128ea19205fe1bcb3c6be22d6dfd9ed546
7823
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ettore Di Giacinto
|
fcb93b128e |
feat(distributed): let a worker name a broadcast the frontend decides on
A worker has no database and cannot NOTIFY, and it does not need to: every
message it sends is produced inside a handler the frontend invoked, so
there is always an open control response to write on. This adds the two
ends of that, and the authorization decision that sits between them, and
nothing that dispatches yet.
workerctl.Envelope gains Subject, a REQUEST and not an instruction. Empty
means the line is for this caller alone, which is what every pre-existing
progress line is, and omitempty keeps those lines byte-identical for an
older reader. It qualifies a progress line and never a reply line: a reply
is the worker's verdict about the work, and there is no version of
"publish my verdict for me" this control plane has to carry.
nodes.MayBroadcast is the replacement for pkg/natsauth's allow list, and
the inversion is the point. NATS read an EMPTY allow list as NO
RESTRICTION, which is why phase 3 refused to delete the backend branch and
spelled it {"_INBOX.>"}. This one reads an empty list, and an absent node
type, as DENY EVERYTHING, and a table-driven spec pins that by emptying
the agent entry and asserting all three of its subjects are then refused.
Matching goes through messaging.SubjectMatches, the one definition in the
tree, so a filter that fires here fires on the carrier.
nodes.Rebroadcaster.Handle returns a bool and never an error. A refused or
failed re-broadcast is logged and the RPC continues, because the RPC's
outcome is the worker's verdict about the work and a publish failure says
nothing about it. The return shape is asserted at compile time in the file
that states the rule, so changing it to an error does not compile.
ControlClient.CallStreaming's progress callback now takes the line's
subject alongside its raw bytes. The client no longer decodes progress at
all: what a line means is a question about the verb and whether a named
broadcast may be made is a question about the node, and it knows neither.
Both moved into installProgressBridge, which every one of the three
streaming call sites in unloader.go goes through. A line naming a subject
is dropped there rather than delivered as install progress, because
backend.install and backend.upgrade are a backend worker's verbs and a
backend worker is allowed no subjects.
agents.StreamPublisher is the producing end, a messaging.Publisher writing
NDJSON envelopes onto an in-flight control response and flushing each one,
so a tick reaches the frontend while the handler is still running.
Serialized, because two concurrent encodes on one http.ResponseWriter
interleave bytes and tear the framing.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Ettore Di Giacinto
|
e5ad2e18c4 |
test(e2e): build the cluster binary and drop a load-sensitive budget
Two things made every end-to-end result on this branch unreliable. make test-e2e-cluster did not rebuild ./local-ai. It only checked that a file by that name existed, so an edit that was never rebuilt ran the whole suite against an older build while reporting on the working tree, and a missing binary skipped every spec and exited 0 with "Test Suite Passed". The target now depends on a new e2e-binary target, which is the plain go build CI already used rather than make build, since that one also builds the React UI this suite never touches. The harness carries the other half: localAIBinary now FAILS, locally as well as under CI, when the binary is older than the newest non-test Go source in the tree, which covers a run started with LOCALAI_E2E_BINARY or by invoking ginkgo directly. Test files are excluded from that scan because they compile into the ginkgo suite and never into local-ai. The CI job drops its own build step and the env var so that one place owns the build and it happens after protogen-go. test-e2e-distributed was audited for the same hole and has none: ginkgo compiles that suite from the working tree on every run and it execs no prebuilt binary. build-mock-backend already rebuilt unconditionally. "Worker tunnel under load" bounded the worst probe inside the bulk transfer window against the worst probe under the empty-load window. A max over n samples is a biased estimator when the two n differ, and here they always do: the bulk window is by construction longer and draws several times as many chances at an unrelated scheduling outlier. Anything loading the box widens that gap, so the spec reddened on what else was running: 255ms against a 161ms budget with make lint beside it, 57ms alone. It now bounds the probe COMPLETION RATE instead, which is the statistic a serialised session actually moves and a mean over dozens of samples in both terms, so a uniform slowdown cancels in the ratio. Measured with make lint running: 0.97 direct and 0.60 relayed against a floor of 0.125. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
9d7a2457d9 |
feat(distributed): stop a backend on one route, whatever the worker is
nodes.<id>.backend.stop was the last worker-facing NATS subject, and it existed only because ONE publisher had not moved. An agent worker already mounted workerctl.PathBackendStop on the tunnel it holds, and a backend worker already took its stop there, so RemoteUnloaderAdapter branched on NodeType to pick a carrier for a verb both kinds of worker served the same way. The branch is gone, and with it nodeTypeOf and its NodeTypeBackend default, which removes one of the ten NodeType branches left to sweep. The adapter loses its messaging.MessagingClient outright rather than keeping an unused field: it now holds no publisher, so re-routing any verb back onto the bus is a change to the struct and to every caller of the constructor, and does not compile until all of them agree. messaging.SubjectNodeBackendStop and subjectNodePrefix are deleted, the agent worker's subscription with them. pkg/natsauth drops the per-node backend.stop grant from the agent SUB list. That is a narrowing of eleven entries to ten, never to nothing: NATS reads an EMPTY allow list as unrestricted, so the coverage spec asserts both that the retired subject is no longer covered and that the queue subjects an agent worker lives on still are. The e2e half proves it against a real enforcing server: one spec subscribes successfully on an agent-minted JWT, the next is refused the retired subject on a JWT minted the same way. Both halves of the old split were pinned, so both pins are re-aimed rather than deleted, and the two node types are asserted separately rather than as one parameterised case, because only two cases can show that the two used to differ. Three assertions that the adapter published nothing are deleted instead: with no publisher to hold, no change could ever redden them. The CLI's handler set moves into agentWorkerControlHandlers so a spec can stand it up and post to it. That wiring was a bare literal no spec pinned, and deleting the subscription made it the ONLY carrier for backend.stop: a dropped field would have been a 404 the frontend reads as a worker too old to serve the verb, and nothing in the repo would have noticed. Mutations: the agent branch restored off the control route reddens two specs; the backend branch restored, separately, reddens five; PathBackendDelete in place of PathBackendStop reddens nine across both node types; dropping the CLI wiring line reddens the new wiring table; re-adding the allow-list entry reddens the unit spec and the JWT e2e spec; and restoring the publisher for real does not compile. Four comments this change falsified are fixed, in core/cli, pkg/model and the distributed-mode docs, which now say both kinds of worker serve POST /v1/control/backend/stop and what each does with it. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
1a8384a8e1 |
test(distributed): name the replica endpoint the router now requires
Two specs in the distributed e2e suite have been red since |
||
|
Ettore Di Giacinto
|
5effa47527 |
feat(distributed): make MCP execution and discovery a selection
mcp.tools.execute and mcp.discovery were the only NATS subjects that combined a queue group with a reply, and no carrier in this design provides both. They never needed one: a queue group is a way of choosing a subscriber, and choosing is a query. The frontend now lists the approved, non-draining agent nodes, asks the node_connections table in one joined statement which of those tunnels a live replica holds, prefers one this replica holds so the call skips the relay hop, and issues an ordinary control RPC on the path task 4 already mounted. A peer-held tunnel is reached through the relay. That is a choice a broker's hidden balancing could not make. The selection reads presence and nothing else. It is filtered only on node type and on the two statuses an operator controls, never on a health verdict written on another clock, because refusing a worker that is connected and answering is the same defect as picking one that is gone. An empty fleet answers ErrNoAgentWorker, which is deliberately neither ErrWorkerUnroutable nor anything cluster.IsWorkerAnswer accepts: nothing was asked of any worker, so no reap guard may act on it. A reply carrying an Error is the worker's own answer and is returned unchanged; it is never offered to a second worker, which would turn "this MCP server rejected your arguments" into "the fleet is broken" and could run a tool twice. A call that never reached a worker is retried against a different pick, at most three times, and whatever error is finally returned is returned unwrapped so its identity survives the loop. MCP prompts and resources now answer 501 in distributed mode instead of an empty 200. They are served only from sessions the frontend holds, and in distributed mode it holds none. That gap predates the removal of the bus and is not closed by it; this only stops it being silent. Agent workers keep every other subject, including nodes.<id>.backend.stop. Their minted JWT loses the two MCP subjects and keeps a non-empty allow list, because NATS reads an empty one as no restriction at all. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
64059cd7d7 |
feat(distributed): give agent workers a tunnel of their own
Phase 2 gated agent nodes out of tunnel credentials at the mint site. That was right while nothing dialled into an agent worker: a credential would have replaced nothing, and the gate was structural rather than a second check that could drift. It is wrong now that the frontend needs to reach an agent worker by RPC. attachTunnelToken mints for backend and agent nodes and CLEARS for anything else, through one tunnelEligible predicate rather than two conditions that can be widened separately. ConnectHandler still never reads NodeType, so an empty hash is still what refuses an ineligible node. An agent worker now starts a loopback control server behind the same bearer check a backend worker uses, and holds one tunnel whose only stream tag is http: it runs no backend processes, so the grpc tag has nothing to route to and is not offered. Its MCP tool, MCP discovery and backend.stop verbs are served from ONE implementation reached by both the bus and the tunnel, so a frontend cannot get different bytes depending on which carrier delivered. The tunnel is an ADDITION. --nats-url is still required, and agent jobs, MCP execution, MCP CI jobs and nodes.<id>.backend.stop all still travel on the bus. Absence semantics are unchanged. An agent node now has a real node_connections row whose departure ages past the grace, so the node type check in HealthMonitor.tunnelDeparted stopped being an optimisation and became the rule; its comment says so, and the spec that pins it is shown red under a mutation that deletes the check. The scheduler needed no change: every placement query already filters node_type = backend, so an agent node never reaches nodeMayTakeWork. Shared rules moved to one site each. The request bounds, the POST-only check and the unknown-path 404 live in workerctl and are called by both worker packages; the bearer check that guards every extra route is one function in core/services/nodes used by both server constructors. workerctl.AllPaths splits into BackendPaths and AgentPaths, with AllPaths as their deduped union, because a backend worker does not mount the agent verbs and asserting otherwise would fail a correct worker. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
49128cf486 |
feat(distributed): keep the listener draining and let it come back
PostgreSQL holds undelivered notifications in a queue it shares with every session on the server, and it kills a listener that stops draining. Two failure modes follow, and both are silent: a carrier that blocked on a slow resolver would lose its connection and with it every later broadcast, and a carrier that reconnected without re-registering would be connected and deaf. The receive and dispatch halves were already separate. What was missing is everything around them. The listener path moves into listener.go and gains a carrier-level Dropped() so a replica that is behind can be seen; the queue depth and the spill retention become Config fields with exported defaults; the LISTEN session gets an application_name so an operator can count listeners in pg_stat_activity and a spec can drop exactly one of them; and OnReconnect fires after the re-LISTEN, on a goroutine of its own, because a callback re-hydrates from a database and must never run on the path whose only job is to drain. That callback is reached through an optional interface assertion, so deleting its invocation compiles and every adopter silently stops converging. The spec is the only guard, and it is named in a comment at the site. The slow consumer is proved through the transport rather than a seam: an ACCESS EXCLUSIVE lock on bus_messages stalls the resolver's spill SELECT for exactly as long as the spec holds it, and the listener is shown still draining and dropping while it does. The dropped connection is a pg_terminate_backend matched on the carrier's own application name. Neither Dropped nor IsConnected is on messaging.Broadcaster, and a spec asserts that over the interface type. Both are facts about a frontend; the conditions a scheduler acts on are facts about a worker, and no consumer holding the interface can read one as the other. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
8f71c08d94 |
fix(distributed): order LISTEN and UNLISTEN on one lock
Unsubscribe decided a channel had lost its last subscriber under one lock and issued the UNLISTEN after releasing it. A Subscribe on the same root could decide to LISTEN in that window, and the two reached the connection in the wrong order: the root ended up not listened with a live subscription on it. It does not heal, because the next Subscribe sees the registration already there and never re-LISTENs, so the whole root stays deaf on that replica until the connection drops. The decision and the statement it implies now happen under one lock, held across both, at both call sites. A second lock and not the registration lock: issuing waits on the listener goroutine, delivery takes the registration lock, and holding that across the wait deadlocks the carrier. The race is spec'd through a barrier seam rather than by racing goroutines. The natural window is microseconds wide, and a spec that waits for it to open passes by luck; the seam scripts the interleaving, so the spec decides in both directions. Resolving a spilled message moved off the listener. PostgreSQL keeps undelivered notifications in a shared, fixed-size queue, so a listener that stops draining it can block COMMIT for every publisher on the server, not only this one. The listener now only drains; one resolver goroutine reads the row back and dispatches, which also keeps a spilled message and an inline one on the same subject in the order they were published. Three wiring lines that could be deleted with the suite staying green: the sweeper's start is now pinned by a Config interval, and the two lines that carry the bus into the deployment now refuse to boot when either is missing. A subscription can also report what it dropped, so the party that missed a message is the party that can see it. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
cf619fc91b |
feat(distributed): carry fan-out on PostgreSQL LISTEN/NOTIFY
Distributed mode needs an operator to run a NATS cluster. This adds the carrier that replaces its fan-out half, so a deployment eventually needs PostgreSQL and its own HTTP listener and nothing else. pgbus holds one PostgreSQL session per replica, pinned for the life of the process because LISTEN registrations belong to one backend session and a pooled handle would lose them on the next checkout. Publishes go out on the pool with pg_notify. Subjects map onto a channel by their first token, from a closed set of roots. A subject outside the set is refused at publish AND at subscribe rather than mapped to a channel of its own: a channel name is capped at 63 bytes, and one LISTEN per job id would be unbounded. Refused rather than dropped, because a subject that goes nowhere and reports nothing is the class of defect this work exists to remove. PostgreSQL refuses a notify payload of 8000 bytes or more, and several subjects on this bus exceed that in normal operation: a job result carries a whole LLM output, a gallery progress event carries one entry per node. Those are written to a row and the notification carries the id. What is measured against the cap is the ENCODED notification, not the caller's payload, because the subject and the envelope travel too. The filter grammar is not respelled here. Subscribe asks messaging.ValidFilter and delivery asks messaging.SubjectMatches, which makes this the first production caller of a matcher that had only test doubles. New refuses a DSN that names a different database from the pool: that pairing publishes successfully, delivers nothing, on every replica, and reports no error anywhere. Nothing publishes on it and nothing subscribes yet. The construction is wired anyway, because the DSN has exactly one legitimate source and a setting that decides whether any broadcast is delivered should not be invented by whichever call site is migrated first. Delivery is at-most-once, like NATS core. Nothing downstream may read a message it did not receive as evidence about a node: a carrier that cannot deliver is not a worker that is gone. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
cbdd495850 |
refactor(distributed): one subject matcher, shared by the carrier and the doubles
Three copies of "does this filter match this subject" lived in the tree: one in testutil.FakeBus, a byte-identical second inside galleryop's own private fakeBus, and a third in pkg/natsauth with different semantics. The first two are doubles the specs publish through, and the carrier the pgbus work is about to add needs the same rule in production. Two spellings drift, and the drift reads as a peer that receives an event on one replica and not on another. messaging.SubjectMatches is now the only definition either double uses. The natsauth copy stays: it matches a NATS server allow list, so it has to implement the '>' tail wildcard this one deliberately refuses, and Task 16 deletes that package anyway. '>' is refused rather than implemented because no surviving subscription uses it, and a caller who writes one must get no messages rather than silently getting every message on the prefix. The refusal is checked BEFORE the filter == subject fast path: a verbatim port checks equality first, and then the filter "a.>" matches the literal subject "a.>", which is the contract leaking. One table row pins that ordering and it is the only row that does. messaging.ValidFilter refuses an empty filter, a '>' filter and an empty token so a subscriber is told at subscribe time instead of staying silently empty for the life of the process. FakeBus.Subscribe calls it, which is what keeps the double honest about what the carrier will do, and three new testutil specs pin that wiring: the previous state of the tree had no spec at all that failed when the double's wildcard routing was replaced by exact matching in any package the plan named. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
6717856212 |
test(distributed): prove phase 3 end to end, under real processes
Five cluster specs that run the binaries an operator runs, plus the repair
of eighteen specs phase 2 left red.
The eighteen were router_tracking and full_flow, failing since
|
||
|
Ettore Di Giacinto
|
dd9aff58ff |
feat(distributed): take the backend worker off NATS entirely
A local-ai worker no longer opens a bus connection. connectNATS and its
spec are gone; Run registers once, starts its tunnel, arms /readyz on that
tunnel, and heartbeats. The worker's bus credential flags (--nats-jwt,
--nats-user-seed, --nats-require-auth, the three TLS flags) and
Config.NatsAuthRequired go with it. --nats-url stays, accepted and
ignored, so an existing worker command line still parses.
/readyz was the thing most likely to wedge a tunnel-only worker: it
required a live NATS link, so a worker with no bus would have reported
itself unready forever. nodes.NATSReadiness becomes nodes.TunnelReadiness
over a local interface{ Connected() bool }, and worker.Tunnel gains
Connected(), backed by a mutex-guarded session field the loop publishes
and clears. A closed-but-not-yet-cleared session reads as disconnected:
the loop waits for every in-flight stream before it clears the field, and
the probe must answer not-ready through that wait.
The heartbeat gate is DELETED rather than re-pointed at the tunnel. The
heartbeat is the worker's own answer that its process is alive; whether
the frontend can reach it is a separate fact the frontend already holds
and ages against LOCALAI_WORKER_RECONNECT_GRACE. Withholding the
heartbeat would report an unreachable worker as an absent one on the one
path with no grace, where the health monitor marks it offline and its
pending backend ops are deleted behind it. heartbeatLoop is given no view
of the tunnel, so a gate cannot be added back without changing its
signature.
Removing the NATS credential manager from this path also removes a defect
it carried: its refresh loop re-registered on a timer to renew a JWT, and
Register CLEARS a node's NodeModel rows. Any backend worker running on
frontend-minted credentials had its replica rows deleted roughly every
18 hours.
Of core/cli/workerregistry, everything survives. The manager is still
used in full by core/cli/agent_worker.go, which still needs NATS: Acquire,
Provider, RefreshLoop, HasCredentials and TunnelToken are all untouched.
The backend worker simply calls RegisterFullWithRetry directly now.
WorkerPermissions is documented as serving agent nodes, and its non-agent
branch narrowed to _INBOX.> on both sides. It is NOT deleted: NATS reads
an empty allow list as no restriction, so returning nil would upgrade
every JWT the frontend still mints for a backend node from its own inbox
to the whole account.
Agent workers keep the bus everywhere: their CLI flags, their
subscriptions, the agent branch of WorkerPermissions, and the compose
service with its LOCALAI_NATS_URL and depends_on: nats.
Also corrected two flags the Nodes page advertised that do not exist
(--distributed-nats, --distributed-db), and a log line plus several
comments that still named a bus the code no longer touches.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Ettore Di Giacinto
|
749cc7ad81 |
fix(distributed): let absence be decided by something, at all three call sites
Removing "Presence: clusterRegistry" from the options literal in initDistributed left all seven suites and tests/e2e/distributed green. The predicate was right and its input was silently nil, which returns the deployment to absence being decided by nothing, with no log line and no failing request. That is the fourth finding of this exact shape in this phase. The two assignments move out of a twenty-field literal into distributedSchedulerOptions, a named function a unit spec can reach. Deleting either is now red. The health monitor takes its presence reader and grace as a required positional pair instead, so deleting those does not compile at all. requireAbsenceWiring then refuses to start a distributed frontend whose scheduler or health monitor has no source of absence, because refusing to boot is the only symptom either failure has. With a fresh heartbeat and a permanently gone tunnel there was no reaper at all. A heartbeat says the worker's supervisor is alive; it says nothing about whether anything here can reach that worker's backends, because those are reached over the tunnel. A proxy that stops upgrading WebSockets, a rotated registration credential or a reconnect loop longer than the grace left a node listed healthy forever while every request for a model already loaded on it failed "no route to that worker", and every reaper keyed on the heartbeat. The health monitor now reads presence from the same place and against the same window as the scheduler and demotes such a node. That also ends the 15s re-promotion: the demotion arm returns before the recovery arm, so the scheduler's demotion is no longer undone on the next tick, and recovery needs the tunnel back rather than just the heartbeat. The demotion is status-only. MarkOffline would DELETE the node's rows, and deleting rows on a presence read would give any future defect in that read the widest blast radius in the system for nothing the demotion does not already deliver. LRU eviction is the third path that commits work to a node, and it read only the stored status. A node full enough to be an eviction target is exactly the node the VRAM and idle selectors never offer, so pickReachableNode structurally cannot cover it. It now runs its chosen node through the same nodeMayTakeWork predicate, demotes it and evicts again rather than handing back an install that cannot land. Presence is read after the transaction and not inside it: reading it inside would hold a FOR UPDATE lock across a query needing a second pooled connection, which is how concurrent evictions deadlock a pool. Also: a router built with a presence reader and no grace now has its documented default pinned by a spec rather than only claimed by a comment; ageDeparture asserts RowsAffected, since an UPDATE matching nothing succeeds and the inside-the-grace spec returned the same verdict either way; the scheduler comment that still described the bus is corrected; the docs stop conflating heartbeat recovery with tunnel recovery and name the third reader; and an overlong rewrapped line in membership.go is folded. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
93af91419b |
feat(distributed): read worker absence from the database, not from a bus timeout
The scheduler decided whether a worker had gone away from nats.ErrNoResponders: one frontend's observation that nobody answered IT within a request budget. Two replicas asking in the same moment could disagree and demote each other's workers, and a worker re-homing its tunnel between replicas looked identical to one that had died. SmartRouter now reads cluster.Presence instead. Only PresenceGone -- no live replica holds the tunnel AND the departure has outlived the reconnect grace -- excludes a node from placement, and it is a fact every replica reads identically from the database. PresenceReconnecting, PresenceUnknown and a failed presence query are all non-verdicts and place work as normal: excluding on a database hiccup would cost the fleet its capacity for a reason that has nothing to do with any worker. nodeAnswersOnBus is deleted. It excluded on a sentinel no control RPC can produce, so it decided nothing while PingNode cost a relayed round trip per scheduling decision to feed it. PingNode goes with it, from the adapter and from NodeCommandSender. isRequestTimeout drops nats.ErrTimeout: every verb this adapter sends now travels over the worker's tunnel. The predicate is named nodeMayTakeWork rather than nodeHasRoute. "Route" is ErrWorkerUnroutable in this package, the condition nobody may act on; PresenceGone is the one a scheduler may. Spelling them the same way is the collapse this work exists to prevent. Also folds in ReapStale's return rename: it counts connection rows CLEARED, never rows deleted, and reading it as a delete count would make a worker that is re-dialling right now look forgotten. The spec pinning that a message merely quoting "nats: timeout" is not a timeout was scripting a SUCCESSFUL reply carrying the phrase, which comes back with a nil error and never reaches the classifier. Restoring the string match left it green. It now scripts a 5xx whose body carries the phrase, and asserts that the phrase reaches the classifier as a precondition. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
4c3e0deb19 |
test(distributed): pin the unreadable-request rule at all three file verbs
The rule "a body this worker could not parse is a non-2xx, never the worker's answer" is written at three exits in control_files.go and only ensure was pinned. Turning stage's or listdir's decode exit into a 200-with-error left worker and nodes entirely green, and what that converts is a frontend's malformed request into the worker's own verdict about a file, which passes cluster.IsWorkerAnswer and reaches a reap guard. The production code was already right; nothing held it there. The e2e NATS JWT spec was asserting the opposite of the code and passing. It published nodes.<id>.files.in and called it an allowed subject after that grant was deleted, and it could not tell: a permission violation does not close the connection, so FlushTimeout and IsConnected both stay happy. It now reads LastError, the way its sibling always has, and asserts the denial plus the one publish right a backend worker has left. Also pinned, each mutation-verified alone: the CreateTemp branch (an existing staging-tmp at 0500 reaches it without a seam), the walk's context check (a caller that gave up must fail the listing, never be answered with a short one), and the cache and data directory layout. The data directory was derived twice, once in worker.go and once for the listdir verb; worker.go now reads the same helper, so a move cannot leave a verb listing files the file server does not serve. The per-verb RPC ceiling moves from an argument at five call sites into fileRPCBudget, so no site can name the wrong one, and the two values are asserted. The body-cap table now holds both directions locally and with two different claims: a body exactly at the cap proves the bound is a ceiling and not an off-by-one, and an absolute megabyte proves the cap stays above real gallery traffic. Only the second notices a cap shrunk to 64 KiB. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
7fc617c8ed |
feat(distributed): serve file staging over the worker tunnel
The four nodes.<id>.files.* subjects were the last commands a serve-backend worker took off the bus. They are now HTTP routes under workerctl.Prefix, on the same loopback server and behind the same bearer check as the ten lifecycle verbs, so the frontend reaches them through the worker's tunnel. files.listdir is the verb this matters most for. Its reply had to fit a payload the bus would carry, which put a wide model directory close to the limit; a response body has no such ceiling, so nothing truncates the listing at either end. A short listing reads to the frontend as files the worker does not have. S3NATSFileStager becomes S3FileStager and calls ControlClient, which means every failure now lands in the bucket phase 3 exists to keep straight: a route this frontend could not use is unroutable and nothing may act on it, while the worker's own answer, including "that file is not there", is evidence a caller may act on. Each RPC's deadline is DERIVED FROM the caller's context rather than started fresh, at every one of the five call sites, so a caller that gave up stops the RPC too. A worker started without an object store mounts no file verb at all and answers 404, which is the same answer a build too old to know them gives. The subjects and the backend worker's files.> publish grant go with them; a backend worker now publishes nowhere but its own inbox. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
5880f4e6dd |
test(distributed): pin the no-demotion rule at every call site it is stated
Review fix round 1. Two blocking findings and seven non-blocking; both blocking ones are M12's shape again, and this time on the invariant itself. No production behaviour changes here: everything below was already correct and merely unpinned, so re-inserting the defect left all 679 specs green. The only non-comment edits are one struct-field comment and one log message. "A failed control RPC no longer demotes a node" is stated three times in this package and was pinned once, at ListBackends. Putting MarkUnhealthy back at either op-drain site passed. What that buys in production is the fleet-wide eviction this phase exists to prevent: MarkUnhealthy removes a node from ListDuePendingBackendOps AND from scheduling, so a frontend replica that has just lost its tunnels demotes every node it holds an op for, for a reason that is about the frontend. The reconciler's is the worse of the two, being a background loop nobody is watching. Both now have a spec, each with the recorded op failure as its negative control so "still healthy" cannot pass by nothing having happened. The sweep the review asked for found four more rules stated at more call sites than they were pinned at, and two the review had not: The still-installing surfacing at the manager layer has two call sites and was pinned at InstallBackend. Dropping it from UpgradeBackend reported a spent budget as GREEN SUCCESS: the admin sees the upgrade finished while the worker is still re-pulling gigabytes. The agent-node skip has two call sites and was pinned at ListBackends. Without it the fan-out enqueues a row for every agent node, and an agent worker serves no control plane, so that row can never drain: it retries until the dead-letter cap. The still-installing conversion has three call sites and was pinned at two; the legacy force-install fallback was the gap. Its budget was unpinned too, so the new spec asserts both, on the upgrade budget rather than the install one, since the fallback re-fires an install as part of an upgrade. The carrier split has two call sites and was pinned at one. Hardcoding NodeTypeBackend in UnloadRemoteModelContext passed, and an agent node holding a node_models row would then have its stop sent over a tunnel it does not hold, fail, and leave the row behind. The new spec unloads a model held by one node of each kind and asserts each stop went to that node's own carrier and to no other. router_nats_liveness_test.go asserted demote-on-absence, which production can no longer produce, and its header described the pre-cutover world. The exclusion is unreachable by construction rather than by argument: cluster, the package supplying every control-path dial error, does not link nats.go at all. The file now says that, and gains the assertion that IS load-bearing, a table naming each sentinel a control RPC can answer with and requiring that none of them excludes. Widening the exclusion to ErrWorkerUnroutable reddens four of its entries plus the real-adapter scheduling spec. unroutable keeps no budget-first guard and the reason is now written at it: unlike controlFailure it reads one already-recorded error rather than racing a live deadline, and an expiry is not in streamRefusals, so it falls to the umbrella without one. The two implement the same split at two layers and each now names the other. Fourteen comments still described the bus. Among them the reconciler saying a drain would "churn NATS every tick", a spec comment naming a subject builder this branch deleted, and the agent-skip comment explaining the skip by a subscription that no longer exists. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
44f12b2adb |
feat(distributed): call the worker's control routes instead of the bus
The ten backend and model lifecycle verbs stop being NATS requests and become HTTP calls on the worker's own control routes, reached through that worker's tunnel on the `http` stream tag that already carries file staging. Nine subject builders and the per-op install-progress subject are deleted with their entries in the worker's NATS permissions; the request and reply DTOs are untouched, so a body on the wire is byte for byte what the subject carried. This closes the merge gate Task 3 left open, which was worse than lost commands. Once the worker stopped subscribing, PingNode was still asking nodes.<id>.backend.list and nodes.<id>.models.running, so EVERY healthy worker answered no-responders, nodeAnswersOnBus read it as absence and pickReachableNode demoted it on the scheduling path. PingNode is a control RPC now, and no control RPC can produce ErrNoResponders, which is the only error that exclusion acts on. Two specs drive pickReachableNode against a real adapter and a worker answering over its control plane, which is the only arrangement that can see the difference: the router's own double never touches a transport and stayed green for the whole window the defect was live. How a control RPC FAILS is the whole of this change, so it is decided in ONE function reading ONE table. A worker's answer passes through unwrapped, so cluster.IsWorkerAnswer still sees it and a reap guard may act on it; everything else is wrapped in ErrWorkerUnroutable so nothing can. There is no third branch, because a third branch is how the eight collapses on this branch happened: each was a site that decided for itself which errors were evidence. A 404 under the prefix is its own sentinel, because it is the worker stating a deployment fact about ITSELF rather than a verdict about a backend, and only the legacy upgrade fallback may act on it. The caller's budget is checked FIRST. A timeout is not a verdict: a refusal arriving in the instant a deadline expires would otherwise be reported as the worker's non-transient answer, which reaps a row, and nothing orders the two timers. A 5xx and an undecodable body are transport failures, not answers. An empty ModelsRunningReply means "this worker is running nothing", which the reconciler acts on, so it must never be manufactured from a body that would not parse. A stream that ends before its reply line is the same rule one layer up: a tunnel dying mid-install is not the worker saying the install failed. backend.stop is split by node type rather than moved. Agent workers hold no tunnel, so they have no control plane to serve, and they still subscribe to nodes.<id>.backend.stop to drop cached MCP sessions; that subject and its agent permission both survive. It is the honest intermediate state until agent workers hold tunnels too. A failed control RPC no longer demotes a node anywhere. ErrNoResponders meant "not on the bus"; a control failure means "this frontend could not route to it", which is equally what a healthy worker re-homing its tunnel between replicas produces. Absence is a fact read from the database, and the scheduler starts reading it in a later task. The rolling-update fallback re-fires a DESTRUCTIVE force-reinstall, so it runs only on the worker's own 404. Its negative direction was pinned at the admin call site and unpinned at the reconciler's, where widening the condition to any error left all 676 specs green: a background drain nobody is watching would then force-reinstall every queued backend the moment a replica lost its tunnels. Three specs cover it, arranged so the force install IS reachable in the negative case and a fallback that fired would show as a call and a drained row. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
f9d0d4c5c6 |
fix(worker): pin the rune cut, answer unload honestly, drop the dead publisher
Review fix round 1. Seven non-blocking findings; the blocking one is a merge gate for Task 4 rather than anything in this diff, and the report's concern about it is corrected: until Task 4 lands, PingNode probes two subjects no serve-backend worker subscribes to any more, so every healthy worker reads as absent and is marked unhealthy on the scheduling path. The rune-boundary cut in truncate was true behaviour with nothing holding it: a byte-wise mutation survived all 201 specs. isRuneStart is replaced by utf8.RuneStart, the same predicate the cluster package uses for this rule, and two specs pin it, one with a rune straddling the bound and one with a rune ending exactly on it so the fix cannot be "always walk back". unloadModel answered Success:true whatever Free did. That is the worker saying "done" about work it did not do, and the frontend's only caller is EvictLRU, so a false yes told the scheduler VRAM had been released and let it place the next model on a node still holding the old one. It now reports the failure, following stopModelExact, which is the honest pattern already in this package. Still a 200: the worker answered, only its verdict is negative. An address with nothing loaded still answers success, which is a true answer rather than a claim about work done. NewDebouncedInstallProgressPublisher had no production caller after the last commit, only its own spec. Deleted rather than wired: wiring it would publish every event on two carriers, which is what the carrier decision exists to avoid. Its specs now run against the sink, plus one that pins the identity stamped on each event, since the subject used to carry the op and node id and now nothing but the body does. The install progress wiring was exercised by no spec, because with no gallery nothing ever invokes the download callback. The guard moves into startProgress, shared by install and upgrade, which also emits one resolving event before any gallery work. That is worth having on its own: a cold install spends minutes on a manifest and a progress stream with nothing on it is indistinguishable from a broken one. It also makes the wiring observable end to end, and four specs now drive the real installBackend and upgradeBackend over HTTP with no override. model/stop and backend/stop keep taking Background rather than the caller's context, and the sites now say why. model/stop is the acknowledged stop path: it reserves the process, frees it, kills it, waits for exit and releases the port, and abandoning that because the caller hung up would leave a process marked stopping, a port not returned to the allocator and a row nothing reconciles. In stopBackendExact the Free is a courtesy before a kill that happens anyway. model/unload differs because Free IS the operation there. A route set with no prefix or no registrar is now a startup error rather than a silent no-op: a server that comes up healthy while every route the caller registered answers 404 is, through a tunnel, indistinguishable from a version skew. And the AllPaths spec no longer claims to catch a constant that was never added to the set, which it cannot; it asserts the whole set instead, which catches a verb dropped from it. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
49b3d22974 |
feat(worker): serve the control plane over the tunnel, not over NATS
Ten NATS subscriptions on the worker become ten HTTP routes under
/v1/control/, served on the loopback HTTP server the worker already runs
and reached only through the tunnel's existing `http` stream tag.
The carrier is the tag that already exists rather than a new one. A new
tag would have had to invent correlation, per-request deadlines,
unbounded payloads and a progress stream, and each of those is a place
this branch has already put a defect. It would also have added a fifth
entry to the worker's stream-refusal vocabulary, which decides what a
frontend reaps on and took eight fixes to settle. Riding `http` means a
control RPC to a worker another replica holds takes the same relay the
inference path takes, which is the path that has been measured.
The request and reply DTOs are untouched, so a body on a control route
is byte-for-byte what the corresponding subject carried. No subject was
deleted: agent workers still subscribe to nodes.<id>.backend.stop.
Install and upgrade stream. They answer application/x-ndjson: zero or
more {"progress":...} lines carrying the same event the per-op NATS
subject carried, then exactly one {"reply":...} line, always last. That
deletes the 8000-byte notification cap structurally instead of
reproducing it on a new carrier: a progress line is written into the
response the caller is already reading, so there is nothing to size and
no subscribe-before-request window. The debouncer is shared with the
NATS publisher rather than forked, so the ~4/s tick bound is one fact.
A verb's own failure is a 200 with Error set, never a 5xx. The frontend
maps a transport failure onto "no route to that worker", which nothing
may act on, and the worker's answer onto evidence a reap guard may act
on; answering 500 for a failed install would put the worker's verdict
in the bucket reserved for a broken link. Only a request that could not
be read or routed is non-2xx.
Control RPCs carry the caller's budget. r.Context() replaces four
context.Background() calls at the gallery-install sites, and the one
pre-existing fixed timeout on model.unload is now derived from the
caller's context so a shorter budget is honoured. No timeout is invented.
The inner `go func()` in the install and upgrade handlers is deleted
rather than nested: it existed because one subscription served every
install, and over HTTP each request already has its own goroutine.
Per-backend serialization stays lockBackend, which is what actually
prevented two requests racing the gallery directory.
Bounds against a boundary the worker now serves: every body is capped at
8 MiB before any decode; the 404 echoes at most 128 bytes of the request
path, cut on a rune boundary so a half rune cannot travel downstream as
a replacement character; non-POST is refused before the body is read so
a probe cannot fire a command; the streaming responses set nosniff.
The routes mount through nodes.AuthenticatedRoutes, which hands the
registrar a private mux and puts the whole prefix behind the same
constant-time bearer check as the file routes. The worker's HTTP server
now takes the supervisor as a required parameter, so there is no way to
start it without the control plane mounted.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Ettore Di Giacinto
|
6e50f060a3 |
fix(cluster): pin the derived retention to the sweep that applies it
The retention a worker's departure is kept for is now derived from the reconnect grace, so a purge can never outrun the window Presence measures against. Nothing pinned that. The sweep could be reverted to pass the constant, or the setter emptied out, and the suite stayed green either way: the specs covered the arithmetic helper, and the fix is the wiring. The loop now has a spec of its own. It departs two workers either side of the difference between the floor and the derived retention, and the row that must go is what witnesses the sweep running at all, so the row that must stay cannot survive by nothing happening. The default grace goes from 60s to 90s. Two of the worker's ceiling backoffs is 60s, but the failed dial between them costs its handshake timeout too, which puts the worst case at 70s, and the backoff resets only after a session long enough that a replica accepting a dial and then dying denies it. So the ceiling is reachable exactly during the rolling restart this window exists for, and 60s sat on the edge of it. Too short reports a live worker as gone and costs a model reload; too long reaps a dead one later. The cheaper mistake is the long one. A held row whose owner is dead and whose stamp is stale is the state a rolling upgrade actually produces, and it was the one state no spec built. It has an answer now, and the two ways to get this wrong land either side of it: reading the stamp first says gone, reading held-ness without the liveness join says connected. Two comments claimed more than the code did. There IS a grace at which a live worker is reported as gone, which is the point of it being a duration; and the switch that reads held-ness first is only a partial second gate, since with the SQL gate gone and a dead owner it answers gone rather than reconnecting. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
6b4ce58207 |
feat(cluster): answer presence with four values on the database clock
A worker whose tunnel is gone is not, by that fact, a worker that has left. Absence is what makes the scheduler stop placing work, reap the worker's rows and evict its models, and one of those paths runs during inference, so the deployment needs to tell a worker re-homing between frontend replicas from one that is really gone before anything acts. Registry.Presence answers that in one joined statement, with four values and not a boolean: unknown when there is no row at all (this package cannot tell a worker that has never dialled from one whose departure aged out, and must not guess), connected while a live replica holds the tunnel, reconnecting while the departure is inside the grace, and gone once it is older. Only the last is a verdict a caller may act on. Held-ness is asked FIRST and the departure only refines it, in the SQL and again in the switch that reads it. Every writer here clears disconnected_at in the statement that writes the owner, but that is a property of these writers rather than of the table: a replica running a binary from before the column existed re-claims without clearing the stamp, so during a rolling upgrade a held row carries an old departure, and a read that consults the stamp first reports a connected worker as gone for the whole upgrade. Both windows are computed by the database, for the reason every other window in this package is: they are compared across replicas, and replicas disagreeing about whether a worker is gone is the flapping this branch exists to remove. No behavioural spec can see the difference, since the test container shares the host clock, so the statement shape is pinned instead. The grace is an operator's knob, defaulting to twice the worker tunnel's maximum reconnect backoff. That made the fixed departure retention wrong: an operator raising the grace past it gets a purge that deletes departures before the grace elapses, so a worker that is gone reads as unknown forever and nothing ever reaps it. The retention is now derived from the grace, with the old constant as its floor. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
d99f7ff7ab |
fix(cluster): pin the departure retention to the loop that applies it
The membership tick's call to PurgeDepartedBefore was the only production wiring this change introduced, and removing it left the suite green. A retention nothing applies is a departure that never ages out, which is the state the sweep's held-ness filter exists to make reachable at all. A spec now ages a released row past DepartedRetention on the database clock, starts a real Membership, and waits for the row to go. Release and Deregister leaned on "no owner id is ever empty" to avoid touching an already-departed row, which is the accident Owner refuses to lean on. A departed row keeps its epoch and carries an empty owner, so a release or a deregistration naming an empty id matched it and stamped a fresh departure over the old one, making a worker that left long ago look like one that has only just gone. Both now filter on connectionIsHeld. The comment on DisconnectedAt claimed a held row never carries a departure. A binary from before this column existed claims without clearing the stamp, so a rolling upgrade produces exactly that row. The comment now says what holds, and says to ask held-ness first and read the stamp second. The sweep's vocabulary follows the code: it records departures where the comments still said it deleted rows, and its log line separates the instance rows it deleted from the connection rows it left behind. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
49e77447fb |
feat(cluster): record a departure instead of erasing the connection
Releasing a worker tunnel deleted its node_connections row, so "this worker's link dropped a moment ago" and "this worker has never connected here" were one observation: no row. Nothing above could tell a worker re-homing between replicas from a worker that is gone, and any grace period built on top would have had nothing to measure from. The row now survives a departure. Release clears owner_instance_id and stamps disconnected_at on the database clock; the membership sweep and Deregister do the same for every connection a dead or departing replica held; Claim clears the stamp in the same upsert that writes the owner, so a reconnect is never observed half-applied. PurgeDepartedBefore deletes a departure once it is older than DepartedRetention, and the membership tick owns that schedule. Owner and OwnerRow report a departed row as ErrNoConnection, through the one predicate connectionIsHeld, the way instanceIsLive is the one predicate for replica liveness. This change records the departure and does not interpret it: how long ago it happened is nobody's answer yet. The sweep only clears rows that are still held. An empty owner is in no instance's id, so without that filter every heartbeat would restamp every departed row and no departure could ever age out. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
6b712e76db |
fix(cluster): keep the refusal vocabulary in one table
The worker re-classified a failure a local service had already classified. classifyServiceFailure preserved exactly one of the four refusal codes, which was faithful to its own comment for as long as there was one worth keeping; once ErrStreamNotServed existed, a service returning the code whose whole job is to say "I learned nothing" had it promoted to ErrStreamTargetUnavailable, which every reap guard acts on. ErrStreamTagUnknown was promoted too, and cost nothing only because both sides of that one reap. No in-tree service produces either, which is the same "unreachable, therefore safe" argument that let the request-frame merge survive a whole phase, and LocalService is exported. The cause was a fifth site enumerating the vocabulary by hand, so the fix is one table. streamRefusals pairs each sentinel with its wire code and with whether a frontend may act on it as evidence about a backend, and the writer, the reader, IsWorkerAnswer and the new IsStreamRefusal all read it. A fifth code is now taught to every one of them at once. The codes are also pinned against literals written out in a spec, the way this branch already pinned the NATS vocabulary. The round-trip table cannot see a rename, because a rename moves the writer and the reader together; an unrecognised code is deliberately not the worker's answer, so renaming "unavailable" would turn every crashed backend on a tunnelled worker into a row nothing can ever reap, silently and with the suite green. Three comments the previous fix falsified, corrected: - tunnelHeaderTimeout still said the window bounds only framing the frontend writes immediately after opening the stream. That is true on the direct path and false on the relay path, and it was the argument for treating an expiry as the frontend's fault. - classifyServiceFailure's deny-list is three causes, not two: on a dial error net.Error.Timeout also covers ETIMEDOUT and EAGAIN. Both are kept deliberately, because reaping a wedged or resource-starved backend is the eviction this phase exists to prevent, and ECONNREFUSED still reaps. isReadTimeout is renamed reportsTimeout, which is what it asks. - The operator table named three refusals and said a refusal is acted on. It now lists four, with when each is sent and whether the row is reaped. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
c19ed5ab32 |
fix(cluster): stop a late request frame reading as the worker's verdict
Making a worker's refusal reaping evidence created a defect one layer along, at the producer. The worker refused a ReadStreamRequest failure with ErrStreamRequestInvalid and its own comment said "Includes the deadline above expiring", which was harmless while every refusal reached the frontend as "no route" and became a reap the moment one of them did not. So a request frame that had merely not ARRIVED yet was reported as a non-transient verdict about a backend. It is reachable on the relay path, which carries most production traffic: the worker's header timer starts when the OWNING replica opens the stream, while the frame is written by the DIALLING replica only after the relay's acceptance travels back to it, so a whole peer-link round trip runs inside that window, on a link this design deliberately loads with multi-gigabyte artifacts beside token streams. For a long-deadline caller the endpoint is ConnectionEvictingClient, which stops the model across the fleet. It also falsified the "neither clears on its own" argument that licensed the reap. There is now a fourth refusal, ErrStreamNotServed, for what a worker could not serve for a reason of its OWN. It is deliberately outside IsWorkerAnswer, so it reaches a consumer under the no-route umbrella and reaps nothing, which is the same treatment an unrecognised code already gets. Four producers move onto it: a request frame that timed out (a malformed one stays a verdict, because that is a frontend bug no retry fixes), both SetReadDeadline failures, which are facts about the stream and not about a target nothing has dialled yet, and WriteStreamRefusal's default for a reason nobody classified. classifyServiceFailure keeps ErrStreamTargetUnavailable as its default on purpose: inverting it would make errno enumeration the single point of failure for the reap, and a miss there is a row nothing can ever delete. What it gains is a deny-list of two causes that are provably this worker's own clock or its own context. Also: - The read-site caller-deadline guard in the handshake was unpinned: the existing seam spends the budget before the handshake starts, so only the write could ever fail. A spec whose deadline falls between the request and the reply pins it, and each guard now reddens on its own. - The documented worker-first failure line omitted the JSON error envelope the old frontend returns, so an operator grepping it found nothing. - The peer-link disclosure names the aimable per-session receive window in all four places, and LastDialErrorOf records why a third consumer must go through IsWorkerAnswer rather than roll its own list. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
d26263f9c0 |
fix(distributed): let a worker's own refusal be evidence about its backend
A worker that refuses a stream has answered, and cluster.Dial keeps the three tunnelproto sentinels out of the ErrNoRoute umbrella precisely so a consumer can act on that. No consumer did. Since workers stopped listening, a backend process that crashed on a healthy worker is no longer a dead listener's codes.Unavailable: the worker refuses the stream with ErrStreamTargetUnavailable, gRPC flattens it into Unavailable anyway, and nodes.unroutable reported the whole thing as "this frontend has no route". Every reap path then answered ProbeUnknown and left the row, so the replica slot never freed and at the default MaxReplicasPerModel=1 the only cleanup left was LRU eviction of models that were working. isWorkerAnswer is exported as cluster.IsWorkerAnswer, so the errors the dialer keeps out of the umbrella are by construction the errors the consumers treat as the worker answering. nodes.unroutable and pkg/model's transportFailure both use it; ConnectionEvictingClient, the site reached during inference, goes through transportFailure rather than asking the transport directly. A reply code this frontend does not recognise is still not an answer, so a newer worker's vocabulary costs a retry and not a replica. The reap guards keep the allow-list rather than requiring ErrNoRoute: an unrecognised dial error must mean "no route", never "the backend is gone". Also in this final pass over the branch: - Docs: recommend upgrading FRONTENDS first, with the symptom of each order. Workers-first fails now that a 4xx registration is a verdict rather than an outage, so an old frontend's "address is required for backend workers" makes each restarted worker exit and drains the fleet a node per restart. - Docs: LOCALAI_WORKER_TUNNEL=false is a fatal startup error, not a degraded mode, in both places that described it; and a frontend rollback needs every worker restarted, because re-registration force-clears the address columns. - A replica with no advertised address now says so every five minutes and names the workers only it can reach, instead of one startup warning for a cost paid for the life of the process. - callerRanOut's rule now holds at all three siblings, so an expired caller deadline stops reading as a broken tunnel; probeHealth's withdrawn reason for using the raw client is corrected; the dead DoOrCached is deleted and its coverage kept on DoOrCachedResult; sweepLeakedInFlight enumerates the outcomes that reach it. - The peer route's self-declared id is recorded as a phase-3 deferral, in the handler, in the isolation claim it narrows, and in the operator docs. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
0dc6ebd525 |
fix(cluster): stop blaming a peer for the caller's own expired deadline
Review round 1 on the end-to-end proof. Zero blocking items, eleven non-blocking, and three of them turned out to be production defects rather than notes on the report. The one that matters is a misclassification the phase is built to prevent. A dial carries the caller's deadline down to the socket, so when the budget runs out the socket's timer fires and the error travels back up through the WebSocket handshake and the multiplexer. The context's cancellation is a separate timer whose func the scheduler has to run before ctx.Err() stops returning nil, and nothing orders the two. Under contention the socket's error is back in PeerPool.Open first, ctx.Err() reads nil, and a peer that is listening and healthy is reported as ErrPeerUnreachable to a caller that simply ran out of time. An unreachable peer is a fact a caller may act on and an expired deadline is not, and core/services/nodes routes around a replica it is told is unreachable. callerRanOut answers that question in one place: ctx.Err() when it is set, and otherwise the wall clock against the caller's own deadline. That is sound because it is the same instant the socket compared itself against, so if the socket's timer fired this comparison is past it too. The ambiguous instant resolves towards the caller, which is the direction that never blames a peer. The spec that caught it, peerlink_test.go's "blames the caller's deadline", was red in three of seven -race runs and had been since Task 5, which is often enough to read as noise and is why single-run verification never saw it. Rather than leave the proof to a coin flip, a second spec makes the window deterministic: Open is handed a context whose deadline has passed and whose cancellation has not been delivered, against an address nothing is listening on, so the dial fails for real. It reddens without the fix. The peer link's yamux windows were applied to one end only. A receive window is advertised by the side that RECEIVES, so configuring the dialler alone tunes exactly one direction, and the direction left on the 256 KiB default is the one that carries a relayed model artifact INTO the replica that owns the worker's tunnel. That is the largest thing the link ever moves and it is the direction the load measurement exercises: the review read it as flowing toward the dialler and it does not. PeerLinkConfig is now exported and used on both ends. Measured, same box, 128 MiB staged through the relay against the same transfer without one: the relayed path cost 1.6x to 2.0x the direct path's transfer window before, and 1.06x to 1.25x after. The SSRF reachability spec could be fooled into reporting an SSRF that did not happen. It bound the victim on 127.0.0.2 at an ephemeral port and required 127.0.0.1 at the same port to refuse, so any other spec in the run holding that number made the dial succeed; red one run in seven, green five of five in isolation. It now picks from below the kernel's ephemeral range, the same fix the harness got for the adjacent-port collision. The rest are the specs and the report saying what they mean. Scenario 1's advertisement assertion could not tell "the worker advertises nothing" from "the JSON key moved", which matters because removing the advertisement is the change it covers. It was green against a renamed key. The roster now keeps the raw key set beside the decoded fields and the spec requires both keys present before reading them as empty. Scenario 4's refusal-body check was a four-way disjunction admitting bare "tunnel", "not connected" and "unroutable". Those alternatives were inert and each would be satisfied by refusals that say nothing about routing, in the one assertion the whole negative control rests on. It is "no route" alone. The head-of-line gate bounded the worst probe by the whole transfer window, which admits about eightfold degradation and loosens as the box slows. It is now half the window, plus a scale-free ratio against the worst probe under the SAME cold load with nothing to transfer, which is the control that isolates the transfer from the load. Not tighter than that, and the reason is measured rather than cautious: under a concurrent -race suite the worst relayed probe reached a fifth of its window, so a quarter-window gate would have had 1.2x of margin, and a spec that fails one run in three is worse than no spec. The report entry printed p90 and p99 off samples of twenty, where both land on the same element and p99 often lands on the max, so one number appeared three times under three names. A quantile is now printed only when the sample can separate it. Two claims in the report were wrong and are withdrawn rather than softened. Scenario 2's race is closed by the trailing re-read of the owner, not by the pre-assertion the report credited: a move to the non-owner mid-request would serve directly and still return 200, and only the trailing read reddens on it. And "the median request is unchanged" holds on this box and not on the reviewer's, where the relayed median rises up to 82% and p99 up to 3.5x. What survives on both is structural: the worst probe is a small fraction of the window in which bytes are moving, so the session interleaves rather than serialising. Sharing a session with a bulk transfer costs latency; it does not cost service. The disk footprint note undercounted, and the reviewer lost a run to a full disk on this box, so it is worth having right: two bulk models seeded into two frontends and staged to the worker is about 768 MiB, not 512 MiB. Left alone deliberately: the worker's backend port allocator still hands out ports without checking they are free, and its default range still overlaps the kernel's ephemeral range. It is confirmed, it is out of scope here, and it is being tracked as a named follow-up rather than fixed under an e2e task. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
0683fb1579 |
test(distributed): prove the worker tunnel end to end, under real inference
Everything this phase built was proven by unit and integration specs. This is the first run of it against the real binaries: a frontend replica per process, a worker that binds nothing routable, real inference over the result. Four scenarios, each with the question "what would make this pass if the tunnel were doing nothing" answered rather than left open. A worker with no advertised address is reached through its tunnel. The roster is asserted to report it advertising nothing, so there is no address a frontend could have dialled instead, and node_connections is asserted to name the replica that serves the request. A request landing on the replica that does NOT own the worker is relayed to the one that does. With N replicas behind round robin that is (N-1)/N of production traffic, so it gets the FIRST request for its model: the backend install, the file staging on the http tag, and the gRPC load and predict all cross the relay. Which replica owns the tunnel is read from the ownership table through the production Owner query and mapped to a frontend index through the address the harness pins per replica; the non-owner is derived from that reading and asserted to be a non-owner immediately before the request, rather than assumed from the harness default. Sending the same request to the owner reddens it. Killing the owning replica re-homes the worker onto the survivor. The worker dials a balancer rather than a replica, because LOCALAI_REGISTER_TO is resolved once at boot and is the tunnel endpoint as well as the registration one: aimed at a single replica, a worker has nowhere to reconnect to when that replica dies, and the re-home cannot happen at all. Removing the kill reddens it. And the negative control for the whole suite, which is why the other three mean anything. Frontend and worker share a host here, so every backend port the frontend names in a stream target is one it could have dialled directly; if it did, the first three would pass with the tunnel inert. LOCALAI_WORKER_TUNNEL is no longer usable for this, because it is a fatal startup error and a worker that never started says nothing about a worker reachable some other way. The balancer answers the tunnel connect path itself instead, leaving a worker that registers, heartbeats, reports healthy and holds no tunnel. It is asserted to have dialled and been refused, asserted to be held by nobody, and then asserted unreachable with the refusal naming the missing route. Then the block is lifted, nothing else changes, and the same request succeeds: that is what attributes the refusal to the tunnel rather than to any of the ordinary reasons an e2e inference fails. The fifth spec measures the head-of-line blocking this phase deferred three times. 128 MiB crosses the session while a warm model is probed back to back, direct and relayed. Median latency is unchanged, the worst probe is about 3x the baseline median and about a seventeenth of the transfer window, and the transfer runs at 415-490 MB/s direct and 222-268 MB/s relayed. A session that head-of-line blocked would park a probe for the length of the window. Leave the yamux windows untuned; and note this is loopback, so it says the multiplexing does not serialise and says nothing about a link with a bandwidth-delay product. The load spec is measured against a control that the first version did not have. It passed with the bulk artifact cut to 4 KiB, because the window it read probes against was mostly cold-load overhead: it would have reported a clean bill on a session carrying no large message. The same cold load now runs twice, once empty and once bulk, and the difference between the windows is asserted to be real before any latency is read from it. Two defects on the base commit came out of this. cluster_peerlink_test.go has been red since the relay landed, deterministically, in isolation and in the suite. It asserted that an accepted peer stream is refused at once, on the premise that phase 1 installs no relay. The relay correctly waits fifteen seconds for a frame naming the worker, and the spec's budget was five. It now writes a relay request for a node no replica holds and asserts the refusal is ErrNotOwner and specifically not ErrNoConnection, which is a stronger spec than the one it replaces and the only thing in the e2e suite that exercises the relay's refusal path. The harness handed a worker's own HTTP port to a backend process. It took two ports from freeport and used one as the gRPC base and the other for the file transfer server; freeport returns adjacent ports often, and the backend allocator hands out base, base+1, base+2, so the second backend started on a worker was regularly given the HTTP server's port and died with EADDRINUSE. No spec had started two backends on one worker before, so it had never fired; the load spec starts five and it failed about one run in three. Each worker now reserves a contiguous bind-probed block laid out the way production lays it out, below the kernel's ephemeral range, with LOCALAI_GRPC_MAX_PORT bounding the allocator to it. The underlying production defect is not fixed here and is recorded in the report: allocatePort never checks that a port is free, and its default range overlaps the ephemeral range on every Linux box. Constraint 6, whether distributed mode should now refuse to start without an advertised address, is DEFERRED, and the comment and the docs that described the cost were understating it. A replica with no advertised address writes no instances row, and Owner joins a connection against a live instance, so a worker whose tunnel lands there is unroutable from every OTHER replica while being registered and healthy. Refusing to start would still be wrong, because the deployments it would break are single-host ones with no peers to be unreachable by, and telling those apart at startup is a design with its own specs. Both places now say what actually happens. Suite wall clock 592s for 15 specs, up from 502s for 10 of which 2 were red. The CI budget of 20 minutes does not move. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
3338d7bc56 |
fix(distributed): refuse a worker that cannot tunnel, and say why it was refused
Review round 1 on the change that stopped workers listening. One blocking item
and seven notes.
LOCALAI_WORKER_TUNNEL=false was the blocking one, and the ruling was to make it
fatal rather than to correct the comment that still promised it fell back to the
advertised address. There is no fallback left: a worker on this branch
advertises nothing and binds only loopback, so turning the tunnel off leaves it
reachable by nothing while it registers, heartbeats and reports healthy, and the
scheduler keeps placing models on it. That is the worst available failure shape,
so a new Config.validateStartup refuses it before prefetch, registration and
NATS, while the worker is still invisible to the cluster. It absorbs the
pre-existing empty-registration-token check, which had the same shape and no
spec. The flag is kept rather than deleted so an operator who set it is told the
promise is gone instead of having the setting ignored, and the guard around
StartTunnel is removed, because a branch nothing can take reads as a supported
no-tunnel mode that does not exist.
The justification for erroring on an install that names no address was wrong,
and the review is right that this is the dangerous form of overclaiming, because
the conclusion holds and the mechanism does not. It said the resulting empty
target would be refused as an invalid stream and that the refusal would read as
the worker answering about its backend. Nothing in this repo branches on
cluster.ErrNoRoute, and nodes.unroutable treats any recorded dial error as
unroutable, so that refusal reaches every reap guard as ProbeUnknown and deletes
nothing. The site now stands on what holds, that an install naming no port
produced nothing routable and the failure belongs to the install rather than to
a later probe, and records the retracted claim so nobody re-derives it. This
retracts the same paragraph in the body of
|
||
|
Ettore Di Giacinto
|
1cf847f29e |
feat(distributed): stop workers listening, and stop them advertising
A worker now opens no listener on a routable interface and states no endpoint at registration. Backend processes and the file-transfer server bind loopback, and the frontend reaches both through the tunnel the worker dials. The bind address is built from loopbackHost, the same constant the tunnel's grpc tag dials, so "the worker binds where its tunnel dials" is one fact in one place rather than two literals that can drift. All three advertisement sites are closed, not one: the registration body, RegisterNodeRequest, and the per-backend address in the install reply. That third one was hiding a live bug. stopModelExact refuses a stop whose ExpectedAddress does not match what the worker recorded for the process. The worker recorded 127.0.0.1:port; handleBackendInstall reported advertiseHost:port; the router stored the reported one and sent it straight back. On any worker whose advertise host was not 127.0.0.1, every acknowledged model stop failed with an address mismatch. Nothing caught it because the e2e harness set LOCALAI_ADVERTISE_ADDR=127.0.0.1, which made the rewrite a no-op. Removing the rewrite makes the two strings the same by construction. The brief was wrong about two of the four functions it called dead. effectiveBasePort is the base of the backend port allocator and resolveHTTPAddr is the file server's bind address; deleting them would have deleted the port allocator and the file server. Only the two advertise* helpers were dead, and addr_test.go is rewritten rather than deleted, because the port arithmetic it pinned still needs pinning. NodeModel.Address survives with a narrowed meaning and is renamed WorkerLocalAddress, along with the install reply field that feeds it. The frontend still has to say WHICH backend process on a worker it means, and the port in this string is how it says it: it travels as a stream target and the worker dials its own loopback. The gorm column and the json key stay "address", so neither a migration nor an API break rides along. Every fall-back to the node's address is gone. installBackendOnNode now errors when a worker reports success without naming one, because substituting the now-always-empty node address would name an empty target, and the worker refuses that as an invalid stream, which is classified as the worker answering about its backend. That is the "a present worker reads as something it is not" class this phase forbids. DistributedModelStore.Range had the same shape and was already wrong: it built each remote model's client from the node's base gRPC port, never the port a backend process listens on, so Free and Status went to the wrong place. It uses the replica's address now. BackendNode.Address and HTTPAddress are kept but made provably inert: no writer, no reader that acts on them, and Register force-clears both on re-registration so an upgraded worker's stale advertisement does not outlive its own upgrade in the API and the Nodes page. Dropping the columns is a ~90-site edit across the specs, the e2e suite, the MCP dto and the UI; it is recorded as a follow-up rather than folded in here. A persistent tunnel 401 still does not trigger re-registration, and now for a reason rather than a deferral. Register CLEARS the node's replica rows, so re-registering on a 401 would delete a live worker's rows on every retry, and under the name collision that causes the 401 the two workers would take turns doing it forever: a credential failure causing model reclamation. It also cannot fix the named cause, since a collision is indistinguishable from a restart. The 401 log now names both causes and says nothing can reach this worker, which is true only now that it has no listener. The container healthcheck did not break the way the brief expected, since the listener still exists on loopback and the probe runs inside the container. It did have a real #10987 defect that this change makes the common case: it read LOCALAI_SERVE_ADDR only, while effectiveBasePort reads LOCALAI_ADDR first, so a worker on a non-default base port was probed on 50050 and reported unhealthy while working. It follows the same precedence now. Docs, the compose file and the e2e harness are updated in step: no inbound rule or published port is needed for a worker, the two advertise variables are gone, the remaining address variables are read for their port only, the firewall-the-file-transfer-port warning is narrowed to the LOCALAI_HTTP_ADDR opt-out, and the upgrade-order note no longer claims the worker still listens. The Nodes page showed node.address, which is now always blank, so it shows the node id instead. Eight mutations, all red on a named spec, including reverting the loopback bind, re-adding the address to the registration body, restoring both node-address fall-backs, dropping the force-clear, storing the endpoint's address again, and un-fixing the healthcheck. One of them caught a defect in a spec I had just written: it asserted 200 where the endpoint returns 201, which went unnoticed because core/http/endpoints/localai is not on the task's verify list. It is run here. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
ed9a4b6b52 |
docs(grpc): withdraw the lint-cost claim, and make the one nolint tidy-proof
I reported that enabling gocritic pushed make lint past 600s. That was wrong, and it was wrong in a way worth naming: those runs happened right after I changed pkg/grpc's and core/services/nodes' interfaces, so the Go build cache was cold for essentially the whole repository including every backend, and test suites were running concurrently on the same machine. I attributed a cold-cache full-repo typecheck under load to the linter I had just enabled, and raised it as a cost without ever timing it against a baseline. A number with no control is not a measurement. Measured properly, with the golangci cache cleaned before every run and isolated GOCACHE directories for the cold ones so the shared cache was not wiped: warm, base 15s then 7s and current 8s then 7s; cold, base 87s and current 78s running base first, base 136s and current 79s running current first. The spread between the two cold base runs is larger than any gap between base and current, so gocritic with only the ruleguard checker costs nothing measurable. So the rule stays, unscoped. Scoping it to core and pkg was the fallback for a cost that does not exist, and adding that configuration would buy nothing. The one override gets the protection it needs instead. InFlightTrackingClient's nolint is exactly the kind of thing a later reader tidies away, so it now opens by saying not to, and states what breaks rather than what is intended: WrappedBackend embeds the full Backend interface, so adopting it there would promote every inference method as untracked pass-through, the build would stay GREEN, and in-flight accounting would silently stop covering whatever was added next. WrappedBackend's own doc carries the counterpart warning so a reader arriving from either side finds it. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
b8d47cc29b |
test(nodes): make the joiner spec deterministic, and gate the wrapper shape in lint
Re-review round 2. One blocking item, and it was a spec I wrote: eight goroutines raced at the probe cache and nothing made them coalesce, so a straggler that missed the flight re-entered the probe and double-closed a channel. It panicked about one run in three and took the four-suite race block down. The green verification I reported was not reproducible, which means one green run was never evidence for a spec that coordinates goroutines. Its comment claimed the probe blocked until every goroutine was inside flight.Do, and that gap was exactly the panic: the comment described the design intended rather than the one written. It is deterministic now rather than tolerant. singleflight.DoChan registers its channel on an in-flight call under the group's own mutex and returns without running its function, so calling it while the leader is provably parked inside the probe joins that exact flight with no window and no dependence on the scheduler. The spec asserts the join really happened, that the joiner got the reason and not only the answer, and that the probe ran once; the entered channel is sent on rather than closed so a second probe fails an assertion instead of panicking. Twenty runs green under race against the committed code, five out of five red on the mutation back to a closed-over variable. The future-decorator gap is closed in the lint gate, but not the way the review suggested, and the reason is worth recording. HasMethod rejects inline signatures outright, its method-reference form needs a package ruleguard's own typechecker can import and that typechecker cannot import this module, and Implements tests the value method set while every Unwrap is on a pointer receiver, so it fired on all three wrappers that already had one. So the safe shape is structural instead. grpc.WrappedBackend gives the same pass-through method set plus Unwrap on a value receiver, and a decorator that embeds it is transparent by construction; forgetting stops being expressible rather than merely discouraged, which is the move loopbackService already makes in the worker. FileStagingClient and ConnectionEvictingClient embed it and their hand-written Unwrap methods are gone. The ruleguard rule then only has to catch the raw embedding, needs no type filter, and cannot misfire. It was verified to fire on a throwaway wrapper and stay silent on a correct one, and reports nothing across core and pkg with the baseline disabled. InFlightTrackingClient is the one exception and says why in a nolint: it embeds ControlBackend deliberately so that leaving an inference method unwrapped breaks the build, and WrappedBackend embeds the full interface, so adopting it would silently restore pass-through for every inference method and delete that guarantee. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
b4d8e23abb |
fix(grpc): let the transport answer through the wrappers, not only past gRPC
Re-review round 2. One blocking defect, and it was the concern I filed myself last round and mis-scoped as a future trap. It was live, and it sat on the most destructive reaping path of the five. RouteResult.Client is an InFlightTrackingClient, over a FileStagingClient when a stager is configured. model_router puts that on the cached remote model and pkg/model's checkIsLoaded asks IT whether the transport failed. Both wrappers embed grpc.Backend, which does not declare LastDialError, so the type assertion read nil and the guard added last round fell straight through to the old eviction. That eviction sends backend.stop over NATS to every node holding the model and deletes every replica row, where the other sites delete one. The spec covering it built a bare client by hand, which is why it passed while production did not. This is the third time in this task a correct fix was disarmed one layer out, so the fix is a mechanism rather than two methods. BackendUnwrapper is one line per decorator, LastDialErrorOf walks the chain, and both consumers now call it instead of each keeping its own assertion. One implementation, no per-caller policy to get wrong. Sweeping every type that embeds or holds a grpc.Backend found a third decorator the review had not named, and it is itself a reaping consumer of the same collapsed signal. ConnectionEvictingClient is built for remote models in initializers.go and its evict callback runs ShutdownModel; it fires during INFERENCE rather than on a health check, so a tunnel blip mid-request was enough to stop a model that was loaded and serving. It consults the transport first now. A locally spawned backend has no custom transport, so that path is unchanged byte for byte. Everything else touching a Backend is a consumer rather than a decorator; there is no fourth. The probe cache joiner shape is pinned. It was the right design last round with nothing holding it: the mutation back to a closed-over variable passed all 602 specs in the package. Eight goroutines coalesced on a probe that blocks on a channel now assert every joiner gets the leader's REASON and not just its answer, which is the difference between a leader declining to reap and its seven joiners reaping on the leader's own observation. The LastDialError scope note claimed an exactness it does not have at checkIsLoaded, which reads a shared long-lived client after releasing opMutex. It now says which caller is not exact, why the imprecision is accepted there, and what making it exact would cost. The four-outcome table in the docs still said a worker with no live owner is treated as absent and rescheduled, contradicting the code and the paragraph nine lines below it. None of those outcomes is absence any more, and the table says so, names the fifth, and points at the heartbeat as the thing that does decide presence. Five mutations, each reddening named specs, including the two the reviewer found surviving. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
a8ac2af167 |
fix(cluster): make "no route" a condition of its own, and let it out of the package
Review round 1 on task 6. Five blocking findings, all with the same root: the conditions the dialer kept apart were erased one layer out, because every one of them arrived at core/services/nodes as a gRPC codes.Unavailable, which is also what a backend process that died produces. Four call sites acted on that by deleting a replica row, one of them after a single failed probe. The fifth condition is ErrNoRoute: this replica could not get a request to a worker's backend, and no claim at all about the worker. A worker's presence is its HEARTBEAT, which nodes owns; a route is a separate fact that cluster owns, and the two now differ. They differ in normal operation, not exotically: a worker that has not dialled its tunnel yet after a frontend-first upgrade is unroutable on every request while it heartbeats and serves. Two properties, both mutation-tested. Every failure to resolve or open a route carries ErrNoRoute, so a consumer has one check to make. No failure carries an absence sentinel: routeFailure is the single place that rule lives, and it keeps ErrNoConnection and ErrInstanceNotFound in the message and out of the unwrap chain, the guarantee unreachableError already made for peers. Everything else stays matchable, so ErrNotOwner and ErrPeerUnreachable are unchanged for anyone who can act on them. A worker's own refusal carries no umbrella, because a worker that answers has demonstrated it is there and that is the only real evidence on the path. Crossing the boundary needed a value, not a code. NewClientWithDialer wraps the dialer and records each outcome; LastDialError hands it back behind a narrow interface, and nodes.unroutable turns it into ErrWorkerUnroutable with the cluster sentinels still in the chain. A spec asserts a dial failing with ErrNoRoute plus ErrPeerUnreachable arrives matching all three and matching neither absence sentinel. The sweep found a fourth site the review had not named: pkg/model checkIsLoaded evicts a remote model on a connection error, and a tunnel dial failure is one. Four other reap sites were cleared with reasons - inflight and the worker authoritative pass reap only on semantic answers, scale-down is driven by last_used, abandoned loads decide on the node's heartbeat. Every fixed site also grew the opposite spec, so the new check cannot pass by never reaping. probeCache carries the reason through singleflight rather than a closed-over variable. A variable is only written by the goroutine that runs the probe, so the leader would correctly decline to reap while every joiner reaped on the leader's own observation; a mutation reproduces exactly that. The docs sentence promising LOCALAI_WORKER_TUNNEL=false restores direct dialling is gone. There is no such path, so it said the operator could take a worker dark and call it a rollback. Replaced with the upgrade order that is actually safe. The deadline spec the reviewer found vacuous now waits on the dial context's own Done channel before touching the stream, so the armed deadline has really expired; the mutation that survived for the reviewer reddens it. Nine mutations, each reddening a named spec, including both halves of isAbsenceClaim independently. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
75953d9f63 |
feat(cluster): reach every worker through its tunnel, never its address
The tunnel, the fence, the registry and the relay were all built and none of them carried a byte: every dial from the frontend still went to the address a worker registered. This is where that stops. One WorkerDialer resolves where a worker's tunnel is held, opens a stream on it locally or relays through the owning replica, and hands back a conn past both handshakes; gRPC, the file stager's HTTP client and the log-streaming WebSocket are all pointed at it. A worker's address stops being somewhere to connect to and becomes the name of which backend process a stream is for. It still appears in URLs, logs and errors, because that is what identifies the process; what it no longer decides is where the bytes go. Nothing falls back to dialling it. BackendClientFactory now has exactly one method, NewClientForNode, and returns an error where there is no way to reach the worker. The direct-dial constructor was removed rather than kept beside it, because leaving one on the interface keeps the bypass one word away from every call site that holds an address, which is all of them. The second construction path is closed too. DistributedModelStore built remote models with a nil client, and pkg/model.Model.GRPC then dialled the raw address lazily on first use - reached in production by ShutdownModel's Free and by the backend monitor's Status. Those models now carry the tunnel-backed client, and a model that cannot be given one is logged and not listed. Four conditions stay unmixable, and one path produces absence: the dialer answers ErrNoConnection only where Owner's liveness join did. A peer that will not answer, a stale ownership row, a worker's own refusal and a missing relay path are each reported as themselves. This matters because nodes ACTS on absence, and the collapse would have it reclaim the models of a worker that is connected and busy. That is not hypothetical. Writing the mutation for it exposed the bug in this change's own first draft: probeHealth returned bare false when it could not build a client, and tryWarmPath deletes the replica row on a false probe. A frontend whose dialer broke would have emptied node_models for the whole deployment while every model kept running. probeHealth now returns alive and probed separately, the reconciler gets a ProbeUnknown outcome that neither advances nor clears a failure streak, and the health monitor skips rather than counting a miss. Task 5 left the relay's open timeout at a fixed 15s and said so: no operator has the information to set it, because the number that matters is the original client's remaining budget, which is invisible on the relay side. The dialer has that budget, so it now states it in the relay request frame and the owner takes the smaller of the two. It can only shorten - a patient client must not be able to park a relay goroutine and a stream slot on a worker that stopped accepting. Zero is written as no budget at all, since on the far side the number zero is a caller with nothing left and would refuse healthy traffic. Seven mutations, each reddening a named spec: peer-unreachable as absence; the local-failure guard dropped; max instead of min on the budget; the nil-client model restored; ProbeUnknown falling through to the reaper; OwnerRow instead of Owner; probed collapsed into alive. The first budget spec passed for the wrong reason - a handshake deadline, not the relay - and was replaced by three that each assert one link, including one where the spec plays the owning replica and reads the budget out of the frame instead of inferring it from a clock. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
1036f5664e |
fix(cluster): express the splice policy once, and pin the relay's budgets
Review round 1 on task 5. Eight non-blocking items, all addressed.
The classifier read Remote in two predicates with a report-by-default
fallthrough behind them, so reverting either read left the whole suite green:
the error reached the same answer down the other path. A correctness argument
that rests on mutation evidence cannot afford a shape that cannot be mutated in
pieces, so the two predicates collapse into one muxVerdict deciding each error
type once. Falsifying either Remote read now reddens exactly one spec.
Three claims the comments made loudly and nothing tested:
- clearing the header read deadline before the splice. Deleting the clear
left all 49 focused specs green, while in production it is the difference
between a relayed response that streams for an hour and one that dies after
fifteen seconds of quiet;
- the open budget bounding the open and nothing after it;
- closing the worker-side stream when the acceptance reply cannot be
written, which leaks one stream on the worker per failure.
All three are pinned now. The first two share a spec that sets both budgets to
50ms and then watches the conversation outlive them by ten times, which is an
assertion about an event that must not happen and so is the one wait a channel
cannot replace. The third drives the relay with a peer stream that delivers a
request and then fails every write, because no pair of live yamux sessions can
be made to fail that write on cue.
The disjoint-vocabulary argument was specced for the accepted frame only. Both
refusal directions are covered now, and asserted as "not one of the other hop's
sentinels" rather than merely "an error", since reading a relay refusal with the
tunnel's reader always errors and the question is whether it errors as the wrong
thing.
The open budget stays non-configurable, and says so: the number that matters is
how long the original client will wait, which is not known on this side and is
not something a deployment-wide constant can stand in for. The honest fix is the
caller's remaining budget travelling in the request frame, which belongs to the
dialler that has the budget.
Two comments corrected: nothing deadlines the peer stream after the clear, so
the tunnelled protocol's own deadlines cannot be what justifies clearing it; and
the membership sweep deletes departed replicas but reports only how many, so
identifying them is work that would have to be done, not knowledge waiting to be
plumbed. Recorded at muxVerdict: a remote RST that does not ride a
typeWindowUpdate frame yields the bare sentinel and is still silenced, which is
unreachable between two go-yamux peers but keeps the new rule from reading as
unconditional.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
|
||
|
Ettore Di Giacinto
|
cce914b3fd |
feat(cluster): relay a peer's stream onto a worker tunnel held here
A worker holds ONE tunnel and it lands on ONE frontend replica, so with N
replicas behind a load balancer roughly (N-1)/N of requests arrive somewhere
that cannot reach the worker directly. This is the piece that carries them:
the SessionStore stream handler reads which worker a peer's stream is for,
opens a stream on the tunnel this replica holds, and splices the two.
Splice has had no production caller since phase 1. It has one now, and being
the first caller it settles the two endings phase 1 deliberately left open,
both of which read as normal termination until now:
- a peer-initiated *StreamError{Remote: true}, which yamux builds only from
an RST frame the far side sent (stream.go:432-449); a reset this side asks
for carries Remote: false, and Splice never resets anything, its own Close
sending a FIN;
- a graceful ErrRemoteGoAway, which handleGoAway returns for code
goAwayNormal (session.go:829-833) and close hands unwrapped to every live
stream (session.go:328-337).
Both truncate whatever was in flight. Reporting them as normal termination is
how a half-finished inference comes to look like a short one that completed,
so both are now reported; the local forms stay silent, because those are the
teardown Splice provokes itself. The decision cannot live in a caller reading
Splice's result, since a result already mapped to nil carries nothing left to
reclassify, so it lives at the classifier with the reasoning beside it. The
relay logs it at debug: a client cancelling a relayed request produces one per
cancellation, and the truncation is separately visible to the frontend's own
gRPC or HTTP client.
The relay hop gets its own request and reply frames. They have to be distinct
from the worker tunnel's, because a relayed stream carries both hops' frames
back to back, and a vocabulary shared between them would let a reader applied
to the wrong hop hand back a plausible sentinel belonging to the other. Its
three refusals stay apart for the reason the worker's three do: ErrNotOwner is
a routing fact and the caller should resolve the owner again; unavailable is
infrastructure at this replica and a retry is worth something; bad-request is
the caller's bug. None of them is, or may be built over, an absence error.
One hop, always. A stream naming a worker this replica does not hold is
refused, never resolved and relayed onward, so a stale ownership row cannot
become a loop between two replicas each certain the other holds the worker.
PeerPool is constructed and closed alongside SessionStore, so both halves of
the peer mesh now have an owner and a shutdown.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Ettore Di Giacinto
|
3b6d32c1c4 |
fix(worker): make the tunnel credential's node-type gate actually structural
Re-review follow-up, three items. Two are the overclaiming-comment class again, and the first is that class with a real defect underneath it. attachTunnelToken said "enforcement is therefore structural": an ineligible node never gets a credential, so its hash stays empty and the tunnel route's empty-hash branch does the refusing. That was true for a node that had always been an agent and false for one that had not. Register upserts by NAME, so a backend node re-registering as an agent keeps its ID, and Register's struct Updates zero-skips the credential column while writing the new node_type. The early return left the credential the node earned as a backend sitting on a row that is now an agent, and ConnectHandler never looks at node_type. Fixed by making the claim true rather than by softening it, because the mint-site gate was chosen precisely on the grounds that it was structural: an ineligible node now has its column CLEARED, unconditionally, so the invariant does not depend on what the row happened to contain. A spec pins it and was red before the change. Same shape as the Register-upserts-by-name hazard already carried forward: a name is not an identity. Second, loopbackHost claimed to be the only host any tunnel stream is ever dialled on. It is not: fixedService dials whatever Run built it from, which is this worker's own LOCALAI_HTTP_ADDR, and loopbackAddr rewrites only a wildcard bind, so an operator who binds the file-transfer server to a routable address gets a routable dial. The property that matters is narrower and is what the comment says now: the frontend cannot STEER the dial. The grpc tag builds its address from a constant and a validated port with nothing from the wire reaching the dialler, and the http tag ignores its target entirely. Worth stating exactly rather than summarising, because the argument about what a stream can reach rests on knowing which hosts are reachable, and an overstatement at that site is what would let someone conclude the constant alone is doing the work. Third, a spec named "without allocating it" measured no allocation. It now asserts the mechanism the defence actually rests on, that the reader consumes the two length bytes and not one byte of the body, through a counting reader. The input carries a body on purpose: against input that ends after the header the assertion would pass with the limit check deleted. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] |
||
|
Ettore Di Giacinto
|
5108be222d |
fix(worker): spec the tunnel's routing table, which was the SSRF boundary
Review follow-up. One blocking finding and seven others. The blocking one first, and it is this project's recurring shape: the untested path. loopbackService is the function whose comment calls the discarded host "the security property this function exists for", and nothing tested it. The reviewer replaced its body with a dial of whatever the frontend named, no port range, and all 131 specs passed. Every spec installed the permissive test dialler, so the real routing table was exercised nowhere. It now has specs, and the property is stated as reachability rather than as a property of the code: a listener on 127.0.0.2 that only the frontend's target names must NOT be reached. Plus the port-range table, fixedService, loopbackAddr, tunnelEndpoint, and the table itself, which moved out of Run into tunnelServices so it can be built without starting a worker. One spec drives a real stream through that table over the wire, so the routing rules are exercised end to end at least once rather than only in isolation. The reviewer's mutation now reddens ten specs, and six narrower ones redden between two and four each, so no spec is riding on another. The shape changed too, not only the coverage. The dial address is built from a loopbackHost constant and strconv.Itoa of a validated int, so nothing derived from the wire reaches DialContext at all: restoring the hole takes ADDING a data flow, not deleting a check. And a taxonomy fix found while specifying it. A port outside this worker's allocator range was reported as unavailable, which tells a frontend to retry something that can never work. It is a bad request now, and a backend that is merely not listening yet stays unavailable, which is the retryable one. Agent nodes no longer get a tunnel credential. Nothing dials into an agent worker, so a tunnel replaces nothing for it and no client would open one, and the gate is at the mint site rather than in the handler: with no credential minted the hash stays empty and the existing empty-hash refusal covers it, so enforcement is structural. Two comments and one doc paragraph said an anonymous registrant gets a "working" credential. With auto-approve off the node is pending and the credential is inert, which is the distinction this same change argues three files away to justify minting for pending nodes at all. A refusal reason over the frame limit was cut on a byte boundary and could split a rune. It cuts on a rune boundary now, and the code survives truncation, which is what keeps a refusal classifiable. Also: the pending-node spec asserted only that a credential was non-empty, so a credential derived from the shared token passed it; it now pins per-node-ness the way the headline spec does. The tunnel handler's citations into nodes.go were stale before this branch landed, having been written against a file the same commit was editing, and are by function name now. The static-NATS path says plainly that an externally forced rotation locks it out until restart, and where that gets fixed. tunnelproto gained direct specs, including that a read failure is never reported as a refusal. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] |
||
|
Ettore Di Giacinto
|
29a2020f3d |
feat(worker): dial, hold and serve the tunnel, on a credential of its own
The worker end of the tunnel. It dials wss://<register-to>/api/cluster/connect, holds one yamux session as the CLIENT, and serves every stream the frontend opens on it. Nothing dials into the worker, which is the point: no inbound port, no reachable address. Each stream opens with a length-prefixed frame naming a tag and a target, and the worker answers before either side speaks the tunnelled protocol. The reply is sent on every stream, not only on refusal, because the protocols carried here are client-speaks-first and a reply sent only sometimes would arrive interleaved with a response body. Two tags today: grpc reaches a backend process, and only on 127.0.0.1 within this worker's own backend port range, because a tunnel terminates inside the worker and letting the frontend name a host would make every worker a proxy into its own LAN; http reaches the worker's file-transfer server, whose address the frontend is not asked about. An unknown tag, an unreachable local service and an unparseable request are three refusals and stay three on the wire. A frontend gives up on the first and retries the second. Each is answered AND the stream is ended: a worker that says why and leaves the stream open has parked the caller on a request nobody will answer, and a deadline on the far side cannot tell that from a slow worker. The specs assert the stream ends rather than that an error occurred, which is what phase 1 shipped in three places and held in none. Reconnects double from 500ms to a 30s ceiling, each wait drawn between half the interval and all of it, and the interval returns to its floor only after a session that LASTED. Resetting on connect is how a rolling restart, where every dial succeeds and dies moments later, becomes a retry storm against the first replica back up. Nothing is assumed to survive a reconnect: the credential is read at dial time, never captured. And the credential is now real. The tunnel endpoint advertised authenticating a worker against its own secret, but registration stored the hash of the shared registration token, so a leak plus a known node ID still opened a tunnel. Registration now mints a per-node secret, returns the plaintext once as tunnel_token, and stores only its SHA-256 in a new column; the endpoint compares against that and does not fall back to the old one. Rotating on every registration follows from storing only the hash, since a re-registering worker cannot be told the secret it already holds; its live tunnel is unaffected, because the credential is checked when a tunnel is dialled and never again. Unlike the agent API key and the NATS JWT next to it, the credential IS issued to a node awaiting approval: the tunnel route re-reads the node's status on every dial and refuses a pending one, so it is inert until an admin acts, and withholding it would strand every worker that registers exactly once. A node that has not registered since this change cannot tunnel, and the column cannot be back-filled because the plaintext only ever existed in the response that minted it. The boot warning that said tunnels need LOCALAI_REGISTRATION_TOKEN is replaced: it was true while the tunnel authenticated against that token's hash, and says the wrong thing now. What is still true, and is what it warns about instead, is that without one, registration itself is unauthenticated. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
5b7d65e679 |
fix(testutil): clear the maintenance timeouts at connection startup
The guard added last commit was circular. It cleared the maintenance database's timeouts by executing SET statement_timeout = 0 on a connection that had already inherited that database's bound, so the statement clearing the bound ran under the bound it was clearing. Under the white-box spec's deliberate 1ms that gave it 1ms, and it failed roughly once in fifty at 8-way concurrency with SQLSTATE 57014. The guard against invisible load-dependent flakes had become one. The clearing is now delivered as a connection startup option, options=-c statement_timeout=0 -c lock_timeout=0 on the maintenance DSN, so there is no statement left to abort. Raising the imposed bound would only have bought headroom and left the circularity in place. pgx puts every URL query parameter into settings, options is absent from notRuntimeParams so it becomes a runtime parameter, and runtime parameters are copied into the startup message (pgconn/config.go:340-378, 606-617; pgconn/pgconn.go:382-388). The spec now discriminates on pg_settings.reset_val, the value in force when the connection started: 0 for a startup option, 1ms for a session SET. A first attempt using a deliberately slow first statement did NOT discriminate, because under the circular design the SET is itself the first statement, so by the time a spec runs anything the session is already unbounded. Reinstating the circular clearing now reddens the spec deterministically rather than intermittently. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
a816bf9b84 |
fix(testutil): stop the shared-database change from disarming two regressions
Two advisory-lock specs named their database by literal, ALTER DATABASE testdb. Once the test helper started handing every spec its own database on a shared server, that statement landed on the maintenance database and did nothing to the one the spec was holding, so both specs went green having never reproduced the condition they exist for. They regress a model-load advisory-lock wedge that has already shipped to production once, so the previous commit's de-flaking silently disarmed a regression test for a real deployed bug. Both sites now read the name back with current_database() and, more importantly, assert the override actually landed before relying on it. A literal name can go stale again; an assertion that the setting is in force cannot pass while it is not. Removing either production override now fails the matching spec with the real 55P03 and 57014 again. That literal also meant every CREATE DATABASE and every DROP ... WITH (FORCE) ran under the 300ms bound it set on the maintenance database, which is a new load-dependent single-spec flake inside the change that was meant to remove one. The helper's maintenance connections now pin one connection and clear both timeouts on it, so no setting a spec makes can bound them, and a white-box spec imposes the leak deliberately and proves it does not reach them. Also pins the reclaimOne gate deferral the previous commit added without a test, by panicking inside the re-claim's own claim statement, and drops the per-dial empty-token log line to debug now that the boot warning says it once. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
48ece89c63 |
fix(cluster): harden the worker tunnel, and stop starting a database per spec
Review follow-up. Twelve findings, none blocking, grouped here by what they protect. Panics. The handler now recovers between the WebSocket upgrade and the hand-off, the way the peer link next door already did: net/http recovers the panic but leaves the hijacked socket open, so without this a worker keeps a session this replica has no entry for and will never detach. The claim gate in Attach and reclaimOne is now released with defer, so a panic under Claim cannot wedge one node's gate for the life of the process. SetTunnels gained the nil-receiver guard its sibling Stop has. Operability. A deployment with no registration token stores an empty token_hash on every worker, so every tunnel dial 401s forever on a frontend that looks correctly configured. That now warns at startup, logs its own line rather than sharing the "wrong token" one, and is stated in the docs together with the fact that setting the token later needs the workers to register again. Authorization. A node still awaiting admin approval is refused with 403. The rest of /api/node/ gates on nothing, but the two places that hand a node something durable, its API key and its NATS credential, both refuse a pending one, and a tunnel is that kind of grant. Draining and unhealthy nodes keep their tunnels on purpose. Comments that claimed more than the code. The global auth middleware does run on this path and then declines to reject; the future per-node secret only lands without a change here if it lands in TokenHash; the empty-hash guard is defensive rather than deciding; ClusterPathPrefix is no longer only replica-to-replica; the docs no longer say a reaped replica re-claims unconditionally. And the test harness. SetupTestDB started a PostgreSQL container per BeforeEach with a readiness deadline it asserted on, which is one chance per spec to fail one spec inside its setup, anywhere, never twice in the same place: the shape of the flake seen twice here and never reproduced. It now starts one container per process and creates a database per call, which is the pattern tests/e2e already proved. Isolation is unchanged and is now asserted for the first time. All 69 call sites are untouched; the eleven consumer packages run 1404 specs green, and jobs went from 34.3s to 3.3s, agents from 13.8s to 1.9s, cluster from 97.4s to 37.5s. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] |
||
|
Ettore Di Giacinto
|
6e55092a4b |
feat(cluster): open the door a worker dials its tunnel through
A worker needs no inbound port: it dials GET /api/cluster/connect, the connection becomes one multiplexed yamux session, and the frontend opens a stream on it per request. This adds the endpoint that accepts that dial and attaches it to the tunnel registry. The dial is authenticated against the NODE's own stored token hash rather than the deployment's registration token. That is the mechanism, not yet the isolation, since a worker still registers by presenting the shared token; what it rules out is the shortcut of comparing against the configured value, which would have to be unpicked the day workers get their own secrets. Every refusal happens BEFORE the WebSocket upgrade, so a dialer reads an HTTP status rather than a handshake error. The route is registered in every deployment, single-binary ones included, which is what puts it in front of the route-coverage test that holds that rule in place; with no node registry it refuses every dial, and tells a credentialed one the frontend has no cluster rather than that its token is wrong. A lookup that FAILED is answered as a failure. Reporting a database that could not be read as "unauthorized" would send a worker re-registering, throwing away the identity its tunnel and loaded models are keyed by. Wires the tunnel registry in core/application/distributed.go and hands it to the membership loop. Without that call the re-claim after a replica is reaped had no production caller and could never run. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] |
||
|
Ettore Di Giacinto
|
62476e553e |
fix(cluster): keep a session close out of the per-node claim gate
The gate is justified by being held for one claim round trip, and Attach held it across the close of the session it superseded. Closing a yamux session closes the underlying conn and then waits for both its send and recv loops to exit, and the send loop can be inside a write bounded only by ConnectionWriteTimeout, so that is a wait on other goroutines. It must not stand between a worker re-dialling this node and its claim. The gate is now released after the store and before the close, which also makes Attach match reclaimOne, where it has always been released explicitly on every path. This is safe because a superseded session is no longer reachable from the map by the time it is closed: the next re-dial replaces an entry that already names the new session. Pin the re-claim half of the gate too. A worker that re-dials between a re-claim's commit and its record leaves the row carrying the re-dial's epoch while the entry carries the re-claim's, so the attachment holding the socket releases an epoch the row does not have and the row outlives it, with nothing to sweep it while this replica is alive. Only Attach's half of the serialisation was asserted; keying the two apart left every spec green. Also take the test hook's action under the lock that guards whether it has fired. It was written from the spec's goroutine and read from whichever goroutine issued the statement, which is a race in the harness that pins the serialisation specs. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
63ed55a929 |
fix(cluster): make a claim and its record indivisible per node
Two Attach calls for one node both claim, and PostgreSQL serialises the two upserts, but nothing ordered the two map writes against the two commits. The entry left installed could be the one whose claim lost the row, and its Detach then released an epoch the row does not carry, so the release matched nothing and the row survived the socket. Nothing swept that row. This replica is alive and heartbeating, so ReapStale leaves its rows alone, and no reconnect is coming for a worker that has gone. Owner kept naming this replica as the live owner of a tunnel it no longer held, and every dialer sent here was answered ErrNotOwner, which is the relay into a replica that cannot serve the request that this phase exists to prevent. Claims for one node now pass through a gate, so claim and record are indivisible. It is per node rather than one lock over the registry, the way PeerPool locks per peer: the claim is a database round trip, and a slow one for a single worker must not hold up Open for every other. Detach is not gated, because it takes no context and must never park behind an in-flight database call, and it changes no epoch. Reclaim takes the same gate, which makes its claim the newest one for that node, so it records the epoch on whatever attachment is installed rather than only on the one it listed. Refusing to record onto an attachment that replaced the listed one would leave that row with nothing able to release it. The interleave the gate does not cover is Detach, and a claim whose attachment detached while it was in flight is now released again rather than left behind. Also: restore Start's doc comment, which SetTunnels had swallowed; keep reaping other replicas when this one fails to rebuild its own row, rather than skipping the sweep along with the re-claim; scope the comment about an unnoticed dead socket to the keepalive of the session whoever accepted the tunnel built, since the worker session config does not exist yet; and pin the sortedness of Held, the nil-session refusal, and both re-claim interleaves with specs. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
f3ba1f692b |
feat(cluster): hold worker tunnels, and re-claim them after a sweep
Phase 1 left the connection fence with a table and no sockets behind it. This adds the registry that holds them: Attach claims the node and then stores the session, Open hands out a stream over the tunnel this replica holds, Detach releases the claim it was handed, and Held names what this process is carrying. The claim is written before the session is stored. A claimant that installs itself and only then finds it cannot claim has, for that window, published a tunnel no row records, so Held names it while a peer asking Owner is told the worker is connected nowhere. ErrNotOwner is produced at one place, the map miss. It is a routing fact: some other replica may hold that worker perfectly well. A broken socket under a held entry is returned as itself, because answering "not held here" would send a dialer looking elsewhere for a worker this replica is holding. Epochs are compared for equality and never ordered. Claim guarantees an epoch is unique and never reissued; it does not guarantee the later claim draws the larger number, because the sequence value on the insert path is drawn before the row lock. The membership loop now re-claims on re-register, which closes the hole phase 1 named in ReapStale. A replica that stalls long enough is swept by a peer, losing its instance row and, in the same transaction, every connection it owned; Register rebuilds the instance row and nothing else, so without this it serves workers that every other replica reports as connected nowhere. Re-claiming draws a fresh epoch, so an attachment carries two: the token Attach handed back, which is what Detach matches and which never moves, and the epoch of the row currently held, which is what Release is given. Collapsing them would leave the re-claimed row outliving the socket with no caller able to remove it. A tunnel whose session is already closed is skipped rather than claimed back, because claiming is an upsert and would take the row from whoever holds the worker now. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
e4777915f9 |
fix(cluster): make the ownership comments say only what holds
Review round 1 on the joined Owner read. The behaviour was accepted; three
comments claimed more than the code delivered, one spec pinned less than its doc
promised, and one pre-existing spec ranked epochs.
instanceIsLive said every reader of instance liveness uses it, which was false:
ReapStale spelled the complement by hand. The complement is now written as
NOT (instanceIsLive), so "stale" is exactly "not live", including how each side
treats a NULL last_seen, and the sentence is true. Inverting that predicate
fails 3 reaper specs, so the routing is held.
The Select("node_connections.*") in Owner was justified by a SELECT * hazard
that cannot occur: with a join present and nothing selected, gorm expands the
model's own columns table-qualified (callbacks.BuildQuerySQL), and the suite is
green with the Select removed. It stays, because the projection should be a
property of this query, and the comment now says that instead.
Owner gained the dialect guard Claim has. now() and make_interval are
PostgreSQL, so on the SQLite single-binary path it failed with "no such
function: now", which reads as a missing migration; that regression already
shipped once in phase 1. The refusal is deliberately not ErrNoConnection: a
deployment with no cluster has no answer about ownership, and reporting absence
would let a caller conclude the worker is not connected. A spec in the
non-PostgreSQL block holds all three properties.
The new specs aged rows by ten minutes, which any window between zero and ten
minutes satisfies, so nothing tied Owner's window to the one the sweeper uses.
They now age to just past InstanceLiveness, and a sibling ages to half of it and
must still resolve. Widening the window tenfold fails 2 specs, narrowing it
tenfold fails 1; before this both were silent.
The concurrent-claim spec asserted the stored epoch was the highest handed out,
and justified it with claims drawing their epoch after the row lock, which
contradicts Claim's own doc: the insert path draws nextval while the tuple is
built. It now asserts the stored epoch is one of the epochs handed out, and
ranks nothing.
OwnerRow's doc justified the function with a sweeper that does not call it.
ReapStale deletes orphans with a set difference; the callers are this package's
specs and one e2e assertion. It says that, and states plainly that a caller
needing to know who owns a node in order to dial it wants Owner.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Ettore Di Giacinto
|
5e2938ebf0 |
feat(cluster): resolve tunnel ownership against a live owner
OwnerRow is a bare row read of node_connections. A connection row outlives the replica that wrote it: a replica that dies stops heartbeating, but its rows survive until a peer's sweep removes them, which is up to InstanceLiveness plus one InstanceHeartbeat later. For that whole window the table names a process that is gone. The next component phase 2 builds is the relaying dialer, and a dialer reading OwnerRow would relay into a corpse for roughly 35 seconds after every replica death, then report the worker as unreachable when it is in fact absent, which is the distinction the phase 1 end-to-end specs pinned. Owner is the resolving read: one statement joining instances, returning ErrNoConnection when the row is missing OR its owner is not live. Both cases are one answer on purpose, since both mean no replica here holds this tunnel; they differ only in which sweep has run. It is one statement, not a row read followed by an instance lookup, because between two statements the owner can die and the caller would act on an owner the second read would have rejected. OwnerRow stays, unjoined, for readers that need the row itself, and a spec holds the two apart: with an aged-out owner, OwnerRow still names it and Owner refuses, so neither can quietly become the other. The liveness predicate is now one string, instanceIsLive, shared by Live and by Owner's join. Two spellings of one fact drift, and this drift would show as a relay to a replica one query calls dead and another calls alive. It is table-qualified so it is unambiguous inside the join, and the cutoff stays on the database clock, so replica clock skew cannot widen or narrow the window. Both mutations were run. Dropping the liveness predicate from the join fails 3 specs, the aged-owner one among them. Replacing the database clock with a Go-side time.Now() fails 1: the aged-owner specs still pass, because the two clocks agree on one host, and only the recorded-SQL spec sees the literal timestamp. That is why that spec exists. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |