2 Commits

Include renames

Author	SHA1	Message	Date
Ettore Di Giacinto	ed461cd86c	test(distributed): prove phase 3 end to end, under real processes ... Five cluster specs that run the binaries an operator runs, plus the repair of eighteen specs phase 2 left red. The eighteen were router_tracking and full_flow, failing since 1cf847f29 on "reported backend installed but named no address for the process". Two contracts had changed under them: an install reply that names no worker-local address is refused rather than substituted, and a frontend with no worker dialer reaches no backend at all. Nobody noticed for a phase because phase 2 verified with --label-filter='Cluster', which excludes both suites. ServeBackendLifecycle and tunnelBackendClients state both facts once for every spec. The transport double is the part that matters. It translates a refused connect into cluster.ErrStreamTargetUnavailable, which is what a real worker answers when its backend process has died and what IsWorkerAnswer lets a reap guard act on. A bare ECONNREFUSED reaches those guards as "no route" and reaps nothing, so a double returning the raw syscall error could never fail the way production fails; putting it back reddens the stale-record spec and nothing else. The new specs cover: a backend worker with no bus URL in its /proc environ registering, being scheduled onto and serving inference; a backend install and a backend listing driven through the replica that does NOT own the worker, with the owner read through the production Owner query and re-read after; that install's progress proven to arrive before its terminal reply, made deterministic by a gallery server that holds the worker's fetch open so a reply cannot exist yet; a worker whose tunnel is genuinely gone, waited for rather than assumed, losing nothing inside the reconnect grace and re-homing after; a heartbeating worker with a permanently dead tunnel losing its healthy status while an agent worker in the same cluster keeps it; and the suite's negative control, where a control RPC to a tunnel-less worker fails naming the missing route, reaps nothing, and succeeds the moment the tunnel returns. Every scenario was attacked. The churn one was WRONG on the first attempt and only the attack found it: its hold window sat entirely inside cluster.InstanceLiveness, so a killed replica still read as a live owner throughout, presence was "connected", and the spec passed with the reconnect grace set to a nanosecond. It now blocks the tunnel before the kill and waits for the ownership row to actually empty. Attacks that redden the rest: posting at the owner, writing the install reply before the work, collapsing PresenceReconnecting into PresenceGone, removing the non-backend node-type guard, and not blocking the tunnel. Agent workers turn out to be protected twice over; no single mutation reaches them. Harness: Options.AgentWorkers and Options.ReconnectGrace, WorkerEnviron (read from /proc, because Cmd.Env is the harness agreeing with itself), NatsURL, FrontendBackendsDir, AgentWorkerName, PostJSON, and a node String() so a failing roster assertion is readable instead of several hundred bytes rendered as numbers. Budget: 20 specs at 787 to 808 seconds over three runs, up from phase 2's 591 to 612. --timeout goes to 30m so a loaded runner reports a cause rather than a spec name. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-09-20 03:05:34 +00:00
Ettore Di Giacinto	615eab5494	feat(distributed): call the worker's control routes instead of the bus ... The ten backend and model lifecycle verbs stop being NATS requests and become HTTP calls on the worker's own control routes, reached through that worker's tunnel on the `http` stream tag that already carries file staging. Nine subject builders and the per-op install-progress subject are deleted with their entries in the worker's NATS permissions; the request and reply DTOs are untouched, so a body on the wire is byte for byte what the subject carried. This closes the merge gate Task 3 left open, which was worse than lost commands. Once the worker stopped subscribing, PingNode was still asking nodes.<id>.backend.list and nodes.<id>.models.running, so EVERY healthy worker answered no-responders, nodeAnswersOnBus read it as absence and pickReachableNode demoted it on the scheduling path. PingNode is a control RPC now, and no control RPC can produce ErrNoResponders, which is the only error that exclusion acts on. Two specs drive pickReachableNode against a real adapter and a worker answering over its control plane, which is the only arrangement that can see the difference: the router's own double never touches a transport and stayed green for the whole window the defect was live. How a control RPC FAILS is the whole of this change, so it is decided in ONE function reading ONE table. A worker's answer passes through unwrapped, so cluster.IsWorkerAnswer still sees it and a reap guard may act on it; everything else is wrapped in ErrWorkerUnroutable so nothing can. There is no third branch, because a third branch is how the eight collapses on this branch happened: each was a site that decided for itself which errors were evidence. A 404 under the prefix is its own sentinel, because it is the worker stating a deployment fact about ITSELF rather than a verdict about a backend, and only the legacy upgrade fallback may act on it. The caller's budget is checked FIRST. A timeout is not a verdict: a refusal arriving in the instant a deadline expires would otherwise be reported as the worker's non-transient answer, which reaps a row, and nothing orders the two timers. A 5xx and an undecodable body are transport failures, not answers. An empty ModelsRunningReply means "this worker is running nothing", which the reconciler acts on, so it must never be manufactured from a body that would not parse. A stream that ends before its reply line is the same rule one layer up: a tunnel dying mid-install is not the worker saying the install failed. backend.stop is split by node type rather than moved. Agent workers hold no tunnel, so they have no control plane to serve, and they still subscribe to nodes.<id>.backend.stop to drop cached MCP sessions; that subject and its agent permission both survive. It is the honest intermediate state until agent workers hold tunnels too. A failed control RPC no longer demotes a node anywhere. ErrNoResponders meant "not on the bus"; a control failure means "this frontend could not route to it", which is equally what a healthy worker re-homing its tunnel between replicas produces. Absence is a fact read from the database, and the scheduler starts reading it in a later task. The rolling-update fallback re-fires a DESTRUCTIVE force-reinstall, so it runs only on the worker's own 404. Its negative direction was pinned at the admin call site and unpinned at the reconciler's, where widening the condition to any error left all 676 specs green: a background drain nobody is watching would then force-reinstall every queued backend the moment a replica lost its tunnels. Three specs cover it, arranged so the force install IS reachable in the negative case and a fallback that fired would show as a call and a drained row. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-09-20 03:05:34 +00:00