mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-22 14:14:54 -04:00
test/distributed-e2e-ci
32
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ettore Di Giacinto
|
95cb3cfa41 |
fix(distributed): follow current backend protocol
Master added Animate3D, negative_prompt, and context_size after this branch diverged. The old suite did not exercise those paths, and Kokoros no longer implemented the generated service trait. Extend binary conformance across the tunnel owner and peer relay. Allow long development versions so rebased binaries can register in PostgreSQL. Clear the security findings introduced by the branch's new code. Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] |
||
|
Ettore Di Giacinto
|
9826547c42 |
fix(security): clear the gosec findings this branch introduced
Tighten the permissions that can be tightened and justify the ones that cannot. The e2e harness creates its directories 0750 and writes its model config and process logs 0600; the copied binary keeps 0755 because the suite executes it. Everything else is a gosec rule firing on code that is doing what it is supposed to: jitter drawn from math/rand for a reconnect backoff, a fixed credential for a throwaway cluster, and subprocesses launched from the binary the suite just built. Each carries a #nosec naming the rule and the reason, in the form the tree already uses. Scanned before and after: 13 findings in the packages this branch added, now 0. The one that remains in core/services/worker is byte-identical on master and is not this branch's to silence. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
7d04334633 |
test(distributed): correct cluster suite docs
Record the authoritative Cluster label count and describe the restart helper's bounded graceful shutdown path accurately. Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] |
||
|
Ettore Di Giacinto
|
ff0eddf037 |
test(distributed): reap binary backend children
Exercise normal cluster shutdown through the real binary signal handlers and retain the loaded mock backend PID so the conformance scenario proves teardown leaves no orphan. Keep SIGKILL as a bounded fallback and leave the explicit crash primitives unchanged. Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] |
||
|
Ettore Di Giacinto
|
ceacb22adb |
fix(distributed): prove agent worker credentials
Keep pending agent workers alive with authenticated heartbeats while preserving approval as the tunnel and execution boundary. Exercise the real binary credential handoff through an authenticated inference and verify that credential remains non-admin. Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] |
||
|
Ettore Di Giacinto
|
ce41c56251 |
test(distributed): verify binary machine authentication
Exercise WebUI sessions, fail-closed registration, per-node tunnel credentials, manual approval, heartbeats, and scoped agent credentials through real frontend and worker processes. Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] |
||
|
Ettore Di Giacinto
|
8dbe99dfad |
fix(distributed): harden staged resource ownership
Bind cleanup to storage namespaces and generation-scoped quantization state. Validate backend outputs, prove inputs reach worker staging, and make failed-start cleanup retryable. Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] |
||
|
Ettore Di Giacinto
|
ce993065c4 |
test(distributed): complete binary backend conformance
Exercise the remaining public backend routes through real frontend and worker binaries, and keep fixture staging limits scoped to this conformance deployment. Fix inpainting artifacts so returned generated-image URLs resolve through the configured static mount. Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] |
||
|
Ettore Di Giacinto
|
1e877e9f6c |
test(distributed): cover binary backend routes
Run one real frontend and worker against deterministic fixtures. The owner tunnel now covers every public backend route in this task. Store requests exposed a missing config revision. Forward the persisted revision so distributed routing accepts store models. The harness pins staging capacity because quota-backed CI filesystems can report misleading host totals. Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] |
||
|
Ettore Di Giacinto
|
0becab549e |
test(distributed): pin cluster health timings
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] |
||
|
Ettore Di Giacinto
|
f1f026c324 |
fix(worker): check a gRPC port is free before handing it out
The backend port allocator allocated from its own bookkeeping alone. That bookkeeping records what this worker did with a port, and the collision it cannot see is with something this worker never did: the default base port is 50051, inside Linux's default ephemeral range of 32768 to 60999, so the kernel hands ports in this range to outbound connections and to anything that binds port 0. A backend handed one of those dies on bind, and the frontend sees a backend that will not start. Every candidate is now probed by binding the exact address the backend will listen on, in all four allocation branches: the key's own port, the free pool, a grown port and a stolen one. Probing the free pool matters as much as probing a grown port, because a port this worker released is exactly as available to the kernel as one it never used. A candidate that fails the probe is quarantined rather than blacklisted, since whatever holds it is usually an ephemeral connection that gives it back, and its affinity claim is dropped so an unbindable port does not stay reserved for the key that last held it. Exhaustion now says how many candidates were skipped, which is what tells an operator "something else is in my range" from "my range is too narrow". This does not remove the race and cannot: between the probe and the child's bind the kernel can still give the port away. It removes the far larger window in which the allocator hands out a port the kernel gave away minutes ago, which was the whole of the observed one-in-three harness flake. The e2e harness comment that recorded the missing check is corrected, and the docs say how to move the range out of the ephemeral one entirely. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
d9d83154b8 |
test(distributed): prove the fan-out carrier between two real replicas
Removing the broker left one thing carrying every broadcast family in the product: PostgreSQL LISTEN/NOTIFY, in core/services/pgbus. It is covered thoroughly in process by test-e2e-distributed, and it was covered nowhere at all by real binaries: grepping the six Cluster spec files for pgbus, bus_messages, LISTEN and NOTIFY returned zero hits. Registration, model staging over the tunnel and inference through both the owner and the relay paths were already proven by real processes; the carrier that now carries everything else was not, so a deployment whose replicas each published to themselves and heard nobody would have left every suite green. Two specs, both on two frontends and no workers against one PostgreSQL, publishing at frontend 0 and reading at frontend 1. 1. A gallery operation admitted at one replica, read out of the other, with the queued state observed before the terminal one. 2. A broadcast of about 9.3 kilobytes, which PostgreSQL refuses as a notification payload, making the round trip byte for byte through the bus_messages spill table. The family is a gallery operation for one property nothing else on this carrier has: the answer a peer gives is held in memory ALONE. GET /models/jobs/<id> reads galleryop's statuses map, which on a peer is filled by the gallery.*.progress subscriber and by nothing else, because the only other filler, Hydrate, runs once at startup and every operation here is created long afterwards. Every other family has a durable table behind it that a peer would converge through anyway, and a spec on one of those cannot separate "the broadcast arrived" from "the row was read". That is then made checkable rather than argued. The gallery_operations row is written when the gallery worker DEQUEUES an operation, so an operation still waiting in the queue has NO row, and both specs assert zero rows while the peer is already answering with the operation's own bytes. Both also read the instances table and require the reading replica to be a different live instance from the publishing one, so "the other replica" cannot decay into a spelling of "this replica". Holding the queue is what cluster.Options.Galleries is for. The gallery worker runs one operation at a time on an unbuffered channel, so an install parked inside a gated index fetch parks everything behind it; without that the admission broadcast and the terminal one are separated by two database round trips and no HTTP poller could see between them. The option also turns the startup estimate warmer off, because a second fetcher filling the process-wide index cache would leave the operation never blocking and the spec passing on an ordering nothing enforced. The spill spec is written against a failure this branch has shipped three times: a size-limit spec that cannot fail. The oversized body is an ordinary element name that the real consumer decodes and surfaces, so it is not a body the decoder would have refused at any size. The size is ABSOLUTE at 9000 bytes rather than derived from the cap, and a one-byte control operation in the same run is required to leave no spill row, so moving the 8000-byte cap in either direction reddens the spec. pgbus.FitsInline, which shares its encoder and its comparison with Publish, is asked about both payloads and must answer differently. The spilled row is then decoded and its element name compared byte for byte against what frontend 1 answers. The terminal assertion in spec 1 does not re-check the element name: a terminal status does not carry one, because updateError in galleryop.Start builds a fresh OpStatus holding only the error. It asserts the two fields that status does carry, in the relation that one place writes them. Attacks run, each alone, each reverted, each behaving as predicted. Neutering the pg_notify in pgbus.Publish so every replica knows only what it did itself reddens both specs at frontend 1, which answers 500 for an operation it was never told about; the bus_messages row assertion still passes under it, which is right, since the row is written before the notification. Releasing the queue gate reddens spec 1 at the gallery_operations count, because the operation is dequeued and the row appears. Shrinking the oversized name to 100 bytes reddens spec 2 at FitsInline; inverting that guard so the run reaches the row check reddens it there instead, with no bus_messages row written, which is what makes the row a statement about size. test-e2e-cluster is 26 specs in 933.8 seconds of Ginkgo time, 15m37s wall. The two additions cost 7.0 seconds together, 5.0s and 2.0s: they start no workers, so they pay for no registration, and what they wait on is a broadcast rather than a threshold. test-e2e-distributed is unchanged at 223 plus 8 specs, 130.6 seconds. The budget comment and .agents/building-and-testing.md move from 24 specs at 897 to 907 seconds to 26 at 933.8. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
885911697f |
test(distributed): prove the busless cluster on two frontends and two workers
Tasks 1 to 17 are proven by unit and integration specs and by two e2e passes taken mid-flight. This is the pass that boots the real binaries with every carrier in place and none of the old one, and it does so on the topology the feature was built for rather than on the one-worker shape the rest of the cluster suite uses. Two frontends and two workers is the configuration that matters. With each worker's tunnel landing on a different replica, the owner path and the relay path are live at the same instant against one roster, one scheduler and one health monitor, so a routing mistake has somewhere to show up instead of hiding. It is also the only shape in which "killing a replica re-homes only ITS worker" can be stated at all. Three scenarios, all 2x2: 1. Both workers served from both replicas. No broker as a property of the ARTIFACT (debug/buildinfo reports no github.com/nats-io module, with the module count asserted non-zero so a stripped binary cannot pass vacuously), no broker in either worker's live /proc environment, and no advertised address on either worker. One completion over the owner path and one over the relay, plus the mirror image through the other replica, plus four control-plane listings covering both paths for both workers. 2. The replica owning worker 0's tunnel is killed with that tunnel blocked. Leg 1 asserts nothing and only waits for the killed instance to leave the live set, because before that it still reads as a live owner and the scenario is not yet about absence. Leg 2 then holds a window inside the reconnect grace requiring that nothing acted on the absence. Leg 3 requires the re-home and inference again. Worker 1 keeps serving throughout. 3. The suite-wide negative control. Both tunnel dials refused while registration and heartbeats flow, both workers refused at both replicas naming the routing fact and not a departure, nothing reaped and both heartbeats fresh. Then ONE tunnel is restored and exactly one worker recovers while the other stays refused. Which worker served is read back from node_models rather than assumed: the two models are pinned to one worker each through PUT /api/nodes/:id/labels and POST /api/nodes/scheduling, and every assertion requires the model to be on the expected node AND absent from the other. That the relay hit a non-owner is read from the production Owner query before the request and re-read after it. Attacks run, each alone, each reverted, each behaving as predicted: hand a worker a broker URL reddens scenario 1's environment leg; point the module check at gorm.io reddens its artifact leg; start one worker instead of two reddens all three at the topology guard; delete the relay in WorkerDialer.Dial reddens scenario 1 on exactly the request sent to the non-owner while 2 and 3 stay green; a one-nanosecond reconnect grace reddens scenario 2's leg 2 on the demotion while 1 and 3 stay green; lifting both blocks at scenario 3's differential reddens its "still unreachable" half. The brief's "restore the NatsURL validation" attack cannot be applied: DistributedConfig has no such field left to validate. Label-orphan arithmetic, counting non-skipped It nodes from --dry-run: all 256, dist 231, cluster 24, vllm 1, and 231 + 24 + 1 = 256, so no spec is orphaned by the label filters. Three test-e2e-cluster runs: 897.0s, 897.8s and 906.6s of Ginkgo time, 24 specs, 15 minutes wall. The only failure across the three was a pre-existing spec dying at cluster.Start with frontend-1 exiting status 2, which passed in the other two and is reported as a port-allocation flake rather than a regression. test-e2e-distributed is 223 plus 8 specs in 131.7s. The budget comment and .agents/building-and-testing.md move from 21 specs at 800 to 830 seconds to 24 specs at 897 to 907. The harness gains ProcessEnviron, which reads /proc for any of the three process families; WorkerEnviron and FrontendEnviron become wrappers rather than being deleted, so the specs that call them are not re-aimed for a rename. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] |
||
|
Ettore Di Giacinto
|
016686a3db |
chore(distributed): take the nats-io modules out of the build
Distributed mode has not dialled a message broker since the control plane moved onto the workers' own outward tunnels and every fan-out family moved onto PostgreSQL LISTEN/NOTIFY. What was left was the dependency itself, and the code that existed only to feed it. Dropped from go.mod: nats-io/jwt/v2, nats-io/nats.go, nats-io/nkeys, nats-io/nuid and testcontainers-go/modules/nats, along with the fourteen indirect requires that only the NATS testcontainer pulled in. go.sum carries no nats line either, so the removal is not the partial kind where the require goes and the checksum stays. Deleted with them: pkg/natsauth in full, the broker client's remaining options and TLS files, the per-node JWT minting on both the register and the approve path, and the natsauth.Config parameter threaded through the node routes. The credential manager is renamed and stripped rather than deleted, because it still holds the tunnel token that every re-registration rotates. The bus flags stay accepted and ignored, and are now hidden, on every command that had them, so an existing unit file, compose file or Helm values file still starts on the day of the upgrade. What is not kept is the validation that REQUIRED one: a distributed frontend started with no bus URL is no longer fatal. The TLS paths lose type:"existingfile" deliberately, so a certificate deleted along with the broker cannot fail a startup. One operator-visible behaviour change: --nats-require-auth no longer makes an agent worker wait through admin approval. Ask for that wait with --distributed-require-auth, which already implied it. It is documented in the migration section and pinned from both sides. A deployment now needs PostgreSQL and the frontends' own HTTP listener, and nothing else. coverage-baseline.txt moves from 54.2 to 62.0. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
0eab3eb7f1 |
fix(distributed): read a departed agent tunnel as the routing fact it is
Task 4 gave agent workers tunnels and deliberately left the NodeType skip in HealthMonitor.tunnelDeparted, with a spec asserting that an agent node whose presence reader answers PresenceGone is NOT marked unhealthy. That spec was scaffolding. It was true while an agent worker took its jobs and its verbs over the message bus: a departure row for one said nothing about whether it could work, and an early bug in the new tunnel client could otherwise have demoted a fleet of healthy agent workers. There is no bus. An agent worker is reachable through its tunnel and through nothing else, so a departed agent tunnel means exactly what a departed backend tunnel means: no live replica holds it, the departure has outlived the reconnect grace, and that is a routing fact the scheduler and a reaper may act on. The skip would now hide the only symptom an unreachable agent worker has. This is the deliberate removal Task 4's M6 predicted, and task-4-report.md is where that mutation already stands recorded red against the spec this commit deletes. The skip existed at ONE site. router_liveness.go has none: its candidates come from queries that already filter node_type = 'backend'. The two skips in managers_distributed.go stay, because an agent worker still runs no backend processes, so it has no backend to list and no backend op to apply. Two node types can depart now, which is why the second half exists. Before this, one type could depart and every per-node cache a departure left stale was dropped from wherever its owner happened to notice, so a reader could not tell which caches a demotion invalidated by reading the demotion path. Departure gets ONE notification point. DepartureNotifier is edge triggered, because the monitor runs on a ticker and a departed node stays departed; its subscribers are NAMED, because what has to be caught is a forgotten cache and a count can say only that one of four is missing; and NewHealthMonitor takes it as a required positional argument, so a caller that does not pass one fails to compile. Four caches subscribe: prefix-cache affinity in every model, probe freshness at every address, in-flight staging operations, and the per-node breakdown of every open gallery operation. The prefix-cache one is registered only when prefix-cache routing is enabled, so --distributed-prefix-cache=false stays a true no-op. The notification carries the node's name as well as its id, because the staging tracker keys on the name and the other two key on the id, and a subscriber should not have to read the registry from inside an eviction hook. A departure notification is an act on absence, so it fires only on the routing fact. A tunnel lost inside the grace, a worker that never dialled, a presence query that failed and a stale heartbeat all announce nothing, asserted per node type. The stale-heartbeat branch is excluded on purpose: it already marks the node offline, which deletes its rows and runs the registry's replica-removed hooks, so firing there too would double-evict and make the notification mean two different things at its subscribers. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
12571d22d9 |
feat(distributed): give agent workers a tunnel of their own
Phase 2 gated agent nodes out of tunnel credentials at the mint site. That was right while nothing dialled into an agent worker: a credential would have replaced nothing, and the gate was structural rather than a second check that could drift. It is wrong now that the frontend needs to reach an agent worker by RPC. attachTunnelToken mints for backend and agent nodes and CLEARS for anything else, through one tunnelEligible predicate rather than two conditions that can be widened separately. ConnectHandler still never reads NodeType, so an empty hash is still what refuses an ineligible node. An agent worker now starts a loopback control server behind the same bearer check a backend worker uses, and holds one tunnel whose only stream tag is http: it runs no backend processes, so the grpc tag has nothing to route to and is not offered. Its MCP tool, MCP discovery and backend.stop verbs are served from ONE implementation reached by both the bus and the tunnel, so a frontend cannot get different bytes depending on which carrier delivered. The tunnel is an ADDITION. --nats-url is still required, and agent jobs, MCP execution, MCP CI jobs and nodes.<id>.backend.stop all still travel on the bus. Absence semantics are unchanged. An agent node now has a real node_connections row whose departure ages past the grace, so the node type check in HealthMonitor.tunnelDeparted stopped being an optimisation and became the rule; its comment says so, and the spec that pins it is shown red under a mutation that deletes the check. The scheduler needed no change: every placement query already filters node_type = backend, so an agent node never reaches nodeMayTakeWork. Shared rules moved to one site each. The request bounds, the POST-only check and the unknown-path 404 live in workerctl and are called by both worker packages; the bearer check that guards every extra route is one function in core/services/nodes used by both server constructors. workerctl.AllPaths splits into BackendPaths and AgentPaths, with AllPaths as their deduped union, because a backend worker does not mount the agent verbs and asserting otherwise would fail a correct worker. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
ed461cd86c |
test(distributed): prove phase 3 end to end, under real processes
Five cluster specs that run the binaries an operator runs, plus the repair
of eighteen specs phase 2 left red.
The eighteen were router_tracking and full_flow, failing since
|
||
|
Ettore Di Giacinto
|
e839984cf5 |
feat(distributed): take the backend worker off NATS entirely
A local-ai worker no longer opens a bus connection. connectNATS and its
spec are gone; Run registers once, starts its tunnel, arms /readyz on that
tunnel, and heartbeats. The worker's bus credential flags (--nats-jwt,
--nats-user-seed, --nats-require-auth, the three TLS flags) and
Config.NatsAuthRequired go with it. --nats-url stays, accepted and
ignored, so an existing worker command line still parses.
/readyz was the thing most likely to wedge a tunnel-only worker: it
required a live NATS link, so a worker with no bus would have reported
itself unready forever. nodes.NATSReadiness becomes nodes.TunnelReadiness
over a local interface{ Connected() bool }, and worker.Tunnel gains
Connected(), backed by a mutex-guarded session field the loop publishes
and clears. A closed-but-not-yet-cleared session reads as disconnected:
the loop waits for every in-flight stream before it clears the field, and
the probe must answer not-ready through that wait.
The heartbeat gate is DELETED rather than re-pointed at the tunnel. The
heartbeat is the worker's own answer that its process is alive; whether
the frontend can reach it is a separate fact the frontend already holds
and ages against LOCALAI_WORKER_RECONNECT_GRACE. Withholding the
heartbeat would report an unreachable worker as an absent one on the one
path with no grace, where the health monitor marks it offline and its
pending backend ops are deleted behind it. heartbeatLoop is given no view
of the tunnel, so a gate cannot be added back without changing its
signature.
Removing the NATS credential manager from this path also removes a defect
it carried: its refresh loop re-registered on a timer to renew a JWT, and
Register CLEARS a node's NodeModel rows. Any backend worker running on
frontend-minted credentials had its replica rows deleted roughly every
18 hours.
Of core/cli/workerregistry, everything survives. The manager is still
used in full by core/cli/agent_worker.go, which still needs NATS: Acquire,
Provider, RefreshLoop, HasCredentials and TunnelToken are all untouched.
The backend worker simply calls RegisterFullWithRetry directly now.
WorkerPermissions is documented as serving agent nodes, and its non-agent
branch narrowed to _INBOX.> on both sides. It is NOT deleted: NATS reads
an empty allow list as no restriction, so returning nil would upgrade
every JWT the frontend still mints for a backend node from its own inbox
to the whole account.
Agent workers keep the bus everywhere: their CLI flags, their
subscriptions, the agent branch of WorkerPermissions, and the compose
service with its LOCALAI_NATS_URL and depends_on: nats.
Also corrected two flags the Nodes page advertised that do not exist
(--distributed-nats, --distributed-db), and a log line plus several
comments that still named a bus the code no longer touches.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Ettore Di Giacinto
|
cf799566ed |
test(distributed): prove the worker tunnel end to end, under real inference
Everything this phase built was proven by unit and integration specs. This is the first run of it against the real binaries: a frontend replica per process, a worker that binds nothing routable, real inference over the result. Four scenarios, each with the question "what would make this pass if the tunnel were doing nothing" answered rather than left open. A worker with no advertised address is reached through its tunnel. The roster is asserted to report it advertising nothing, so there is no address a frontend could have dialled instead, and node_connections is asserted to name the replica that serves the request. A request landing on the replica that does NOT own the worker is relayed to the one that does. With N replicas behind round robin that is (N-1)/N of production traffic, so it gets the FIRST request for its model: the backend install, the file staging on the http tag, and the gRPC load and predict all cross the relay. Which replica owns the tunnel is read from the ownership table through the production Owner query and mapped to a frontend index through the address the harness pins per replica; the non-owner is derived from that reading and asserted to be a non-owner immediately before the request, rather than assumed from the harness default. Sending the same request to the owner reddens it. Killing the owning replica re-homes the worker onto the survivor. The worker dials a balancer rather than a replica, because LOCALAI_REGISTER_TO is resolved once at boot and is the tunnel endpoint as well as the registration one: aimed at a single replica, a worker has nowhere to reconnect to when that replica dies, and the re-home cannot happen at all. Removing the kill reddens it. And the negative control for the whole suite, which is why the other three mean anything. Frontend and worker share a host here, so every backend port the frontend names in a stream target is one it could have dialled directly; if it did, the first three would pass with the tunnel inert. LOCALAI_WORKER_TUNNEL is no longer usable for this, because it is a fatal startup error and a worker that never started says nothing about a worker reachable some other way. The balancer answers the tunnel connect path itself instead, leaving a worker that registers, heartbeats, reports healthy and holds no tunnel. It is asserted to have dialled and been refused, asserted to be held by nobody, and then asserted unreachable with the refusal naming the missing route. Then the block is lifted, nothing else changes, and the same request succeeds: that is what attributes the refusal to the tunnel rather than to any of the ordinary reasons an e2e inference fails. The fifth spec measures the head-of-line blocking this phase deferred three times. 128 MiB crosses the session while a warm model is probed back to back, direct and relayed. Median latency is unchanged, the worst probe is about 3x the baseline median and about a seventeenth of the transfer window, and the transfer runs at 415-490 MB/s direct and 222-268 MB/s relayed. A session that head-of-line blocked would park a probe for the length of the window. Leave the yamux windows untuned; and note this is loopback, so it says the multiplexing does not serialise and says nothing about a link with a bandwidth-delay product. The load spec is measured against a control that the first version did not have. It passed with the bulk artifact cut to 4 KiB, because the window it read probes against was mostly cold-load overhead: it would have reported a clean bill on a session carrying no large message. The same cold load now runs twice, once empty and once bulk, and the difference between the windows is asserted to be real before any latency is read from it. Two defects on the base commit came out of this. cluster_peerlink_test.go has been red since the relay landed, deterministically, in isolation and in the suite. It asserted that an accepted peer stream is refused at once, on the premise that phase 1 installs no relay. The relay correctly waits fifteen seconds for a frame naming the worker, and the spec's budget was five. It now writes a relay request for a node no replica holds and asserts the refusal is ErrNotOwner and specifically not ErrNoConnection, which is a stronger spec than the one it replaces and the only thing in the e2e suite that exercises the relay's refusal path. The harness handed a worker's own HTTP port to a backend process. It took two ports from freeport and used one as the gRPC base and the other for the file transfer server; freeport returns adjacent ports often, and the backend allocator hands out base, base+1, base+2, so the second backend started on a worker was regularly given the HTTP server's port and died with EADDRINUSE. No spec had started two backends on one worker before, so it had never fired; the load spec starts five and it failed about one run in three. Each worker now reserves a contiguous bind-probed block laid out the way production lays it out, below the kernel's ephemeral range, with LOCALAI_GRPC_MAX_PORT bounding the allocator to it. The underlying production defect is not fixed here and is recorded in the report: allocatePort never checks that a port is free, and its default range overlaps the ephemeral range on every Linux box. Constraint 6, whether distributed mode should now refuse to start without an advertised address, is DEFERRED, and the comment and the docs that described the cost were understating it. A replica with no advertised address writes no instances row, and Owner joins a connection against a live instance, so a worker whose tunnel lands there is unroutable from every OTHER replica while being registered and healthy. Refusing to start would still be wrong, because the deployments it would break are single-host ones with no peers to be unreachable by, and telling those apart at startup is a design with its own specs. Both places now say what actually happens. Suite wall clock 592s for 15 specs, up from 502s for 10 of which 2 were red. The CI budget of 20 minutes does not move. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
e7294b0984 |
feat(distributed): stop workers listening, and stop them advertising
A worker now opens no listener on a routable interface and states no endpoint at registration. Backend processes and the file-transfer server bind loopback, and the frontend reaches both through the tunnel the worker dials. The bind address is built from loopbackHost, the same constant the tunnel's grpc tag dials, so "the worker binds where its tunnel dials" is one fact in one place rather than two literals that can drift. All three advertisement sites are closed, not one: the registration body, RegisterNodeRequest, and the per-backend address in the install reply. That third one was hiding a live bug. stopModelExact refuses a stop whose ExpectedAddress does not match what the worker recorded for the process. The worker recorded 127.0.0.1:port; handleBackendInstall reported advertiseHost:port; the router stored the reported one and sent it straight back. On any worker whose advertise host was not 127.0.0.1, every acknowledged model stop failed with an address mismatch. Nothing caught it because the e2e harness set LOCALAI_ADVERTISE_ADDR=127.0.0.1, which made the rewrite a no-op. Removing the rewrite makes the two strings the same by construction. The brief was wrong about two of the four functions it called dead. effectiveBasePort is the base of the backend port allocator and resolveHTTPAddr is the file server's bind address; deleting them would have deleted the port allocator and the file server. Only the two advertise* helpers were dead, and addr_test.go is rewritten rather than deleted, because the port arithmetic it pinned still needs pinning. NodeModel.Address survives with a narrowed meaning and is renamed WorkerLocalAddress, along with the install reply field that feeds it. The frontend still has to say WHICH backend process on a worker it means, and the port in this string is how it says it: it travels as a stream target and the worker dials its own loopback. The gorm column and the json key stay "address", so neither a migration nor an API break rides along. Every fall-back to the node's address is gone. installBackendOnNode now errors when a worker reports success without naming one, because substituting the now-always-empty node address would name an empty target, and the worker refuses that as an invalid stream, which is classified as the worker answering about its backend. That is the "a present worker reads as something it is not" class this phase forbids. DistributedModelStore.Range had the same shape and was already wrong: it built each remote model's client from the node's base gRPC port, never the port a backend process listens on, so Free and Status went to the wrong place. It uses the replica's address now. BackendNode.Address and HTTPAddress are kept but made provably inert: no writer, no reader that acts on them, and Register force-clears both on re-registration so an upgraded worker's stale advertisement does not outlive its own upgrade in the API and the Nodes page. Dropping the columns is a ~90-site edit across the specs, the e2e suite, the MCP dto and the UI; it is recorded as a follow-up rather than folded in here. A persistent tunnel 401 still does not trigger re-registration, and now for a reason rather than a deferral. Register CLEARS the node's replica rows, so re-registering on a 401 would delete a live worker's rows on every retry, and under the name collision that causes the 401 the two workers would take turns doing it forever: a credential failure causing model reclamation. It also cannot fix the named cause, since a collision is indistinguishable from a restart. The 401 log now names both causes and says nothing can reach this worker, which is true only now that it has no listener. The container healthcheck did not break the way the brief expected, since the listener still exists on loopback and the probe runs inside the container. It did have a real #10987 defect that this change makes the common case: it read LOCALAI_SERVE_ADDR only, while effectiveBasePort reads LOCALAI_ADDR first, so a worker on a non-default base port was probed on 50050 and reported unhealthy while working. It follows the same precedence now. Docs, the compose file and the e2e harness are updated in step: no inbound rule or published port is needed for a worker, the two advertise variables are gone, the remaining address variables are read for their port only, the firewall-the-file-transfer-port warning is narrowed to the LOCALAI_HTTP_ADDR opt-out, and the upgrade-order note no longer claims the worker still listens. The Nodes page showed node.address, which is now always blank, so it shows the node id instead. Eight mutations, all red on a named spec, including reverting the loopback bind, re-adding the address to the registration body, restoring both node-address fall-backs, dropping the force-clear, storing the endpoint's address again, and un-fixing the healthcheck. One of them caught a defect in a spec I had just written: it asserted 200 where the endpoint returns 201, which went unnoticed because core/http/endpoints/localai is not on the task's verify list. It is run here. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
3b19575f81 |
feat(cluster): give phase 1 a call site, and prove it against real replicas
Tasks 1 to 5 built an instances table, a splice, both halves of a peer link and an epoch fence, and nothing in the tree called any of it: no replica registered, no route was mounted, no sweeper ran. Proving phase 1 end to end therefore had to start by wiring it. A frontend in distributed mode now publishes the address its peers dial, heartbeats it, and sweeps replicas that stopped answering along with the connection rows they owned, in one pass so the two can never disagree about who is alive. It serves the peer link and owns the sessions peers dial in, refusing streams on them until phase 2 installs a relay: a session nobody accepts on does not fail a peer's Open, it hangs it. The address is the one peers use, not the one the process binds, and it is derived from the route to PostgreSQL. That derivation only holds while the database is remote, so LOCALAI_DISTRIBUTED_ADVERTISE_ADDR sets it explicitly and a replica that can determine neither warns and keeps serving rather than failing to start. Three e2e scenarios run against real local-ai processes, real PostgreSQL and real dials: replicas publish addresses that can actually be connected to; a sibling opens a stream over the peer link and is refused without the cluster token; and a killed replica is reported unreachable, never absent, loses the claim it held, and takes no worker with it. Each was verified by mutation: eight injected defects, each failing the scenario that claims to catch it. Also moves RegisterClusterRoutes to core/http/routes beside every other registrar, folds AutoMigrate and the epoch sequence into one cluster.Migrate, and turns the peer route's auth-coverage spec into a real assertion: it drives the request through the actual auth middleware instead of comparing two string constants, which the old spec would have passed even with the exemption deleted. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
cb3884fdc1 |
docs(e2e): correct three claims in the distributed e2e comments
The closure note in cluster/failure.go quoted a Gomega error that Gomega does not emit. Describe the argument-count failure and the Eventually().WithArguments() hint instead, so nobody greps for a string that never appears. The advisory-lock note in cluster_failover_test.go called the wedge window unbounded. A SIGKILLed local child closes its socket at once, the Postgres backend reads EOF and is reaped in milliseconds, so the mechanism bounds the window tightly. Say bounded, and keep the low probability but real framing, which was right. The workflow comment attributed HealthCheckInterval to core/services/nodes/health.go. It is declared in core/config/distributed_config.go:64; health.go only carries the ticker on the unexported checkInterval. Point a debugger at the right file. Comments only, no behaviour change. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
065102c1b6 |
test(distributed): correct the claims the e2e comments make
Review of the whole branch found five comments that would send a reader to the wrong place, plus three smaller inaccuracies. Nothing here changes behaviour. The KNOWN RACE note on both backend-log WebSocket handlers said the fix needs an atomic snapshot-plus-subscribe "under the store lock". It does not: BackendLogStore.mu guards only the buffers map, and AppendLine enqueues and fans out under the per-buffer buf.mu. Whoever took the store lock would ship and the race would survive, so both notes now name buf.mu and say what s.mu does and does not exclude. Two comments in the cluster harness quoted Eventually(c.FrontendAlive) .Should(BeFalse()). FrontendAlive takes an index, so Gomega rejects that with "requested 1 arguments but received 0". Both now quote the closure form the specs actually use, and say why the closure is needed. proveHealthCheckingIsAlive claimed to prove the health monitor ran for the whole preceding window. It proves the monitor was alive at the end of it, and inferring backwards needs any wedge to be sticky. In the peer-replica-death spec that inverts: health checks are single-flighted by a session-scoped pg_try_advisory_lock, the spec SIGKILLs the replica that may hold it, and until Postgres reaps the session the survivor acquires nothing and checks nothing silently. Consistently(healthy) can then pass because nothing was checking, with the positive control still succeeding once the lock frees. The doc now states what is proven, names that gap, and says the assertion is a floor rather than a proof. The Makefile still called DISTRIBUTED_TEST_FLAKES a retry count, which is what seeded that error into the two docs just corrected against it, and the workflow called the 15s window a reconcile tick when the mechanism is HealthCheckInterval in the node health monitor. Also: the cluster suite measured 509.1s / 509.8s / 512.3s, so about 8m30s and not the 8m39s/8m40s three files claimed; the dead-worker spec title implied two independent detectors when both probes read one advisory-lock-serialised verdict out of the same row; and the sanitizeDBName length assertion used <= 50, which an empty string also satisfies, where the invariant for an over-long input is exactly 50. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
172e1f0bed |
test(distributed): prove health checking was alive during the failover windows
The two specs that assert a healthy worker stays healthy were pure negatives: they say nothing happened. A cluster whose health checking had wedged, by leaking the advisory lock the monitor takes at health.go:110, would freeze the roster and satisfy both while observing a corpse. Kill the worker once the window closes and require the roster to settle it to offline, so the preceding Consistently is a statement about behaviour rather than about a stopped clock. Applied to the cold-restart spec as well as the peer-death one: a restart is exactly the event that could leave a replacement unable to check anything. Document the hazard that can make an offline assertion hang. The staleness branch skips a node already marked unhealthy (health.go:153-155), a skip meant for nodes an operator took down, which also swallows the flap: an unhealthy mark landing after the heartbeat goes stale means MarkOffline is never called and the node stays unhealthy forever. Name the file and line at the assertion, and have the failure message say so when the roster shows a node stuck there, so a timeout sends the reader to LocalAI rather than to the harness. Stop calling the two-replica registration spec a race. Start spawns workers sequentially and the registrations land about a second apart; it is a shared-roster identity test, and saying otherwise invites someone to trust it for something it does not check. WorkerRegistrar now bound-checks its index like every other index-taking method here. It answered 0 for an out-of-range worker, and 0 is a real frontend index, so the failure mode was a spec killing the wrong replica. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
0a9ae2d8ee |
test(distributed): cover replica death, rolling restart and racing replicas
Four scenarios with no prior equivalent: killing a replica must not disturb a worker that never depended on it, a cold-restarted replica must rehydrate the roster from shared state and keep accepting the worker's heartbeats, a dead worker must settle to offline on every replica, and two replicas registering a worker each must converge on one roster. The timings are measured, not assumed. Node liveness is heartbeat freshness, so the only eviction path is StaleNodeThreshold (60s) plus one HealthCheckInterval tick (15s), and neither is reachable from the CLI. A worker whose registrar was killed was observed going offline at 74.2s. Every window here is sized to outlast that, because an assertion that expires before the system could have reacted proves nothing. Two assertions are deliberately unlike the obvious form. Statuses are compared for equality against a probe that returns a sentinel on error, rather than asserting a name is absent from the healthy list: the list probe returns nil on any error, and "does not contain" is satisfied by nil, so a 401 at the second replica would have passed while observing nothing. And a killed worker is required to settle to exactly offline, because it first flaps to unhealthy at ~8s and back to healthy at ~14s, which any not-healthy matcher would accept. SpreadWorkerRegistrations is new, off by default, and exists so the racing spec is a race: the harness otherwise points every worker at frontend 0, which would have left that scenario asserting on two sequential writes through one process. The default is unchanged because the baseline specs depend on it. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
0864662ba0 |
test(distributed): require the cluster binaries by default under CI
The previous round made a missing binary fail instead of skip, but only when a workflow remembered to set LOCALAI_E2E_REQUIRE_BINARIES. That leaves the silent pass one forgotten line away: the Cluster label partition is two specs, Ginkgo exits 0 on skips, and a job that skips both reports "0 Passed | 2 Skipped" and goes green having never started a cluster. So the polarity is inverted. Binaries are required whenever CI is set, which GitHub Actions always does, and the flag now exists to force the requirement OFF rather than to be remembered ON. A local developer sees no change, since CI is unset in an ordinary shell and a missing binary still skips with a message naming the path and how to build it. off, no, n and disabled are honoured as off; ParseBool rejects them, and reading a word that unambiguous as its opposite would be a worse trap than the one this removes. Also correct a claim the previous commit message got wrong. Comparing the worker's registration id across the two replicas does not pin the topology: NodeRegistry.Register looks a node up by name and preserves the existing id, and both replicas read one Postgres, so registering the worker with every frontend would yield identical ids too. The assertion is still worth keeping for what it does catch, a replica answering from its own registry or database instead of the shared one, and the comment now says that and nothing more. The topology fact moves to where someone would break it: a note on LOCALAI_REGISTER_TO recording that workers register with frontend 0 only, that the cross-replica specs depend on it, and that nothing in those specs can detect a change to it. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
d318e4cfae |
test(distributed): correct the failure-primitive comments and guard the wipe
Review round 1. Comments only, plus one guard. The note on Process.alive claimed the exited check closed the zombie window. It does not. The reaper closes exited only after Cmd.Wait returns, and Wait marks the os.Process done before returning, so exited being closed implies signal 0 already errors and the branch cannot fire earlier than the one it precedes. The window between the child exiting and waitid collecting it stays open in both versions, and the only real mitigation is for callers to poll with Eventually rather than sample once. Keep the check as hygiene, say what it actually does, and say it again on the exited field, so nobody reads the old claim and drops the Eventually. Record what the cold wipe destroys. The harness sets no LOCALAI_STORAGE_URL, so the object store is a directory under DataPath, and quantization and fine-tune outputs live there too. Postgres keeps the job row; the artifact it points at does not survive the restart. A spec that asserts otherwise will fail for a storage reason wearing a failover costume. Tell callers to let a graceful stop finish before restarting: RestartFrontend terminates with SIGKILL, so pairing it straight after StopFrontendGracefully cuts the drain short and silently converts the rolling-update case into the crash case. Refuse to wipe when the cluster has no work dir. frontendDataDir is relative when baseDir is empty, so a Cluster built by some future test helper without one would have RemoveAll walking frontend-N/data inside the source tree. The guard sits before terminate, so a refusal leaves the cluster as it was. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
499824e619 |
test(distributed): add kill and restart primitives to the cluster harness
The point of running LocalAI as real child processes is to be able to take
one away. Add KillFrontend (SIGKILL, the lost replica), StopFrontendGracefully
(SIGTERM, the rolling update), KillWorker, RestartFrontend and FrontendAlive.
RestartFrontend pins the dead replica's original port. Workers read
LOCALAI_REGISTER_TO once at boot and never re-resolve it, so a replica that
returns on a fresh port is unreachable by exactly the workers that registered
with it and the failover under test never happens.
It also wipes the replica's data directory, so the process comes back with
empty local state and has to rehydrate node, session and job state from the
shared Postgres and NATS. Reusing the directory would model a pod with a
persistent volume and hide the class of bug these tests exist to find. That
is only safe because the harness pins LOCALAI_AUTH_HMAC_SECRET; otherwise the
wipe would take {DataPath}/.hmac_secret with it and every session minted
before the restart would 401 afterwards.
FrontendAlive consults the reaper's exited channel before signal 0: a child
that has died but has not yet been waited on is a zombie, and signal 0 to a
zombie succeeds, which would report a dead replica as alive.
The new specs cover argument validation only. Killing, stopping and
restarting a live process needs a built binary plus Postgres and NATS, so
those paths stay unexecuted until the failover suites land.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Ettore Di Giacinto
|
861d3c107c |
test(distributed): give each frontend its own data dir and one pinned secret
Session rows are keyed by an HMAC of the token under a secret generated
per instance into {DataPath}/.hmac_secret. The replicas shared that
secret only because they shared a working directory, and that directory
was the source tree. Give each frontend LOCALAI_DATA_PATH under its own
baseDir and pin LOCALAI_AUTH_HMAC_SECRET, so a session minted at one
replica resolves at every other one by construction.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Ettore Di Giacinto
|
f61bea8e75 |
test(distributed): add admin session helper to the cluster harness
The register handler answers 201 both for "user created, here is your session" and for "this email already exists", so the status code cannot tell a fresh registration from a repeat one. Key on the session cookie instead and fall through to login when it is absent. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
ddd3b8b64c |
test(distributed): make the cluster harness survive a restart
Restarting a frontend replica must not move it: workers read LOCALAI_REGISTER_TO once at boot and never re-resolve it, so a replica that returns on a fresh port is unreachable by the workers that registered with it. startFrontend now takes the port, with <= 0 meaning "allocate". Process logs are opened for append rather than truncated, so a restarted process cannot erase the log of the instance that died, which is the log a failover post-mortem needs. The post-SIGKILL wait is bounded, so one stuck child no longer becomes a suite-wide timeout that names nothing. Stop is nil-safe because Start returns a nil cluster after stopping itself. Start's doc comment no longer claims to wait for worker registration; that needs an authenticated admin session, so it now says callers must poll /api/nodes themselves. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
e3ad0f0798 |
test(distributed): add a process-level cluster harness
Runs local-ai as real child processes, one per frontend replica and one per worker, against containerised infrastructure. The in-process suites cannot express frontend-replica failure: there is no process to kill and no real HTTP boundary between a worker and the frontend it registered with. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |