mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-12 22:33:54 -04:00
Ettore Di Giacinto
0683fb1579
Everything this phase built was proven by unit and integration specs. This is the first run of it against the real binaries: a frontend replica per process, a worker that binds nothing routable, real inference over the result. Four scenarios, each with the question "what would make this pass if the tunnel were doing nothing" answered rather than left open. A worker with no advertised address is reached through its tunnel. The roster is asserted to report it advertising nothing, so there is no address a frontend could have dialled instead, and node_connections is asserted to name the replica that serves the request. A request landing on the replica that does NOT own the worker is relayed to the one that does. With N replicas behind round robin that is (N-1)/N of production traffic, so it gets the FIRST request for its model: the backend install, the file staging on the http tag, and the gRPC load and predict all cross the relay. Which replica owns the tunnel is read from the ownership table through the production Owner query and mapped to a frontend index through the address the harness pins per replica; the non-owner is derived from that reading and asserted to be a non-owner immediately before the request, rather than assumed from the harness default. Sending the same request to the owner reddens it. Killing the owning replica re-homes the worker onto the survivor. The worker dials a balancer rather than a replica, because LOCALAI_REGISTER_TO is resolved once at boot and is the tunnel endpoint as well as the registration one: aimed at a single replica, a worker has nowhere to reconnect to when that replica dies, and the re-home cannot happen at all. Removing the kill reddens it. And the negative control for the whole suite, which is why the other three mean anything. Frontend and worker share a host here, so every backend port the frontend names in a stream target is one it could have dialled directly; if it did, the first three would pass with the tunnel inert. LOCALAI_WORKER_TUNNEL is no longer usable for this, because it is a fatal startup error and a worker that never started says nothing about a worker reachable some other way. The balancer answers the tunnel connect path itself instead, leaving a worker that registers, heartbeats, reports healthy and holds no tunnel. It is asserted to have dialled and been refused, asserted to be held by nobody, and then asserted unreachable with the refusal naming the missing route. Then the block is lifted, nothing else changes, and the same request succeeds: that is what attributes the refusal to the tunnel rather than to any of the ordinary reasons an e2e inference fails. The fifth spec measures the head-of-line blocking this phase deferred three times. 128 MiB crosses the session while a warm model is probed back to back, direct and relayed. Median latency is unchanged, the worst probe is about 3x the baseline median and about a seventeenth of the transfer window, and the transfer runs at 415-490 MB/s direct and 222-268 MB/s relayed. A session that head-of-line blocked would park a probe for the length of the window. Leave the yamux windows untuned; and note this is loopback, so it says the multiplexing does not serialise and says nothing about a link with a bandwidth-delay product. The load spec is measured against a control that the first version did not have. It passed with the bulk artifact cut to 4 KiB, because the window it read probes against was mostly cold-load overhead: it would have reported a clean bill on a session carrying no large message. The same cold load now runs twice, once empty and once bulk, and the difference between the windows is asserted to be real before any latency is read from it. Two defects on the base commit came out of this. cluster_peerlink_test.go has been red since the relay landed, deterministically, in isolation and in the suite. It asserted that an accepted peer stream is refused at once, on the premise that phase 1 installs no relay. The relay correctly waits fifteen seconds for a frame naming the worker, and the spec's budget was five. It now writes a relay request for a node no replica holds and asserts the refusal is ErrNotOwner and specifically not ErrNoConnection, which is a stronger spec than the one it replaces and the only thing in the e2e suite that exercises the relay's refusal path. The harness handed a worker's own HTTP port to a backend process. It took two ports from freeport and used one as the gRPC base and the other for the file transfer server; freeport returns adjacent ports often, and the backend allocator hands out base, base+1, base+2, so the second backend started on a worker was regularly given the HTTP server's port and died with EADDRINUSE. No spec had started two backends on one worker before, so it had never fired; the load spec starts five and it failed about one run in three. Each worker now reserves a contiguous bind-probed block laid out the way production lays it out, below the kernel's ephemeral range, with LOCALAI_GRPC_MAX_PORT bounding the allocator to it. The underlying production defect is not fixed here and is recorded in the report: allocatePort never checks that a port is free, and its default range overlaps the ephemeral range on every Linux box. Constraint 6, whether distributed mode should now refuse to start without an advertised address, is DEFERRED, and the comment and the docs that described the cost were understating it. A replica with no advertised address writes no instances row, and Owner joins a connection against a live instance, so a worker whose tunnel lands there is unroutable from every OTHER replica while being registered and healthy. Refusing to start would still be wrong, because the deployments it would break are single-host ones with no peers to be unreachable by, and telling those apart at startup is a design with its own specs. Both places now say what actually happens. Suite wall clock 592s for 15 specs, up from 502s for 10 of which 2 were red. The CI budget of 20 minutes does not move. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>