fix(cluster): stop a late request frame reading as the worker's verdict

Making a worker's refusal reaping evidence created a defect one layer
along, at the producer. The worker refused a ReadStreamRequest failure with
ErrStreamRequestInvalid and its own comment said "Includes the deadline
above expiring", which was harmless while every refusal reached the
frontend as "no route" and became a reap the moment one of them did not. So
a request frame that had merely not ARRIVED yet was reported as a
non-transient verdict about a backend.

It is reachable on the relay path, which carries most production traffic:
the worker's header timer starts when the OWNING replica opens the stream,
while the frame is written by the DIALLING replica only after the relay's
acceptance travels back to it, so a whole peer-link round trip runs inside
that window, on a link this design deliberately loads with multi-gigabyte
artifacts beside token streams. For a long-deadline caller the endpoint is
ConnectionEvictingClient, which stops the model across the fleet. It also
falsified the "neither clears on its own" argument that licensed the reap.

There is now a fourth refusal, ErrStreamNotServed, for what a worker could
not serve for a reason of its OWN. It is deliberately outside
IsWorkerAnswer, so it reaches a consumer under the no-route umbrella and
reaps nothing, which is the same treatment an unrecognised code already
gets. Four producers move onto it: a request frame that timed out (a
malformed one stays a verdict, because that is a frontend bug no retry
fixes), both SetReadDeadline failures, which are facts about the stream and
not about a target nothing has dialled yet, and WriteStreamRefusal's
default for a reason nobody classified.

classifyServiceFailure keeps ErrStreamTargetUnavailable as its default on
purpose: inverting it would make errno enumeration the single point of
failure for the reap, and a miss there is a row nothing can ever delete.
What it gains is a deny-list of two causes that are provably this worker's
own clock or its own context.

Also:

- The read-site caller-deadline guard in the handshake was unpinned: the
  existing seam spends the budget before the handshake starts, so only the
  write could ever fail. A spec whose deadline falls between the request and
  the reply pins it, and each guard now reddens on its own.
- The documented worker-first failure line omitted the JSON error envelope
  the old frontend returns, so an operator grepping it found nothing.
- The peer-link disclosure names the aimable per-session receive window in
  all four places, and LastDialErrorOf records why a third consumer must go
  through IsWorkerAnswer rather than roll its own list.

Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

docs/content/features/distributed-mode.md

@@ -114,7 +114,7 @@ environment:
 The peer link is served at `/api/cluster/peer` and authenticates with `LOCALAI_REGISTRATION_TOKEN`, the same shared secret workers register with. Replicas that disagree about it cannot link. A replica that stops heartbeating for 30 seconds is dropped from the table by the others, along with the worker-connection rows it owned.
 
 {{% notice note %}}
-**The peer link has no per-replica credential yet.** It checks the shared registration token and takes the replica id in `?id=` on trust. Anything already holding that token - every worker holds it - can therefore open a peer link, relay through it to every worker tunnel a replica owns, and by declaring another replica's id displace that replica's inbound link. Treat `LOCALAI_REGISTRATION_TOKEN` as a cluster-wide secret with the blast radius of the whole fleet: give it its own value per deployment, do not reuse it elsewhere, and keep `/api/cluster/peer` on a network only your replicas and workers can reach. Per-replica credentials for this route are planned.
+**The peer link has no per-replica credential yet.** It checks the shared registration token and takes the replica id in `?id=` on trust. Anything already holding that token - every worker holds it - can therefore open a peer link, relay through it to every worker tunnel a replica owns, by declaring another replica's id displace that replica's inbound link, and hold sessions open against the per-session receive window, which the peer-link code sizes at roughly 31 GiB of unread data per session and which on this route is also a memory budget an attacker can point at one replica. Treat `LOCALAI_REGISTRATION_TOKEN` as a cluster-wide secret with the blast radius of the whole fleet: give it its own value per deployment, do not reuse it elsewhere, and keep `/api/cluster/peer` on a network only your replicas and workers can reach. Per-replica credentials for this route are planned.
 {{% /notice %}}
 
 ### Worker tunnels
@@ -207,7 +207,13 @@ That distinction is the whole point rather than a nicety. A scheduler told that
 - **Frontends first (correct).** Old workers keep running, keep heartbeating and keep their `node_models` rows: the new frontend reports them as unroutable rather than as gone, so nothing is rescheduled and nothing is reaped. What fails is requests for models on a worker that has not been restarted yet. That is a real degraded window, but it is bounded by how fast you roll the workers, it heals itself as each one comes back, and no state is lost.
   - **What you will see while it lasts:** requests for models on a not-yet-restarted worker fail with "no route to the worker", while `GET /api/nodes` still shows that node healthy and heartbeating and its models still listed. Restart the worker and it clears. Nothing needs fixing; you are watching the window close.
 - **Workers first (this fails, do not do it).** An old frontend has no `/api/cluster/connect` route for the worker to dial *and* rejects the new worker's registration outright, because the worker no longer sends an address and the old frontend requires one. A 4xx is a verdict rather than an outage, so the worker reports the reason on the **first** attempt and exits instead of retrying. Every worker you restart is a worker you take out of the fleet until the frontends are upgraded.
-  - **What you will see if you do it anyway:** each restarted worker exits within a second or two of starting, with `registration failed with status 400: address is required for backend workers: the frontend refused this registration`. The fleet drains one node per restart, and the nodes that are left are the ones you have not touched yet.
+  - **What you will see if you do it anyway:** each restarted worker exits within a second or two of starting, with
+
+    ```
+    registration failed with status 400: {"error":{"code":400,"message":"address is required for backend workers","type":"node_error"}}: the frontend refused this registration
+    ```
+
+    The fleet drains one node per restart, and the nodes that are left are the ones you have not touched yet. Grep for `address is required for backend workers` if your log collector reflows the line.
 
 A worker that cannot reach its frontend *at the network level* retries with exponential backoff and never gives up, so restarting a worker is all that is needed to close the frontend-first window. A worker whose registration is **rejected** does not retry, which is what makes the wrong order destructive rather than slow.