fix(distributed): let a worker's own refusal be evidence about its backend

A worker that refuses a stream has answered, and cluster.Dial keeps the three
tunnelproto sentinels out of the ErrNoRoute umbrella precisely so a consumer
can act on that. No consumer did. Since workers stopped listening, a backend
process that crashed on a healthy worker is no longer a dead listener's
codes.Unavailable: the worker refuses the stream with
ErrStreamTargetUnavailable, gRPC flattens it into Unavailable anyway, and
nodes.unroutable reported the whole thing as "this frontend has no route".
Every reap path then answered ProbeUnknown and left the row, so the replica
slot never freed and at the default MaxReplicasPerModel=1 the only cleanup
left was LRU eviction of models that were working.

isWorkerAnswer is exported as cluster.IsWorkerAnswer, so the errors the dialer
keeps out of the umbrella are by construction the errors the consumers treat
as the worker answering. nodes.unroutable and pkg/model's transportFailure
both use it; ConnectionEvictingClient, the site reached during inference, goes
through transportFailure rather than asking the transport directly. A reply
code this frontend does not recognise is still not an answer, so a newer
worker's vocabulary costs a retry and not a replica.

The reap guards keep the allow-list rather than requiring ErrNoRoute: an
unrecognised dial error must mean "no route", never "the backend is gone".

Also in this final pass over the branch:

- Docs: recommend upgrading FRONTENDS first, with the symptom of each order.
  Workers-first fails now that a 4xx registration is a verdict rather than an
  outage, so an old frontend's "address is required for backend workers" makes
  each restarted worker exit and drains the fleet a node per restart.
- Docs: LOCALAI_WORKER_TUNNEL=false is a fatal startup error, not a degraded
  mode, in both places that described it; and a frontend rollback needs every
  worker restarted, because re-registration force-clears the address columns.
- A replica with no advertised address now says so every five minutes and
  names the workers only it can reach, instead of one startup warning for a
  cost paid for the life of the process.
- callerRanOut's rule now holds at all three siblings, so an expired caller
  deadline stops reading as a broken tunnel; probeHealth's withdrawn reason
  for using the raw client is corrected; the dead DoOrCached is deleted and
  its coverage kept on DoOrCachedResult; sweepLeakedInFlight enumerates the
  outcomes that reach it.
- The peer route's self-declared id is recorded as a phase-3 deferral, in the
  handler, in the isolation claim it narrows, and in the operator docs.

Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

core/application/distributed.go

@@ -227,13 +227,20 @@ func initDistributed(cfg *config.ApplicationConfig, authDB *gorm.DB, configLoade
 		// itself; nobody else can. On N replicas behind round robin that is
 		// (N-1)/N of the traffic for that worker.
 		//
-		// It stays a warning all the same. Refusing to start would take out
-		// every existing single-host deployment, whose route to a local
-		// database is loopback and which has no peers to be unreachable by;
-		// the deployments this hurts are multi-replica ones, and telling those
-		// two apart at startup is a change with its own design and its own
-		// specs rather than a line here.
-		xlog.Warn("This replica will not be reachable by its peers: no advertised address",
+		// It does not refuse to START. Refusing would take out every existing
+		// single-host deployment, whose route to a local database is loopback
+		// and which has no peers to be unreachable by; the deployments this
+		// hurts are multi-replica ones, and telling those two apart at startup
+		// is a change with its own design and its own specs rather than a line
+		// here.
+		//
+		// What it does not get to do is stay quiet. One startup line scrolls
+		// away in seconds and the cost is paid for the whole life of the
+		// process, on a symptom (workers that 5xx from most of the fleet) whose
+		// obvious reading is "the worker is broken". So this is an ERROR, not a
+		// warning, and nagUnadvertisedReplica below repeats it for as long as
+		// the state lasts, naming the workers it is currently costing.
+		xlog.Error("This replica is not registered in the cluster: no advertised address. Peers cannot reach it, and any worker whose tunnel lands here will be unroutable from every other replica",
 			"error", err, "knob", "LOCALAI_DISTRIBUTED_ADVERTISE_ADDR")
 	} else {
 		membership = cluster.NewMembership(clusterRegistry, cfg.Distributed.InstanceID, advertised, internal.PrintableVersion())
@@ -260,6 +267,10 @@ func initDistributed(cfg *config.ApplicationConfig, authDB *gorm.DB, configLoade
 	// through.
 	if membership != nil {
 		membership.SetTunnels(tunnels)
+	} else {
+		// The runtime symptom the startup line cannot be. See
+		// nagUnadvertisedReplica.
+		go nagUnadvertisedReplica(cfg.Context, tunnels.Held, unadvertisedNagInterval, logUnroutableWorkers)
 	}
 
 	// The links peers dial IN, with the relay installed on them. This is what
@@ -594,6 +605,64 @@ func initDistributed(cfg *config.ApplicationConfig, authDB *gorm.DB, configLoade
 	}, nil
 }
 
+// unadvertisedNagInterval is how often a replica that could not advertise
+// itself says so again.
+//
+// Five minutes is chosen against the log it lands in, not against the urgency:
+// the condition never clears on its own, so this line is either read once and
+// acted on or it is noise for the life of the process, and a noisy line gets
+// filtered rather than fixed. It is still frequent enough that the state is
+// visible in any window of logs an operator pulls while investigating the
+// symptom it causes.
+const unadvertisedNagInterval = 5 * time.Minute
+
+// nagUnadvertisedReplica repeats, for as long as the process runs, that this
+// replica is invisible to its peers, and names what that is currently costing.
+//
+// It exists because the deferral it accompanies changed cost between phases and
+// nothing about the deployment says so. Before workers held tunnels, a replica
+// with no advertised address was merely unreachable BY peers and could still
+// dial every worker directly, so a startup warning was proportionate. Now a
+// worker's tunnel lands on one replica and every other replica reaches it by
+// relaying to the owner, and the owner is resolved by joining the connection
+// row against a LIVE INSTANCES ROW - which this replica does not have. So every
+// worker that lands here is answered as unroutable everywhere else: on N
+// replicas behind round robin, (N-1)/N of that worker's traffic fails, while
+// this replica serves it perfectly and reports nothing.
+//
+// held is passed as a function rather than the registry so this can be driven
+// without one, and alarm is passed rather than logged inline so a spec can
+// observe the alarms instead of scraping a log.
+func nagUnadvertisedReplica(ctx context.Context, held func() []string, every time.Duration, alarm func([]string)) {
+	ticker := time.NewTicker(every)
+	defer ticker.Stop()
+	for {
+		select {
+		case <-ctx.Done():
+			return
+		case <-ticker.C:
+			alarm(held())
+		}
+	}
+}
+
+// logUnroutableWorkers says what the state costs RIGHT NOW.
+//
+// The two cases are kept apart because they call for different urgency and an
+// operator can tell them apart at a glance. With no worker held this is a
+// misconfiguration that has not been paid for yet; with workers held, every one
+// of them is named, because "which worker is broken" is the question the
+// symptom sends an operator to ask and the answer is that none of them is.
+func logUnroutableWorkers(held []string) {
+	if len(held) == 0 {
+		xlog.Warn("This replica is still not registered in the cluster: no advertised address. No worker holds a tunnel here yet; the first that does will be unroutable from every other replica",
+			"knob", "LOCALAI_DISTRIBUTED_ADVERTISE_ADDR")
+		return
+	}
+	xlog.Error("This replica is not registered in the cluster and holds worker tunnels: those workers are unroutable from every OTHER replica, and requests for their models fail there with no route. The workers are healthy; this replica is invisible",
+		"workers", held, "worker_count", len(held), "knob", "LOCALAI_DISTRIBUTED_ADVERTISE_ADDR")
+}
+
 // advertisedPeerAddr is the host:port peers dial to reach this replica.
 //
 // The operator's value wins outright. Otherwise it is derived from the port

core/application/unadvertised_replica_test.go

@@ -0,0 +1,73 @@
+// SPDX-License-Identifier: MIT
+
+package application
+
+import (
+	"context"
+	"time"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+// The runtime symptom for a deferral whose cost changed between phases.
+//
+// Not refusing to start without an advertised address stays deferred on
+// purpose: refusing would take out every single-host deployment. What is not
+// deferred is telling the operator, repeatedly, that this replica is invisible
+// and which workers that is costing - because the symptom it produces (a worker
+// that 5xxs from most of the fleet) reads as a worker problem, and a single
+// startup line has scrolled away long before anyone goes looking.
+var _ = Describe("the alarm for a replica with no advertised address", func() {
+	It("keeps firing for as long as the state lasts, and names the workers it costs", func() {
+		// Repetition is the property. A one-shot alarm is the startup line
+		// again, which is what was already there and was not enough.
+		ctx, cancel := context.WithCancel(context.Background())
+		DeferCleanup(cancel)
+
+		alarms := make(chan []string, 8)
+		go nagUnadvertisedReplica(ctx, func() []string { return []string{"w1", "w2"} },
+			time.Millisecond, func(held []string) { alarms <- held })
+
+		// Two, not one: the second is what a one-shot implementation fails.
+		var first, second []string
+		Eventually(alarms, "10s").Should(Receive(&first))
+		Eventually(alarms, "10s").Should(Receive(&second))
+		Expect(first).To(ConsistOf("w1", "w2"),
+			"the workers this is costing are the answer to the question the symptom provokes")
+		Expect(second).To(ConsistOf("w1", "w2"))
+	})
+
+	It("reads the held set on every tick rather than the one it started with", func() {
+		// A replica accumulates tunnels while it runs, so an alarm bound to the
+		// set at startup would name an empty list forever on exactly the
+		// deployment where the cost is real.
+		ctx, cancel := context.WithCancel(context.Background())
+		DeferCleanup(cancel)
+
+		workers := make(chan []string, 32)
+		for range 32 {
+			workers <- []string{"w-late"}
+		}
+		alarms := make(chan []string, 8)
+		go nagUnadvertisedReplica(ctx, func() []string { return <-workers },
+			time.Millisecond, func(held []string) { alarms <- held })
+
+		var got []string
+		Eventually(alarms, "10s").Should(Receive(&got))
+		Expect(got).To(ConsistOf("w-late"))
+	})
+
+	It("stops when the process context ends", func() {
+		ctx, cancel := context.WithCancel(context.Background())
+		stopped := make(chan struct{})
+		go func() {
+			defer GinkgoRecover()
+			nagUnadvertisedReplica(ctx, func() []string { return nil }, time.Hour, func([]string) {})
+			close(stopped)
+		}()
+
+		cancel()
+		Eventually(stopped, "10s").Should(BeClosed())
+	})
+})

core/http/endpoints/cluster/peer.go

@@ -38,6 +38,30 @@ func PeerHandler(token string, onSession func(peerID string, sess *yamux.Session
 			return echo.NewHTTPError(http.StatusUnauthorized, "unauthorized")
 		}
 
+		// SELF-DECLARED, and knowingly so. Unlike the worker route next door,
+		// which resolves ?id= to a node row and checks that node's OWN minted
+		// credential, this route has only the shared cluster token to check,
+		// so the id is a label and not a claim anything verifies.
+		//
+		// What that costs, exactly, for anything already holding the shared
+		// token (every worker holds it, and it is the same token that
+		// authenticates registration): it can relay to every worker tunnel this
+		// replica owns, reaching every backend gRPC process and every worker's
+		// file-transfer server; and by declaring a legitimate replica's id it
+		// can make SessionStore.Accept evict that replica's inbound link, at
+		// will. Neither is a new capability in KIND - before workers stopped
+		// listening, a holder of that token could already dial any worker's
+		// advertised ports directly - but the token is now the only thing
+		// between an attacker and the whole fleet's tunnels.
+		//
+		// It is deferred rather than patched, because the cheap patch does not
+		// work: checking ?id= against the instances table stops an invented id
+		// and stops nothing else, since the attack declares a REAL replica's
+		// id, and it would buy a false sense of a closed hole. Closing it takes
+		// a credential per replica, minted where a replica joins the instances
+		// table and presented here, which is a design with its own migration
+		// and its own specs. Tracked as the phase-3 item named at
+		// nodes.BackendNode.TunnelTokenHash.
 		peerID := c.QueryParam("id")
 		if peerID == "" {
 			return echo.NewHTTPError(http.StatusBadRequest, "missing peer id")

core/services/cluster/dialer.go

@@ -112,7 +112,7 @@ func isAbsenceClaim(err error) bool {
 	return errors.Is(err, ErrNoConnection) || errors.Is(err, ErrInstanceNotFound)
 }
 
-// isWorkerAnswer reports whether an error is the WORKER's own refusal, read off
+// IsWorkerAnswer reports whether an error is the WORKER's own refusal, read off
 // the reply it sent.
 //
 // Those three sentinels are the only ones ReadStreamReply produces from a frame
@@ -124,7 +124,17 @@ func isAbsenceClaim(err error) bool {
 // would let a newer worker's vocabulary be read by an older frontend as
 // evidence about a backend, and the consequence of guessing wrong in that
 // direction is a reaped replica; guessing wrong the other way costs a retry.
-func isWorkerAnswer(err error) bool {
+//
+// It is EXPORTED because it is half of a contract, not an implementation
+// detail. Dial keeps these three out of the ErrNoRoute umbrella so that a
+// consumer can act on them; a consumer that cannot ask "was this the worker
+// speaking?" has no way to use that, and for a whole phase none could, so every
+// worker refusal reached the schedulers as "this frontend has no route" and
+// nothing could ever be reaped. The two sides must agree on the SAME set, so
+// there is one predicate and both call it: see nodes.unroutable and
+// model.transportFailure, whose job is to answer "did this call reach a
+// backend?" and for whom a refusal means it did.
+func IsWorkerAnswer(err error) bool {
 	return errors.Is(err, ErrStreamTagUnknown) ||
 		errors.Is(err, ErrStreamTargetUnavailable) ||
 		errors.Is(err, ErrStreamRequestInvalid)
@@ -293,6 +303,32 @@ func (d *WorkerDialer) relay(ctx context.Context, nodeID, tag, target string) (n
 // session, and a frontend that retries would exhaust the worker's stream
 // budget rather than the worker's patience.
 func (d *WorkerDialer) handshake(ctx context.Context, stream net.Conn, nodeID, tag, target string) (net.Conn, error) {
+	// blameCaller attributes a handshake I/O failure to the CALLER's own spent
+	// budget when that is what ended it, and returns nil when it was not.
+	//
+	// The third instance of the rule peerlink.go states in full at callerRanOut:
+	// the handshake deadline IS the caller's deadline whenever the caller's is
+	// the shorter (see handshakeDeadline), so a caller that has run out makes
+	// the socket's own timer fire, and the resulting i/o timeout arrives here
+	// while ctx.Err() may still read nil because nothing orders the two timers.
+	// Reported plainly, that is "the tunnel would not carry the request" for a
+	// worker that is connected, healthy and idle.
+	//
+	// The umbrella stays on either way, so Dial's contract is unchanged; what
+	// changes is that context.DeadlineExceeded is matchable underneath and the
+	// log line names the caller instead of the worker. It deliberately runs
+	// BEFORE the worker-answer check below, so a refusal that arrived in the
+	// same instant the budget expired is reported as the caller's timeout: a
+	// spent deadline must never be able to manufacture evidence about a
+	// backend, which is the same direction peerlink.go takes for absence.
+	blameCaller := func(what string) error {
+		ctxErr := callerRanOut(ctx)
+		if ctxErr == nil {
+			return nil
+		}
+		return routeFailure(nodeID, fmt.Errorf("%s: the caller's own budget ran out: %w", what, ctxErr))
+	}
+
 	if err := stream.SetDeadline(handshakeDeadline(ctx)); err != nil {
 		_ = stream.Close()
 		return nil, routeFailure(nodeID, fmt.Errorf("arming the handshake deadline: %w", err))
@@ -302,11 +338,17 @@ func (d *WorkerDialer) handshake(ctx context.Context, stream net.Conn, nodeID, t
 		// The stream would not carry the request, so the tunnel broke under it.
 		// Nothing was asked of the worker and nothing was learned about it.
 		_ = stream.Close()
+		if blamed := blameCaller(fmt.Sprintf("asking for %q on %q", tag, target)); blamed != nil {
+			return nil, blamed
+		}
 		return nil, routeFailure(nodeID, fmt.Errorf("asking for %q on %q: %w", tag, target, err))
 	}
 	if err := ReadStreamReply(stream); err != nil {
 		_ = stream.Close()
-		if isWorkerAnswer(err) {
+		if blamed := blameCaller(fmt.Sprintf("opening %q", tag)); blamed != nil {
+			return nil, blamed
+		}
+		if IsWorkerAnswer(err) {
 			// The worker wrote a refusal, so it is connected and answering.
 			// This is the ONE failure on the whole path that is real evidence
 			// about the worker, and putting the umbrella on it would throw that

core/services/cluster/dialer_test.go

@@ -292,6 +292,39 @@ var _ = Describe("The worker dialer", func() {
 			Expect(out.err).ToNot(MatchError(cluster.ErrNoRoute))
 		})
 
+		It("blames the caller's own spent budget, not the worker, when the handshake ends on the deadline", func() {
+			// The third and last site of peerlink.go's callerRanOut rule.
+			//
+			// The handshake deadline IS the caller's whenever the caller's is
+			// shorter (handshakeDeadline), so a caller that has run out makes
+			// the stream's own timer fire, and the i/o timeout arrives here
+			// while ctx.Err() may still read nil. Without the guard this reads
+			// as "the tunnel would not carry the request" for a worker that is
+			// connected, healthy, and simply not answering yet, which is what a
+			// worker under load looks like.
+			//
+			// The worker accepts the stream and says nothing, so the ONLY thing
+			// that can end this handshake is the deadline; there is no sleep
+			// and no race.
+			frontend, worker := workerTunnel()
+			_, err := mine.Attach(ctx, "w1", frontend)
+			Expect(err).ToNot(HaveOccurred())
+			go func() {
+				defer GinkgoRecover()
+				_, _ = worker.AcceptStream()
+			}()
+
+			d := cluster.NewWorkerDialer(mine, nil)
+			var out dialResult
+			Eventually(dialAsync(d, deadlinePassed{ctx}, "w1", cluster.StreamTagGRPC, "127.0.0.1:41000"), "10s").Should(Receive(&out))
+			Expect(out.err).To(MatchError(context.DeadlineExceeded),
+				"the caller's budget was spent; the worker never got a verdict")
+			Expect(out.err.Error()).To(ContainSubstring("the caller's own budget ran out"))
+			// The umbrella is still on it, so Dial's contract is unchanged and
+			// no consumer reads this as the worker having gone away.
+			expectNoRoute(out.err)
+		})
+
 		It("reports a broken tunnel held here as itself, not as a routing fact", func() {
 			// ErrNotOwner tells a caller to look for the worker elsewhere. For
 			// a tunnel held right here that sends it back to this replica, and

core/services/cluster/sessions.go

@@ -76,6 +76,15 @@ func (s *SessionStore) Accept(peerID string, sess *yamux.Session) {
 // is false when no link from that peer is held, which a caller must not read as
 // the peer being absent: it may be about to dial, or dialling this replica may
 // simply not be its job.
+//
+// It has NO production caller, and that is stated rather than left to be
+// discovered: nothing in the frontend routes by looking up an inbound link,
+// because the relay is driven by the streams a peer opens on the session, not
+// by this side going to find one. What Get exists for is the specs, which have
+// no other way to observe which session this store holds, and holding exactly
+// one session per peer is the property Accept's eviction is about. Deleting it
+// would delete that observation with it. Anything tempted to route on it should
+// read the paragraph above first: a missing entry is not an absent peer.
 func (s *SessionStore) Get(peerID string) (*yamux.Session, bool) {
 	s.mu.Lock()
 	defer s.mu.Unlock()

core/services/cluster/tunnel.go

@@ -290,6 +290,23 @@ func (t *TunnelRegistry) Open(ctx context.Context, nodeID string) (net.Conn, err
 
 	stream, err := held.sess.OpenStream(ctx)
 	if err != nil {
+		// The same rule peerlink.go applies to a peer, applied here to a
+		// tunnel, because the confusion is the same one: a caller whose own
+		// budget ran out gets the socket's error back before the context's
+		// cancel func has necessarily run, so ctx.Err() can still read nil
+		// while the failure is entirely the caller's. Reporting it plainly
+		// would put "the tunnel this replica holds would not carry a stream"
+		// in an operator's log for a worker that is fine and a client that was
+		// impatient. callerRanOut settles it on the wall clock; see its
+		// comment for why ctx.Err() alone is not the question.
+		//
+		// The caller's error is wrapped rather than returned bare, so
+		// WorkerDialer's contract still holds (every failure to resolve or open
+		// carries ErrNoRoute) and context.DeadlineExceeded stays matchable
+		// underneath for anyone that wants to tell the two apart.
+		if ctxErr := callerRanOut(ctx); ctxErr != nil {
+			return nil, fmt.Errorf("opening a stream to node %q over the tunnel held here: the caller's own budget ran out: %w", nodeID, ctxErr)
+		}
 		return nil, fmt.Errorf("opening a stream to node %q over the tunnel held here: %w", nodeID, err)
 	}
 	return stream, nil

core/services/cluster/tunnel_test.go

@@ -304,6 +304,32 @@ var _ = Describe("The worker tunnel registry", func() {
 			"holding the tunnel is a routing fact, and a failed Open is not what un-holds it")
 	})
 
+	It("blames the caller's own spent budget, not the tunnel, when a stream cannot be opened", func() {
+		// The second of the three sites where peerlink.go's callerRanOut rule
+		// has to hold. A caller whose budget ran out gets the multiplexer's
+		// error back before the scheduler has necessarily run its context's
+		// cancel func, so ctx.Err() can still read nil while the failure is
+		// entirely the caller's; reported plainly it becomes "the tunnel this
+		// replica holds would not carry a stream" in an operator's log for a
+		// worker that is fine.
+		//
+		// deadlinePassed is that window made deterministic: deadline elapsed,
+		// cancellation not delivered. Nothing about the broken session is
+		// faked.
+		frontend, worker := workerTunnel()
+		_, err := tun.Attach(ctx, "w1", frontend)
+		Expect(err).ToNot(HaveOccurred())
+
+		Expect(worker.Close()).To(Succeed())
+		Eventually(frontend.IsClosed, "10s").Should(BeTrue())
+
+		_, err = tun.Open(deadlinePassed{ctx}, "w1")
+		Expect(err).To(HaveOccurred())
+		Expect(err).To(MatchError(context.DeadlineExceeded),
+			"the caller's budget was spent, and only it can say so")
+		Expect(err.Error()).To(ContainSubstring("the caller's own budget ran out"))
+	})
+
 	It("refuses a nil session rather than claiming a tunnel that cannot carry anything", func() {
 		// A claim written for a session that does not exist publishes a tunnel
 		// to every replica in the deployment, and the fence would then have to

core/services/nodes/interfaces.go

@@ -7,6 +7,7 @@ import (
 	"net"
 	"time"
 
+	"github.com/mudler/LocalAI/core/services/cluster"
 	"github.com/mudler/LocalAI/core/services/messaging"
 	grpc "github.com/mudler/LocalAI/pkg/grpc"
 )
@@ -210,6 +211,31 @@ var ErrNoWorkerDialer = fmt.Errorf("%w: no worker tunnel dialer is configured",
 // nil, which means "the call reached a backend" and preserves the behaviour
 // every non-distributed caller has always had. Decorators are looked through;
 // see grpc.BackendUnwrapper for why that is not optional.
+//
+// A WORKER'S OWN REFUSAL also yields nil, and that is the second half of the
+// contract rather than a loophole. cluster.Dial keeps the three tunnelproto
+// sentinels out of the ErrNoRoute umbrella precisely so this function can tell
+// them apart, and for a whole phase nothing did: a backend process that crashed
+// on a healthy worker is no longer a dead listener's codes.Unavailable, it is
+// the worker refusing the stream with cluster.ErrStreamTargetUnavailable, which
+// gRPC then flattens into codes.Unavailable anyway. Reporting that as
+// unroutable made every reap path answer ProbeUnknown and leave the row, so the
+// replica slot never freed and (at the default MaxReplicasPerModel=1) the only
+// remaining cleanup was LRU eviction of HEALTHY models. A worker that answers
+// has demonstrated it is there, so the answer is evidence about its backend and
+// the reap guards may act on it.
+//
+// All three sentinels, not only the unavailable one, and the difference is
+// worth stating because two of them are not observations about the process. An
+// unknown tag means this worker does not serve gRPC streams at all; an invalid
+// request means the stored address is not a port in this worker's range.
+// Neither clears on its own, so a row that carries one is unreachable from
+// EVERY replica for as long as it exists, and reaping it converges: the model
+// is reloaded somewhere that works and re-registers a usable address. The
+// condition the phase refuses to reap on is a TRANSIENT one, and none of these
+// is transient. A reply code this frontend does not recognise is deliberately
+// not in the set (see cluster.IsWorkerAnswer), so a newer worker's vocabulary
+// reaches an older frontend as "no route" and costs a retry rather than a row.
 func unroutable(client grpc.Backend) error {
 	// LastDialErrorOf and not a type assertion: the assertion could not see
 	// past a decorator, and SmartRouter hands every routed client out wrapped.
@@ -217,6 +243,9 @@ func unroutable(client grpc.Backend) error {
 	if dialErr == nil {
 		return nil
 	}
+	if cluster.IsWorkerAnswer(dialErr) {
+		return nil
+	}
 	// Multi-%w: the umbrella this package acts on, and the cluster condition
 	// underneath it, both stay matchable.
 	return fmt.Errorf("%w: %w", ErrWorkerUnroutable, dialErr)

core/services/nodes/probe_cache.go

@@ -33,7 +33,7 @@ type probeCache struct {
 }
 
 // newProbeCache returns a probeCache with the given TTL. Zero TTL disables
-// caching: every call to DoOrCached invokes the probe.
+// caching: every call to DoOrCachedResult invokes the probe.
 func newProbeCache(ttl time.Duration) *probeCache {
 	return &probeCache{
 		ttl:  ttl,
@@ -68,17 +68,18 @@ func (c *probeCache) Invalidate(key string) {
 	delete(c.seen, key)
 }
 
-// DoOrCached returns true if key is fresh; otherwise it runs probe (coalescing
-// concurrent callers via singleflight) and caches a successful result. Failed
-// probes invalidate the cache, so a transient miss doesn't pin every
-// subsequent request to a re-probe.
-func (c *probeCache) DoOrCached(key string, probe func() bool) bool {
-	alive, _ := c.DoOrCachedResult(key, func() (bool, error) { return probe(), nil })
-	return alive
-}
-
-// DoOrCachedResult is DoOrCached with a second result: the reason the probe
-// never reached the backend, or nil when it did.
+// DoOrCachedResult returns true if key is fresh; otherwise it runs probe
+// (coalescing concurrent callers via singleflight) and caches a successful
+// result. Failed probes invalidate the cache, so a transient miss does not pin
+// every subsequent request to a re-probe.
+//
+// It is the ONLY entry point. A boolean-only sibling, DoOrCached, stood beside
+// it until probeHealth stopped using it, after which it was production code
+// held green by nothing but its own specs; the shim that reads it as a boolean
+// now lives in probe_cache_test.go, where its one caller is.
+//
+// The second result is the reason the probe never reached the backend, or nil
+// when it did.
 //
 // The second result travels through the SINGLEFLIGHT, which is the whole reason
 // it is not simply a variable the caller closes over. A closed-over variable is

core/services/nodes/probe_cache_test.go

@@ -11,6 +11,21 @@ import (
 	"golang.org/x/sync/singleflight"
 )
 
+// doOrCached drives the production entry point with a boolean-only probe,
+// which is what most of these specs are about.
+//
+// It is a spec helper and not a method, deliberately. It WAS a method, and once
+// probeHealth moved to DoOrCachedResult it became production code with no
+// production caller, kept green by these specs alone. Moving it here keeps the
+// convenience where its only user is and stops the shim being mistaken for a
+// supported way to probe.
+func doOrCached(c *probeCache, key string, probe func() bool) bool {
+	GinkgoHelper()
+	alive, unreached := c.DoOrCachedResult(key, func() (bool, error) { return probe(), nil })
+	Expect(unreached).To(BeNil())
+	return alive
+}
+
 var _ = Describe("probeCache", func() {
 	It("invokes the probe on a cold cache and caches success", func() {
 		c := newProbeCache(time.Minute)
@@ -20,9 +35,9 @@ var _ = Describe("probeCache", func() {
 			return true
 		}
 
-		Expect(c.DoOrCached("k", probe)).To(BeTrue())
-		Expect(c.DoOrCached("k", probe)).To(BeTrue())
-		Expect(c.DoOrCached("k", probe)).To(BeTrue())
+		Expect(doOrCached(c, "k", probe)).To(BeTrue())
+		Expect(doOrCached(c, "k", probe)).To(BeTrue())
+		Expect(doOrCached(c, "k", probe)).To(BeTrue())
 
 		// Cached: probe ran once.
 		Expect(atomic.LoadInt32(&calls)).To(Equal(int32(1)))
@@ -38,9 +53,9 @@ var _ = Describe("probeCache", func() {
 			return true
 		}
 
-		Expect(c.DoOrCached("k", probe)).To(BeTrue())
+		Expect(doOrCached(c, "k", probe)).To(BeTrue())
 		time.Sleep(5 * time.Millisecond)
-		Expect(c.DoOrCached("k", probe)).To(BeTrue())
+		Expect(doOrCached(c, "k", probe)).To(BeTrue())
 
 		Expect(atomic.LoadInt32(&calls)).To(Equal(int32(2)))
 	})
@@ -56,16 +71,16 @@ var _ = Describe("probeCache", func() {
 
 		// First probe fails — must NOT be cached.
 		result.Store(false)
-		Expect(c.DoOrCached("k", probe)).To(BeFalse())
+		Expect(doOrCached(c, "k", probe)).To(BeFalse())
 		Expect(c.IsFresh("k")).To(BeFalse())
 
 		// Recover: second probe succeeds and is cached.
 		result.Store(true)
-		Expect(c.DoOrCached("k", probe)).To(BeTrue())
+		Expect(doOrCached(c, "k", probe)).To(BeTrue())
 		Expect(c.IsFresh("k")).To(BeTrue())
 
 		// Third call short-circuits on the fresh entry.
-		Expect(c.DoOrCached("k", probe)).To(BeTrue())
+		Expect(doOrCached(c, "k", probe)).To(BeTrue())
 		Expect(atomic.LoadInt32(&calls)).To(Equal(int32(2)))
 	})
 
@@ -92,7 +107,7 @@ var _ = Describe("probeCache", func() {
 			go func(i int) {
 				defer wg.Done()
 				<-start
-				results[i] = c.DoOrCached("k", probe)
+				results[i] = doOrCached(c, "k", probe)
 			}(i)
 		}
 
@@ -187,9 +202,9 @@ var _ = Describe("probeCache", func() {
 	It("treats different keys independently", func() {
 		c := newProbeCache(time.Minute)
 		var aCalls, bCalls int32
-		Expect(c.DoOrCached("a", func() bool { atomic.AddInt32(&aCalls, 1); return true })).To(BeTrue())
-		Expect(c.DoOrCached("b", func() bool { atomic.AddInt32(&bCalls, 1); return true })).To(BeTrue())
-		Expect(c.DoOrCached("a", func() bool { atomic.AddInt32(&aCalls, 1); return true })).To(BeTrue())
+		Expect(doOrCached(c, "a", func() bool { atomic.AddInt32(&aCalls, 1); return true })).To(BeTrue())
+		Expect(doOrCached(c, "b", func() bool { atomic.AddInt32(&bCalls, 1); return true })).To(BeTrue())
+		Expect(doOrCached(c, "a", func() bool { atomic.AddInt32(&aCalls, 1); return true })).To(BeTrue())
 
 		Expect(atomic.LoadInt32(&aCalls)).To(Equal(int32(1)))
 		Expect(atomic.LoadInt32(&bCalls)).To(Equal(int32(1)))
@@ -203,9 +218,9 @@ var _ = Describe("probeCache", func() {
 			return true
 		}
 
-		Expect(c.DoOrCached("k", probe)).To(BeTrue())
-		Expect(c.DoOrCached("k", probe)).To(BeTrue())
-		Expect(c.DoOrCached("k", probe)).To(BeTrue())
+		Expect(doOrCached(c, "k", probe)).To(BeTrue())
+		Expect(doOrCached(c, "k", probe)).To(BeTrue())
+		Expect(doOrCached(c, "k", probe)).To(BeTrue())
 
 		Expect(atomic.LoadInt32(&calls)).To(Equal(int32(3)))
 	})
@@ -217,9 +232,9 @@ var _ = Describe("probeCache", func() {
 			atomic.AddInt32(&calls, 1)
 			return true
 		}
-		Expect(c.DoOrCached("k", probe)).To(BeTrue())
+		Expect(doOrCached(c, "k", probe)).To(BeTrue())
 		c.Invalidate("k")
-		Expect(c.DoOrCached("k", probe)).To(BeTrue())
+		Expect(doOrCached(c, "k", probe)).To(BeTrue())
 		Expect(atomic.LoadInt32(&calls)).To(Equal(int32(2)))
 	})
 })

core/services/nodes/reconciler.go

@@ -614,8 +614,14 @@ func (rc *ReplicaReconciler) sweepLeakedInFlight(ctx context.Context) {
 		}
 		seen[m.ID] = struct{}{}
 		if rc.prober.Probe(ctx, m.NodeID, m.WorkerLocalAddress) != ProbeAlive {
-			// Busy or unreachable. Busy means the counter may well be real;
-			// unreachable is the reaper's business, not the sweeper's.
+			// Anything but alive, and the three of them agree on what this
+			// sweeper should do even though they disagree about everything
+			// else. Busy: the counter may well be real, so leave it.
+			// Unreachable: the row is the reaper's business, not the
+			// sweeper's. Unknown: this frontend has no route and therefore
+			// observed nothing, which is the one outcome that must never be
+			// read as evidence. Resetting a counter on any of the three would
+			// free a reservation a live request is still holding.
 			rc.clearInFlightIdle(m.ID)
 			continue
 		}

core/services/nodes/reconciler_prober_test.go

@@ -3,18 +3,42 @@
 package nodes
 
 import (
+	"bytes"
 	"context"
 	"errors"
 	"fmt"
+	"io"
 	"net"
 
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
+	"google.golang.org/grpc/codes"
+	"google.golang.org/grpc/status"
 
 	"github.com/mudler/LocalAI/core/services/cluster"
 	grpc "github.com/mudler/LocalAI/pkg/grpc"
 )
 
+// refusalFromWorker builds the error a frontend actually holds after a worker
+// refused one of its streams.
+//
+// It goes over the WIRE rather than being handed the sentinel directly: the
+// refusal is written with the worker's own writer and read back with the
+// frontend's own reader, so a reason the protocol cannot carry, or a code
+// mapping that stopped round-tripping, reddens these specs instead of leaving
+// them asserting against a value production never produces. The final wrap is
+// the shape WorkerDialer.handshake returns for a worker answer, umbrella and
+// all: there is none, which is the property under test.
+func refusalFromWorker(reason error) error {
+	GinkgoHelper()
+	var frame bytes.Buffer
+	Expect(cluster.WriteStreamRefusal(&frame, reason)).To(Succeed())
+	readBack := cluster.ReadStreamReply(&frame)
+	Expect(readBack).To(MatchError(reason), "the refusal must survive its own round trip")
+	Expect(readBack).ToNot(MatchError(cluster.ErrNoRoute))
+	return fmt.Errorf("opening %q on node %q: %w", "grpc", "node-1", readBack)
+}
+
 // proberFactory hands the prober one client, and records what it was asked for.
 type proberFactory struct {
 	client grpc.Backend
@@ -65,13 +89,67 @@ var _ = Describe("the reconciler's gRPC model prober", func() {
 		Expect(probe(&proberFactory{client: &fakeBackendClient{healthy: true}})).To(Equal(ProbeAlive))
 	})
 
-	It("still answers ProbeUnreachable for a dead backend on a worker it reached", func() {
-		// The other direction. The new check must not turn the reaper off: a
-		// backend that answered "unhealthy" over a working transport is a ghost
-		// and its row should go.
+	It("still answers ProbeUnreachable for a backend that answered unhealthy", func() {
+		// One of the two shapes a dead backend takes, and the easy one: the
+		// process is up enough to answer and reports itself unhealthy over a
+		// working transport. It is a ghost and its row should go.
+		//
+		// This spec used to be named for the property the table below holds,
+		// which it never tested: a backend process that DIED on a tunnelled
+		// worker does not answer at all, and what the frontend gets back is the
+		// worker's refusal, not an unhealthy reply.
 		Expect(probe(&proberFactory{client: &fakeBackendClient{healthy: false}})).To(Equal(ProbeUnreachable))
 	})
 
+	DescribeTable("answers ProbeUnreachable when the WORKER ITSELF refused the stream",
+		// The dominant shape of a dead backend since workers stopped listening,
+		// and the one that produced a permanently unreapable row. The worker is
+		// healthy, connected and answering; what it answers is that the stream
+		// cannot be served. That is evidence about the backend, so the reaper
+		// may act on it. Reported as ProbeUnknown instead, no reap path deleted
+		// the row, the replica slot never freed, and at the default
+		// MaxReplicasPerModel=1 the only remaining cleanup was LRU eviction of
+		// models that were working.
+		func(reason error) {
+			Expect(probe(&proberFactory{client: &fakeBackendClient{
+				healthy: false,
+				err:     status.Error(codes.Unavailable, "connection error: transport"),
+				dialErr: refusalFromWorker(reason),
+			}})).To(Equal(ProbeUnreachable))
+		},
+		Entry("the worker could not reach the backend process", cluster.ErrStreamTargetUnavailable),
+		Entry("the worker does not serve gRPC streams at all", cluster.ErrStreamTagUnknown),
+		Entry("the worker rejected the stored address", cluster.ErrStreamRequestInvalid),
+	)
+
+	It("answers ProbeUnknown for a refusal code this frontend does not recognise", func() {
+		// The other direction, and the boundary of the exemption above. A
+		// newer worker's vocabulary must not be read as evidence about a
+		// backend: ReadStreamReply returns an unrecognised code as a plain
+		// error, WorkerDialer puts the no-route umbrella on it, and the row
+		// survives. Guessing wrong here costs a retry; guessing wrong the other
+		// way costs a reaped replica.
+		unknownCode := fmt.Errorf("reaching node %q: %w: opening %q: tunnel stream refused with unrecognised code %q: %s",
+			"node-1", cluster.ErrNoRoute, "grpc", "quiesced", "this worker is draining")
+		Expect(probe(&proberFactory{client: &fakeBackendClient{
+			healthy: false,
+			err:     status.Error(codes.Unavailable, "connection error: transport"),
+			dialErr: unknownCode,
+		}})).To(Equal(ProbeUnknown))
+	})
+
+	It("answers ProbeUnknown when the tunnel broke while reading the worker's reply", func() {
+		// The condition a refusal is most easily confused with, kept apart on
+		// purpose: a read failure is the tunnel breaking, not the worker
+		// speaking, and it says nothing about the backend.
+		Expect(probe(&proberFactory{client: &fakeBackendClient{
+			healthy: false,
+			err:     status.Error(codes.Unavailable, "connection error: transport"),
+			dialErr: fmt.Errorf("reaching node %q: %w: opening %q: reading a tunnel stream reply: %w",
+				"node-1", cluster.ErrNoRoute, "grpc", io.ErrUnexpectedEOF),
+		}})).To(Equal(ProbeUnknown))
+	})
+
 	It("does not report a transport that recovered", func() {
 		// LastDialError is cleared by a successful dial, so a client that
 		// failed once and then reconnected must not keep reading as

core/services/nodes/registry.go

@@ -47,6 +47,14 @@ type BackendNode struct {
 	// plaintext once, and stores only this hash, so a leaked registration token
 	// no longer opens a tunnel for every node whose ID an attacker can read.
 	//
+	// Stated exactly, because the useful half of the claim is the half that is
+	// still true. A leaked registration token no longer lets its holder BE a
+	// worker; it still lets its holder REACH every worker, because
+	// GET /api/cluster/peer authenticates with the shared cluster token and
+	// takes its ?id= on trust (see core/http/endpoints/cluster/peer.go). Per
+	// replica-to-replica credentials are a named phase-3 item, not something
+	// this column already delivers.
+	//
 	// Empty means no tunnel credential has been minted for this node yet, which
 	// is what a node registered by an older LocalAI looks like. Such a node
 	// cannot tunnel until it registers again. That is deliberate: the column

core/services/nodes/router.go

@@ -1398,13 +1398,13 @@ func (r *SmartRouter) installBackendOnNode(ctx context.Context, node *BackendNod
 		//
 		// An earlier version of this comment justified it by saying the worker
 		// would refuse the resulting empty target as an invalid stream and that
-		// the refusal would read as the worker answering about its backend. The
-		// first half is true (see cluster.isWorkerAnswer) and the second is
-		// not: nothing in this package branches on cluster.ErrNoRoute, and
-		// `unroutable` treats ANY recorded dial error as unroutable, so such a
-		// refusal reaches every reap guard as ProbeUnknown and deletes nothing.
-		// The decision stands on the grounds above, which do not depend on a
-		// classification the frontend does not currently make.
+		// the refusal would read as the worker answering about its backend.
+		// Both halves are true NOW (see cluster.IsWorkerAnswer and
+		// `unroutable`), and the decision still does not rest on either: a row
+		// written with an empty address would be reaped a probe cycle later
+		// with its cause a hop away from where it was created, and the failure
+		// belongs to this install. Reaping is a recovery, not a substitute for
+		// refusing to write the bad value.
 		if reply.WorkerLocalAddress == "" {
 			return "", fmt.Errorf("worker %s reported backend %q installed but named no address for the process", node.ID, backendType)
 		}
@@ -1995,11 +1995,14 @@ func (r *SmartRouter) stageOptionDir(ctx context.Context, node *BackendNode, dir
 // dials lazily on its first call.
 //
 // The client is the RAW factory client rather than buildClientForAddr's, on
-// purpose. A health check stages no files, so the staging wrapper buys nothing
-// here; and the wrapper hides the transport, because it embeds grpc.Backend and
-// so does not carry LastDialError through. Wrapping would leave this function
-// unable to tell a dead backend from an unreachable worker, which is the whole
-// question it now answers.
+// purpose, and the reason is narrower than it used to be. The staging wrapper
+// no longer hides the transport: since it became a grpc.WrappedBackend it
+// carries LastDialError through, so wrapping would not cost this function the
+// answer it needs. What it buys is nothing at all, because a health check
+// stages no files, and an unused wrapper on the hottest path in the router is
+// an allocation and an indirection per probe. The earlier justification
+// ("the wrapper does not carry LastDialError through") is no longer true and is
+// recorded here so nobody re-derives the decision from it.
 func (r *SmartRouter) probeHealth(ctx context.Context, node *BackendNode, addr string) (alive, probed bool) {
 	client, err := r.clientFactory.NewClientForNode(node.ID, addr, false)
 	if err != nil {

docs/content/features/distributed-mode.md

@@ -85,14 +85,25 @@ Frontend replicas record themselves in an `instances` table and open direct link
 
 When `LOCALAI_DISTRIBUTED_ADVERTISE_ADDR` is unset, the address is derived: LocalAI asks the kernel which local address routes to PostgreSQL, and pairs it with the port it serves on. Every replica reaches the same database, so that address is on a network they demonstrably share.
 
-That only holds while the database is on **another host**. If PostgreSQL runs on the same host or pod (compose, single-node, a sidecar), the route to it is loopback, and advertising a loopback address would send every peer to itself. LocalAI refuses to guess in that case and logs:
+That only holds while the database is on **another host**. If PostgreSQL runs on the same host or pod (compose, single-node, a sidecar), the route to it is loopback, and advertising a loopback address would send every peer to itself. LocalAI refuses to guess in that case. It starts anyway - refusing would break every single-host deployment, which has no peers to be unreachable by - and logs an error at startup:
 
 ```
-This replica will not be reachable by its peers: no advertised address
+ERROR This replica is not registered in the cluster: no advertised address. Peers cannot reach it,
+      and any worker whose tunnel lands here will be unroutable from every other replica
 ```
 
 The replica keeps serving every request that reaches it directly. What it cannot do is be reached by another replica, and on a multi-replica deployment that is worse than it sounds: a **worker whose tunnel lands on this replica is unroutable from every other replica**, because the ownership lookup only accepts an owner that is registered and live. This replica serves that worker fine; the others answer requests for it with `no route from this replica to that worker`. Behind a round-robin load balancer with N replicas, that is (N-1)/N of the traffic for that worker.
 
+Because that symptom looks like a broken **worker** and not a misconfigured **frontend**, the replica repeats itself every five minutes for as long as it runs, and names the workers it is currently costing:
+
+```
+ERROR This replica is not registered in the cluster and holds worker tunnels: those workers are
+      unroutable from every OTHER replica, and requests for their models fail there with no route.
+      The workers are healthy; this replica is invisible   workers=[node-a node-b] worker_count=2
+```
+
+If you are chasing a worker that answers on one replica and 5xxs on the others, grep the frontend logs for that line before looking at the worker. Until a worker's tunnel lands here the same line appears at `WARN` with no workers named, which is the same misconfiguration not yet costing anything.
+
 A single-replica deployment is unaffected: it has no peers, and it holds every tunnel itself. Set the address explicitly to fix a multi-replica one:
 
 ```yaml
@@ -102,6 +113,10 @@ environment:
 
 The peer link is served at `/api/cluster/peer` and authenticates with `LOCALAI_REGISTRATION_TOKEN`, the same shared secret workers register with. Replicas that disagree about it cannot link. A replica that stops heartbeating for 30 seconds is dropped from the table by the others, along with the worker-connection rows it owned.
 
+{{% notice note %}}
+**The peer link has no per-replica credential yet.** It checks the shared registration token and takes the replica id in `?id=` on trust. Anything already holding that token - every worker holds it - can therefore open a peer link, relay through it to every worker tunnel a replica owns, and by declaring another replica's id displace that replica's inbound link. Treat `LOCALAI_REGISTRATION_TOKEN` as a cluster-wide secret with the blast radius of the whole fleet: give it its own value per deployment, do not reuse it elsewhere, and keep `/api/cluster/peer` on a network only your replicas and workers can reach. Per-replica credentials for this route are planned.
+{{% /notice %}}
+
 ### Worker tunnels
 
 A worker can open one long-lived, multiplexed tunnel to the frontend instead of listening on a port of its own. It dials `GET /api/cluster/connect?id=<node id>`, the connection is upgraded to a WebSocket, and every subsequent request the frontend makes to that worker travels as a stream inside it. Nothing dials *into* the worker, so a worker behind NAT, in another Kubernetes cluster or on a laptop needs no inbound port and no reachable address.
@@ -173,22 +188,32 @@ These outcomes are kept apart on purpose, because they call for different action
 | Not the owner | The routing was stale | Resolve the owner again |
 | Peer unreachable | A replica exists and will not answer | Retry |
 | No relay path | This replica cannot reach the owner at all | Report; requests here fail until it can |
-| The worker refused | The worker answered and said no | Report; the worker is connected |
+| The worker refused | The worker answered and said no | Report, and **act on it**; the worker is connected and speaking about its own backend |
 
-**None of them is absence.** A worker's presence is its **heartbeat**, and a route to it is a separate fact that can be false while the worker is registered, heartbeating and serving every request another replica sends it. So the frontend answers "no route", never "this worker is gone", and nothing on this list causes a model to be rescheduled or a `node_models` row to be deleted.
+**None of the first four is absence.** A worker's presence is its **heartbeat**, and a route to it is a separate fact that can be false while the worker is registered, heartbeating and serving every request another replica sends it. So the frontend answers "no route", never "this worker is gone", and none of the first four causes a model to be rescheduled or a `node_models` row to be deleted.
+
+The fifth is different, and deliberately so. A worker that **refuses** a stream has answered, which proves it is connected; what it is refusing is the stream to one backend process on it. That is the ordinary shape of a crashed backend now that workers listen on nothing: the worker's own dial to the process fails and it says so. The frontend treats that as evidence about the backend, so the model's row is reaped and the replica is reloaded, exactly as a dead local backend would be. Without that, a crashed backend on a healthy worker would leave a row nothing could ever delete, its replica slot permanently occupied. A refusal code the frontend does not recognise - a newer worker's vocabulary - is treated as "no route" instead, so a version skew costs a retry rather than a reaped replica.
 
 That distinction is the whole point rather than a nicety. A scheduler told that a connected worker has gone away stops its backend and reclaims every model it is running, and the events that produce "no route" are ordinary ones: a frontend replica restarting, an ownership row a moment stale, a worker that has not dialled its tunnel yet. A worker is treated as absent only when its **heartbeat** goes stale, which is a separate mechanism with its own threshold (see `--stale-node-threshold`).
 
-#### There is no frontend-side fallback, and upgrade order matters
+#### There is no frontend-side fallback
 
-`LOCALAI_WORKER_TUNNEL=false` still stops a worker dialling its tunnel, but it no longer has a frontend counterpart: **no frontend path dials a worker's advertised address**, and a worker on this release advertises none and listens on no routable interface. A worker with the tunnel off is a worker nothing can reach. Setting it is not a rollback. The rollback is to run the previous release on both sides.
+`LOCALAI_WORKER_TUNNEL=false` is a **fatal startup error** on this release. It is not a degraded mode and not a rollback switch: the worker refuses to boot and prints why. Nothing else would be honest, because the setting stops the worker dialling its tunnel while **no frontend path dials a worker's advertised address**, and a worker on this release advertises none and listens on no routable interface, so a worker that started with it off would register, heartbeat, be scheduled onto, and fail every request. The rollback is to run the previous release on both sides.
 
-That makes upgrade order matter, in one direction only:
+#### Upgrade the frontends first
 
-- **Upgrade the workers first, then the frontends.** A worker on this build dials its tunnel, and a frontend of either version reaches it through that. An old frontend that would have dialled its advertised address no longer gets one, so this order is what keeps the fleet routable throughout.
-- **Upgrading the frontends first** leaves every not-yet-restarted worker unroutable until it restarts. Those workers keep running their models and keep heartbeating, and the frontend reports them as unroutable rather than as gone: their `node_models` rows are left alone, nothing is rescheduled, and requests for those models fail loudly with "no route" until the worker reconnects. It is a degraded window, not an eviction, but it is a window, and doing it the other way round has none.
+**Upgrade every frontend replica, then restart the workers one at a time.**
 
-A worker that cannot reach its frontend retries with exponential backoff and never gives up, so restarting a worker is all that is needed to close the window.
+- **Frontends first (correct).** Old workers keep running, keep heartbeating and keep their `node_models` rows: the new frontend reports them as unroutable rather than as gone, so nothing is rescheduled and nothing is reaped. What fails is requests for models on a worker that has not been restarted yet. That is a real degraded window, but it is bounded by how fast you roll the workers, it heals itself as each one comes back, and no state is lost.
+  - **What you will see while it lasts:** requests for models on a not-yet-restarted worker fail with "no route to the worker", while `GET /api/nodes` still shows that node healthy and heartbeating and its models still listed. Restart the worker and it clears. Nothing needs fixing; you are watching the window close.
+- **Workers first (this fails, do not do it).** An old frontend has no `/api/cluster/connect` route for the worker to dial *and* rejects the new worker's registration outright, because the worker no longer sends an address and the old frontend requires one. A 4xx is a verdict rather than an outage, so the worker reports the reason on the **first** attempt and exits instead of retrying. Every worker you restart is a worker you take out of the fleet until the frontends are upgraded.
+  - **What you will see if you do it anyway:** each restarted worker exits within a second or two of starting, with `registration failed with status 400: address is required for backend workers: the frontend refused this registration`. The fleet drains one node per restart, and the nodes that are left are the ones you have not touched yet.
+
+A worker that cannot reach its frontend *at the network level* retries with exponential backoff and never gives up, so restarting a worker is all that is needed to close the frontend-first window. A worker whose registration is **rejected** does not retry, which is what makes the wrong order destructive rather than slow.
+
+##### Rolling a frontend back requires restarting every worker
+
+Registering against an upgraded frontend **clears** a node's `address` and `http_address` columns in the shared database, and re-registration is the only thing that ever writes them back. So a partial rollback does not restore the previous behaviour on its own: the old frontend code reads an empty address for every node that has registered since the upgrade and dials nothing. Roll the frontends back *and then restart every worker* so each one re-registers and repopulates its address. Rolling back is not a frontend-only operation.
 
 #### Workers bind nothing routable
 
@@ -423,7 +448,7 @@ local-ai worker \
 | `--registration-require-auth` | `LOCALAI_REGISTRATION_REQUIRE_AUTH` | `false` | Refuse to start the HTTP file-transfer server when no registration token is set (it would otherwise fail open) |
 | `--distributed-require-auth` | `LOCALAI_DISTRIBUTED_REQUIRE_AUTH` | `false` | Umbrella switch implying both `--registration-require-auth` and `--nats-require-auth` |
 | `--heartbeat-interval` | `LOCALAI_HEARTBEAT_INTERVAL` | `10s` | Interval between heartbeat pings |
-| `--worker-tunnel` | `LOCALAI_WORKER_TUNNEL` | `true` | Hold one outbound multiplexed tunnel to the frontend and serve its requests over it, so this worker needs no inbound port (see [Worker tunnels](#worker-tunnels)). Turning it off makes the worker unreachable: the frontend has no path that dials a worker's advertised address. |
+| `--worker-tunnel` | `LOCALAI_WORKER_TUNNEL` | `true` | Hold one outbound multiplexed tunnel to the frontend and serve its requests over it, so this worker needs no inbound port (see [Worker tunnels](#worker-tunnels)). Setting it to `false` is a **fatal startup error**, not a degraded mode: the frontend has no path that dials a worker's advertised address, so a worker without its tunnel is a worker nothing can reach. To run without tunnels, run the pre-tunnel release on both the worker and the frontend. |
 | `--nats-url` | `LOCALAI_NATS_URL` | *(required)* | NATS URL for backend installation and file staging |
 | `--nats-jwt` | `LOCALAI_NATS_JWT` | *(empty)* | Optional override for the `nats_jwt` returned at registration |
 | `--nats-user-seed` | `LOCALAI_NATS_USER_SEED` | *(empty)* | Optional override for `nats_user_seed` from registration |

pkg/model/connection_evicting_client.go

@@ -45,7 +45,13 @@ func (c *ConnectionEvictingClient) checkErr(err error) {
 	// stops a model that is loaded and serving, on a worker that is
 	// heartbeating. A locally spawned backend has no custom transport, so this
 	// reports nil and the behaviour there is exactly what it always was.
-	if dialErr := grpc.LastDialErrorOf(c.Backend); dialErr != nil {
+	// transportFailure and not LastDialErrorOf: a refusal the WORKER wrote is
+	// the worker answering that it could not reach the process, which is what a
+	// crashed backend produces now that a worker listens on nothing. Treating
+	// that as a transport failure kept a genuinely dead model loaded and
+	// failing every request, which is the mirror image of the mistake this
+	// guard exists to prevent.
+	if dialErr := transportFailure(c.Backend); dialErr != nil {
 		xlog.Warn("Inference failed because the worker could not be reached; keeping the model",
 			"model", c.modelID, "error", dialErr)
 		return

pkg/model/loader.go

@@ -12,6 +12,7 @@ import (
 	"sync/atomic"
 	"time"
 
+	"github.com/mudler/LocalAI/core/services/cluster"
 	grpc "github.com/mudler/LocalAI/pkg/grpc"
 	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
 	"github.com/mudler/LocalAI/pkg/system"
@@ -746,5 +747,20 @@ func transportFailure(client grpc.Backend) error {
 	// over a *FileStagingClient, and an assertion on the outermost type reads
 	// nil for both: they embed grpc.Backend, which does not declare
 	// LastDialError. That is exactly how this guard shipped inert.
-	return grpc.LastDialErrorOf(client)
+	dialErr := grpc.LastDialErrorOf(client)
+	if dialErr == nil {
+		return nil
+	}
+	// A refusal WRITTEN BY THE WORKER is not a transport failure, however much
+	// it looks like one from here: the tunnel carried the request, the worker
+	// read it and answered that it could not reach the process the stream
+	// named. That is the ordinary shape of a crashed backend now that a worker
+	// listens on nothing, and reporting it as "could not reach the worker"
+	// pinned the model in this cache forever. cluster.Dial keeps these three
+	// out of its no-route umbrella for exactly this question; see
+	// cluster.IsWorkerAnswer.
+	if cluster.IsWorkerAnswer(dialErr) {
+		return nil
+	}
+	return dialErr
 }

pkg/model/remote_unroutable_internal_test.go

@@ -3,18 +3,39 @@
 package model
 
 import (
+	"bytes"
 	"context"
 	"errors"
+	"fmt"
 	"net"
 
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
 
+	"github.com/mudler/LocalAI/core/services/cluster"
 	grpc "github.com/mudler/LocalAI/pkg/grpc"
 	pb "github.com/mudler/LocalAI/pkg/grpc/proto"
 	"github.com/mudler/LocalAI/pkg/system"
 )
 
+// refusalFromWorker builds the error the frontend's dialler really returns when
+// a worker refuses one of its streams, by writing the refusal with the worker's
+// own writer and reading it back with the frontend's own reader.
+//
+// It matters that this goes over the wire rather than taking the sentinel
+// directly. A worker that answers is CONNECTED, so its refusal is not a
+// transport failure however much it looks like one from here, and a spec
+// asserting against a hand-made value would not notice if the wire stopped
+// carrying the distinction.
+func refusalFromWorker(reason error) error {
+	GinkgoHelper()
+	var frame bytes.Buffer
+	Expect(cluster.WriteStreamRefusal(&frame, reason)).To(Succeed())
+	readBack := cluster.ReadStreamReply(&frame)
+	Expect(readBack).To(MatchError(reason))
+	return fmt.Errorf("opening %q on node %q: %w", "grpc", "node-1", readBack)
+}
+
 var _ = Describe("the health check on a remote model whose transport failed", func() {
 	// The fourth site of the same shape as the reconciler, the health monitor
 	// and the router, found by sweeping rather than by being named.
@@ -47,6 +68,24 @@ var _ = Describe("the health check on a remote model whose transport failed", fu
 		Expect(stillThere).To(BeTrue())
 	})
 
+	It("evicts a remote model whose backend the WORKER ITSELF could not reach", func() {
+		// The shape a crashed backend takes since workers stopped listening:
+		// the tunnel carried the request, the worker read it and answered that
+		// nothing is listening on that port. That is the worker speaking about
+		// its backend, not a transport failure, and reading it as one pinned a
+		// genuinely dead model in this cache forever, failing every request.
+		client := grpc.NewClientWithDialer("10.0.0.1:9001", false, nil, false, "",
+			func(context.Context, string) (net.Conn, error) {
+				return nil, refusalFromWorker(cluster.ErrStreamTargetUnavailable)
+			})
+		m := NewModelWithClient("refused-model", "10.0.0.1:9001", client)
+		ml.store.Set("refused-model", m)
+
+		Expect(ml.checkIsLoaded("refused-model")).To(BeNil())
+		_, stillThere := ml.store.Get("refused-model")
+		Expect(stillThere).To(BeFalse())
+	})
+
 	It("still evicts a remote model whose worker WAS reached and did not answer", func() {
 		// The other direction, so the new check cannot pass by never evicting.
 		// No custom dialer, so the transport reports nothing and a connection
@@ -85,6 +124,36 @@ var _ = Describe("the eviction wrapper on a remote model whose transport failed"
 			"a worker this frontend cannot route to must not have its backend stopped and its rows deleted")
 	})
 
+	It("evicts when the WORKER ITSELF refused the stream to the backend", func() {
+		// The same rule on the INFERENCE path. A refusal the worker wrote is
+		// the worker reporting its backend gone, so the model must be evicted
+		// here exactly as a locally spawned one would be; the guard is for a
+		// route this frontend lost, which is a different condition.
+		evicted := 0
+		client := newConnectionEvictingClient(
+			failingDial(refusalFromWorker(cluster.ErrStreamTargetUnavailable)),
+			"refused-model", func() { evicted++ })
+
+		_, err := client.Predict(context.Background(), &pb.PredictOptions{})
+		Expect(err).To(HaveOccurred())
+		Expect(evicted).To(Equal(1))
+	})
+
+	It("does not evict on a refusal code this frontend does not recognise", func() {
+		// The boundary. An unrecognised code is a newer worker's vocabulary,
+		// which WorkerDialer reports under the no-route umbrella, and acting on
+		// it would let a worker upgrade stop models that are running.
+		evicted := 0
+		client := newConnectionEvictingClient(
+			failingDial(fmt.Errorf("reaching node %q: %w: opening %q: tunnel stream refused with unrecognised code %q: %s",
+				"node-1", cluster.ErrNoRoute, "grpc", "quiesced", "this worker is draining")),
+			"remote-model", func() { evicted++ })
+
+		_, err := client.Predict(context.Background(), &pb.PredictOptions{})
+		Expect(err).To(HaveOccurred())
+		Expect(evicted).To(BeZero())
+	})
+
 	It("still evicts when the worker WAS reached and the connection failed", func() {
 		// The other direction. No custom dialer, so nothing reports a transport
 		// failure and a connection error means what it always meant.