mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-22 06:04:55 -04:00
Ettore Di Giacinto
705646c1a5
Removing "Presence: clusterRegistry" from the options literal in initDistributed left all seven suites and tests/e2e/distributed green. The predicate was right and its input was silently nil, which returns the deployment to absence being decided by nothing, with no log line and no failing request. That is the fourth finding of this exact shape in this phase. The two assignments move out of a twenty-field literal into distributedSchedulerOptions, a named function a unit spec can reach. Deleting either is now red. The health monitor takes its presence reader and grace as a required positional pair instead, so deleting those does not compile at all. requireAbsenceWiring then refuses to start a distributed frontend whose scheduler or health monitor has no source of absence, because refusing to boot is the only symptom either failure has. With a fresh heartbeat and a permanently gone tunnel there was no reaper at all. A heartbeat says the worker's supervisor is alive; it says nothing about whether anything here can reach that worker's backends, because those are reached over the tunnel. A proxy that stops upgrading WebSockets, a rotated registration credential or a reconnect loop longer than the grace left a node listed healthy forever while every request for a model already loaded on it failed "no route to that worker", and every reaper keyed on the heartbeat. The health monitor now reads presence from the same place and against the same window as the scheduler and demotes such a node. That also ends the 15s re-promotion: the demotion arm returns before the recovery arm, so the scheduler's demotion is no longer undone on the next tick, and recovery needs the tunnel back rather than just the heartbeat. The demotion is status-only. MarkOffline would DELETE the node's rows, and deleting rows on a presence read would give any future defect in that read the widest blast radius in the system for nothing the demotion does not already deliver. LRU eviction is the third path that commits work to a node, and it read only the stored status. A node full enough to be an eviction target is exactly the node the VRAM and idle selectors never offer, so pickReachableNode structurally cannot cover it. It now runs its chosen node through the same nodeMayTakeWork predicate, demotes it and evicts again rather than handing back an install that cannot land. Presence is read after the transaction and not inside it: reading it inside would hold a FOR UPDATE lock across a query needing a second pooled connection, which is how concurrent evictions deadlock a pool. Also: a router built with a presence reader and no grace now has its documented default pinned by a spec rather than only claimed by a comment; ageDeparture asserts RowsAffected, since an UPDATE matching nothing succeeds and the inside-the-grace spec returned the same verdict either way; the scheduler comment that still described the bus is corrected; the docs stop conflating heartbeat recovery with tunnel recovery and name the third reader; and an overlong rewrapped line in membership.go is folded. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
66 lines
3.3 KiB
Go
66 lines
3.3 KiB
Go
// SPDX-License-Identifier: MIT
|
|
|
|
package application
|
|
|
|
import (
|
|
"fmt"
|
|
|
|
"github.com/mudler/LocalAI/core/config"
|
|
"github.com/mudler/LocalAI/core/services/nodes"
|
|
)
|
|
|
|
// distributedSchedulerOptions stamps the absence wiring onto the scheduler's
|
|
// options and returns them.
|
|
//
|
|
// Two assignments in a named function rather than two more fields in the
|
|
// twenty-field literal they used to live in. The literal cannot be reached by a
|
|
// unit spec, because the function that builds it also opens a NATS connection
|
|
// and a database; these two lines can, and they are the two lines this whole
|
|
// change comes down to. Losing them in the literal was silent and green.
|
|
//
|
|
// The grace comes from the same expression the membership loop is given
|
|
// (Membership.SetReconnectGrace), so the window a departure is measured against
|
|
// and the window a departure is RETAINED for cannot drift apart.
|
|
func distributedSchedulerOptions(cfg config.DistributedConfig, presence nodes.NodePresenceReader, opts nodes.SmartRouterOptions) nodes.SmartRouterOptions {
|
|
opts.Presence = presence
|
|
opts.ReconnectGrace = cfg.ReconnectGraceOrDefault()
|
|
return opts
|
|
}
|
|
|
|
// requireAbsenceWiring refuses to start a distributed deployment in which
|
|
// nothing can decide that a worker has gone away.
|
|
//
|
|
// Two components read absence, from one source and against one window: the
|
|
// scheduler, which stops placing work on a departed worker, and the health
|
|
// monitor, which stops reporting one as healthy. Each reads it through a field
|
|
// assigned in a large construction literal in initDistributed.
|
|
//
|
|
// It is checked rather than assumed because losing either assignment is
|
|
// SILENT. A scheduler with no absence source places work on workers that are
|
|
// gone and demotes none; a health monitor with none reports a worker whose
|
|
// tunnel died an hour ago as healthy, forever, with every request for a model
|
|
// loaded on it failing "no route to that worker". Neither logs anything,
|
|
// neither fails a request that would not have failed anyway, and both look
|
|
// exactly like a fleet that is fine. Refusing to boot is the only symptom
|
|
// either failure has, and it is the reason this is a startup error and not a
|
|
// warning: a deployment that came up and quietly decided absence by nothing is
|
|
// the state the tunnel work exists to remove.
|
|
//
|
|
// What this guard itself rests on, stated because it is a real limit: the two
|
|
// helper functions below and above are pinned by unit specs, but the CALL to
|
|
// this one lives in initDistributed, which opens NATS and a database and so has
|
|
// no unit spec at all. Deleting the call, or writing a literal nil where
|
|
// initDistributed passes the cluster registry, compiles and leaves every suite
|
|
// in this repository green. Only tests/e2e/distributed/cluster catches it, by
|
|
// booting the real binary: the error is returned from initDistributed and
|
|
// aborts application startup, so a frontend so wired never comes up.
|
|
func requireAbsenceWiring(router *nodes.SmartRouter, health *nodes.HealthMonitor) error {
|
|
if !router.ReadsAbsence() {
|
|
return fmt.Errorf("the distributed scheduler was built with no source of worker absence: it would place work on workers that have gone away and never demote one")
|
|
}
|
|
if !health.ReadsAbsence() {
|
|
return fmt.Errorf("the node health monitor was built with no source of worker absence: it would report a worker whose tunnel is gone as healthy indefinitely")
|
|
}
|
|
return nil
|
|
}
|