Files
LocalAI/core
Ettore Di Giacinto 5880f4e6dd test(distributed): pin the no-demotion rule at every call site it is stated
Review fix round 1. Two blocking findings and seven non-blocking; both
blocking ones are M12's shape again, and this time on the invariant
itself. No production behaviour changes here: everything below was
already correct and merely unpinned, so re-inserting the defect left all
679 specs green. The only non-comment edits are one struct-field comment
and one log message.

"A failed control RPC no longer demotes a node" is stated three times in
this package and was pinned once, at ListBackends. Putting MarkUnhealthy
back at either op-drain site passed. What that buys in production is the
fleet-wide eviction this phase exists to prevent: MarkUnhealthy removes a
node from ListDuePendingBackendOps AND from scheduling, so a frontend
replica that has just lost its tunnels demotes every node it holds an op
for, for a reason that is about the frontend. The reconciler's is the
worse of the two, being a background loop nobody is watching. Both now
have a spec, each with the recorded op failure as its negative control so
"still healthy" cannot pass by nothing having happened.

The sweep the review asked for found four more rules stated at more call
sites than they were pinned at, and two the review had not:

The still-installing surfacing at the manager layer has two call sites
and was pinned at InstallBackend. Dropping it from UpgradeBackend
reported a spent budget as GREEN SUCCESS: the admin sees the upgrade
finished while the worker is still re-pulling gigabytes.

The agent-node skip has two call sites and was pinned at ListBackends.
Without it the fan-out enqueues a row for every agent node, and an agent
worker serves no control plane, so that row can never drain: it retries
until the dead-letter cap.

The still-installing conversion has three call sites and was pinned at
two; the legacy force-install fallback was the gap. Its budget was
unpinned too, so the new spec asserts both, on the upgrade budget rather
than the install one, since the fallback re-fires an install as part of
an upgrade.

The carrier split has two call sites and was pinned at one. Hardcoding
NodeTypeBackend in UnloadRemoteModelContext passed, and an agent node
holding a node_models row would then have its stop sent over a tunnel it
does not hold, fail, and leave the row behind. The new spec unloads a
model held by one node of each kind and asserts each stop went to that
node's own carrier and to no other.

router_nats_liveness_test.go asserted demote-on-absence, which production
can no longer produce, and its header described the pre-cutover world.
The exclusion is unreachable by construction rather than by argument:
cluster, the package supplying every control-path dial error, does not
link nats.go at all. The file now says that, and gains the assertion that
IS load-bearing, a table naming each sentinel a control RPC can answer
with and requiring that none of them excludes. Widening the exclusion to
ErrWorkerUnroutable reddens four of its entries plus the real-adapter
scheduling spec.

unroutable keeps no budget-first guard and the reason is now written at
it: unlike controlFailure it reads one already-recorded error rather than
racing a live deadline, and an expiry is not in streamRefusals, so it
falls to the umbrella without one. The two implement the same split at
two layers and each now names the other.

Fourteen comments still described the bus. Among them the reconciler
saying a drain would "churn NATS every tick", a spec comment naming a
subject builder this branch deleted, and the agent-skip comment
explaining the skip by a subscription that no longer exists.

Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
2026-09-02 20:45:01 +00:00
..
2026-03-30 00:47:27 +02:00