mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 09:57:57 -04:00
e55cc3e2a7
The worker's gRPC port allocator grew monotonically with no upper bound: nextPort started at the base port and incremented whenever freePorts was empty, and nothing checked 65535. Past that it handed out integers that cannot be bound, surfacing as an opaque "backend won't start". #10961 estimated this needed ~15,000 concurrent-peak allocations, i.e. effectively unreachable. It is not, because of a second defect: the "process died unexpectedly" branch in startBackend deleted the process map entry without releasing its port at all. That port was leaked, never quarantined and never reused. A crash-looping backend leaks one port per restart, so a backend dying every 30s walks 50051 to 65535 in about five days. The leak, not concurrent peak, is the realistic route to exhaustion. Fixing the leak alone would have been wrong. Releasing that port makes it re-bindable, and the death path is the one teardown path with no request/reply to carry StoppedProcessKeys back to the controller (#10952's eager row removal), so a stale NodeModel row could then resolve to a live listener belonging to a different backend. probeHealth verifies liveness, not identity, so the request is silently misrouted. The 15s port quarantine does not cover this: the only reaper is the per-model health check at ~45s, and it can be disabled outright. The residual was masked only because the port was never rebound. So both are fixed together: - The allocator takes an explicit [basePort, LOCALAI_GRPC_MAX_PORT] range and returns ErrNoFreePort naming the range, the live backend count, the quarantined count, and the knob to raise. Exhaustion is now diagnosable instead of surfacing as an unbindable port. - Released ports carry per-key affinity: a port is offered back to the process key that last held it before any other key. Process keys (modelID#replica) and NodeModel rows (nodeID, modelName, replicaIndex) are isomorphic, so a port that can only be re-bound by its previous owner can only ever be named by that owner's row, which that key's re-registration overwrites. Misrouting to a different model becomes impossible by construction rather than by racing the quarantine timer. Affinity is a preference, not a reservation: under range pressure an owned port is stolen with a warning, because a guaranteed outage is worse than a rare misroute window on a port long out of quarantine. Claiming a port evicts its previous owner's entry, keeping ownership injective over ports so the affinity map can never exceed the range width regardless of how many distinct model keys the worker sees. Ownership also expires. It is only load-bearing while a controller row could still name the port, which the per-model reaper bounds at roughly 45s, so it lapses after five minutes and the port becomes ordinary free space again. Holding it indefinitely would have made every distinct model the worker ever served consume a port permanently: every release path is keyed, so nothing would ever be unowned, the allocator would climb to the end of its range on distinct-key count rather than concurrency, stealing would become routine, and the steal warning would tell operators to widen a range that was not the constraint. With expiry, reaching the steal branch means the worker is genuinely out of concurrent capacity, so that advice is correct when it appears. Closes #10961 Closes #10952 Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>