mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-25 15:44:56 -04:00
fix: return 503 when scheduler has no available nodes
When the scheduler cannot find any healthy node to serve a model —
all nodes are full and eviction cannot free a slot, or a node_selector
excludes every candidate — the error fell through to 500. A 500 tells
clients something is broken when the condition is transient and
retryable.
The router now wraps these errors with a new ErrNoAvailableNodes
sentinel. The HTTP error handler maps it to 503 via applyNoAvailableNodes,
following the same pattern as applyBackendAdmission (429). Unrelated
scheduler errors (DB timeouts, registry lookups) still return 500.
Three return sites are wrapped:
- resolveSelectorCandidates: selector matches zero healthy nodes
- scheduleNewModel eviction-busy: all models have in-flight requests
- scheduleNewModel eviction-failed: eviction itself errored
The existing scheduleAndLoad wrapper ("no available nodes: %w") preserves
the sentinel through the chain via errors.Is, as does ModelRouterAdapter.
Assisted-by: AGENT:regolo/glm5.2 [TOOL]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This commit is contained in:
1 parent
7f3029e0d3
commit
ea26692a07
6 files changed
+86
-3
No files matched your search
@@ -90,6 +90,7 @@ The `/v1/responses` endpoint returns errors with this structure:
|
||||
| 422 | Unprocessable Entity | Validation failed (e.g., invalid parameter range) |
|
||||
| 429 | Too Many Requests | All backends are saturated (per-model `max_concurrent` or process-wide `--max-concurrent-backend-requests` ceiling reached). Includes a `Retry-After` header and `type: "rate_limit_error"` so OpenAI-compatible clients and harnesses back off automatically |
|
||||
| 500 | Internal Server Error | Backend inference failure, unexpected server errors |
|
||||
| 503 | Service Unavailable | No healthy node available to serve the model (cluster is full, eviction cannot free a slot, or a `node_selector` excludes all candidates). Also used during model-load cooldown and while a model is still cold-loading. Retryable |
|
||||
|
||||
## Global Error Handling
|
||||
|
||||
|
||||
Reference in new issue
Block a user