mirror of
https://github.com/mudler/LocalAI.git
synced 2026-06-06 07:46:15 -04:00
858257eaf0
* fix(distributed): self-heal stale 'model not loaded' routing In distributed mode the registry can list a model as loaded on a node while the worker has evicted it (autonomous LRU eviction, an out-of-band unload, etc.) yet the backend process survives. The router's cached-node check only verifies the process is alive (probeHealth), so it routes there and inference fails with "<backend>: model not loaded" — and stays broken until the controller restarts and rebuilds its registry. InFlightTrackingClient now reconciles this: when a tracked inference call returns a model-not-loaded error, it drops the stale replica row (RemoveNodeModel) so the next request reloads the model on a healthy node instead of routing back to the evicted one. The original error is returned unchanged; only the registry is corrected. Assisted-by: Claude:claude-opus-4-8 go vet Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(distributed): typed model-not-loaded error via gRPC status code Replace the controller-side error-string match with a shared, code-aware helper. Go error types don't survive the gRPC boundary, so the signal is carried as a status code (FailedPrecondition): - pkg/grpc/grpcerrors: ModelNotLoaded(backend) constructor + IsModelNotLoaded(err) checker (status-code first, message fallback for backends not yet migrated). - InFlightTrackingClient.reconcile now uses grpcerrors.IsModelNotLoaded. - Migrate the Go backends that emit this error (parakeet-cpp, cloud-proxy, rfdetr-cpp) to the typed constructor. Acting on a false positive is harmless (the model is just reloaded). Assisted-by: Claude:claude-opus-4-8 go vet Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
36 lines
1.3 KiB
Go
36 lines
1.3 KiB
Go
// Package grpcerrors defines well-known error signals shared between backends
|
|
// (which produce them) and the router (which consumes them). Go error types do
|
|
// not survive the gRPC boundary, so these conditions are carried as gRPC status
|
|
// codes and detected via the code rather than by matching the error message.
|
|
package grpcerrors
|
|
|
|
import (
|
|
"strings"
|
|
|
|
"google.golang.org/grpc/codes"
|
|
"google.golang.org/grpc/status"
|
|
)
|
|
|
|
// ModelNotLoaded returns the canonical error a backend returns when it has no
|
|
// model loaded for the request. It carries codes.FailedPrecondition so callers
|
|
// can detect it across the gRPC boundary without matching the message string.
|
|
func ModelNotLoaded(backend string) error {
|
|
return status.Errorf(codes.FailedPrecondition, "%s: model not loaded", backend)
|
|
}
|
|
|
|
// IsModelNotLoaded reports whether err signals that the backend has no model
|
|
// loaded. It prefers the typed gRPC status code (FailedPrecondition) and falls
|
|
// back to the message for backends that have not yet adopted ModelNotLoaded.
|
|
//
|
|
// Acting on a false positive is harmless: the only consequence upstream is that
|
|
// the model is reloaded, which is idempotent.
|
|
func IsModelNotLoaded(err error) bool {
|
|
if err == nil {
|
|
return false
|
|
}
|
|
if status.Code(err) == codes.FailedPrecondition {
|
|
return true
|
|
}
|
|
return strings.Contains(strings.ToLower(err.Error()), "model not loaded")
|
|
}
|