mudler's LocalAI [bot] 54d5c18bfb fix(model): only announce a load at INFO when a load actually happens (#11017) ...

backendLoader logged "BackendLoader starting" at INFO as its very first
statement, unconditionally. That reads as "a model is being loaded", but
backendLoader is not only a load path: in distributed mode Load()
deliberately bypasses the local cache and calls backendLoader on every
inference request so SmartRouter can re-pick a replica per request. The
model is already resident, no process is spawned, and nothing is loaded,
yet the banner fires at request rate.

On a live cluster this produced ~5 "BackendLoader starting" lines per
second for a single embedding model, sustained, starting 22 seconds
after the load had already completed. The model was state=loaded with
in_flight=0 and exactly one backend process on the worker. It looked
exactly like a retry storm and cost real debugging time during an
unrelated production investigation. The adjacent "effective runtime
tuning" banner, documented as "logged once per load", had the same
problem for the same reason.

Emit both banners at INFO only when the model is not already resident,
and keep the per-call trace at DEBUG for anyone following the routing
path. isResident is a plain store lookup with no health probe and no
eviction, so it is safe on the per-request hot path (unlike
checkIsLoaded, which probes and can evict).

Same class of defect as #10985: a log line that sends the reader after
the wrong thing.


Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

2026-07-21 15:33:29 +02:00

17 KiB

Raw Blame History

View Raw