mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 09:57:57 -04:00
248e1ef9a2
A backend reinstall could poison every subsequent model load on a worker
node until the worker process was restarted.
gallery.InstallBackend (and gallery.UpgradeBackend) replace a backend by
renaming the live directory to `<name>.install-backup`, moving the staged
directory into place, then deleting the backup. A working directory
follows the inode across a rename, so a backend process that outlives
that swap ends up with a deleted inode as its CWD, and every getcwd(2)
in it fails with ENOENT.
Observed on a Jetson Thor worker in distributed mode after two
successive reinstalls of cuda13-nvidia-l4t-arm64-longcat-video-development.
A later model load failed with:
rpc error: code = Internal desc = failed to load LongCat model: [Errno 2] No such file or directory
The backend's own traceback shows it dying while importing torch, before
touching any model file:
backend.py line 142 in LoadModel
backend.py line 300 in _import_torch
torch/_library/custom_ops.py lib._register_fake(...)
torch/library.py:183 caller_module = inspect.getmodule(frame)
inspect.py:1013 f = getabsfile(module)
inspect.py:983 return os.path.normcase(os.path.abspath(_filename))
<frozen posixpath>, line 415, in abspath
FileNotFoundError: [Errno 2] No such file or directory
os.path.abspath calls os.getcwd() for a relative path. Scanning /proc
inside the worker container found the deleted CWD directly:
pid 23467 CWD DELETED: /backends/cuda13-nvidia-l4t-arm64-longcat-video-development.install-backup (deleted)
Restarting the worker container cleared it (dead CWD count 1 -> 0).
Python backends import torch lazily inside LoadModel, so such a survivor
still answers HealthCheck and keeps its gRPC port. It looks healthy and
only detonates when a model is actually loaded through it.
The install paths already stop running processes before replacing the
directory (installBackend's force branch, upgradeBackend, backend.delete),
but they resolve them by name. That bookkeeping reaps nothing whenever
the recorded name no longer resolves into the install's identity set: a
legacy entry with an empty backendName, backendIdentity degraded to
name-only matching after a ListSystemBackends failure, or an earlier
reinstall having already rewritten the metadata.json that carries the
alias. Any of those leaves a live process whose directory is about to be
unlinked, and nothing downstream notices, because the reuse gate checks
liveness and name -- and the name is precisely what does not change
across a reinstall.
Record the directory each supervised process runs out of, plus that
directory's identity at spawn time, and compare with os.SameFile before
reusing the process. This needs none of the name bookkeeping to have
been correct. Both reuse gates are covered: processMatchesBackend (the
install fast path) and startBackend's own already-running branch, which
now force-stops such a survivor so the fresh spawn chdirs into the newly
installed directory. Processes with no recorded directory are accepted,
so a rollout does not restart every running backend once.
This matters more with #11024 pending: making GPU backends visible to
the upgrade checker will have AutoUpgradeBackends fan upgrades out to
worker nodes at scale, and every one of those is a reinstall. Left as
is, a rare manual-upgrade footgun becomes a fleet-wide one.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>