feat(distributed): cache staged-artifact hashes and publish load lifecycle
Every load request re-hashed every staged artifact on the controller
(probeExisting and the upload path both re-read the full file), which for
a large multi-file model on NAS-backed storage is minutes of pure
re-reading per request even when nothing changed - observed as ~9 minutes
of "Upload skipped (file already exists with matching hash)" before every
avatar generation. Cache the local hash in the same .sha256 sidecar the
worker-side transfer server already maintains, invalidated whenever the
sidecar is older than the file.
The whole staging+loading phase was also invisible: the NodeModel row was
only written after LoadModel succeeded, so /api/nodes and the UI showed
nothing while a cold load spent 10+ minutes staging - indistinguishable
from nothing happening. Publish the lifecycle instead: "staging" as soon
as the node is chosen, "loading" when the checkpoint load starts, and the
existing "loaded" on success, with the row removed on any failure so a
dead load does not leave a phantom replica. The early row also reserves
the replica slot against concurrent schedulers. The nodes view already
renders non-loaded states on model chips; style "staging" like "loading".
Audited every state-filtered registry/router query: eviction, routing,
reconciler and idle-model queries all filter state='loaded' explicitly,
so the new transitional rows are visible to observability surfaces but
inert to scheduling decisions (except slot occupancy, intentionally).
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(gallery): coalesce artifact download progress
Buffer high-frequency downloading events and forward only the latest event on a periodic tick. Flush progress synchronously at phase boundaries and shutdown to preserve ordering and final state.
Assisted-by: Codex:gpt-5
* fix(gallery): wire progress coalescing into model installs
Route artifact progress through the 250 ms coalescer and flush it on every model operation exit. Keep the legacy download callback unchanged.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(backend): add magpie-tts-cpp text-to-speech backend
Add a Go + purego backend wrapping the magpie-tts.cpp ggml port of NVIDIA's
Magpie TTS Multilingual 357M (encoder + autoregressive decoder over NanoCodec
tokens), producing 22.05 kHz mono audio in 5 baked voices (Aria, Jason, John,
Leo, Sofia; case-insensitive names or indices 0-4) across 9+ languages from a
single self-contained GGUF. Mirrors qwen3-tts-cpp / moss-tts-cpp: dlopen the
static-ggml shared library, bind the flat magpie_tts_capi_* C-API via purego
(no local C shim needed, the upstream .so exports it directly), and serve the
gRPC TTS + TTSStream methods behind base.SingleThread (the C context is not
reentrant across synthesize calls).
The backend CMakeLists translates the Makefile's -DGGML_{CUDA,METAL,VULKAN,HIP}
flags into upstream's MAGPIE_GGML_* toggles (upstream FORCE-overwrites the ggml
cache entries from those), pinned to magpie-tts.cpp v0.1.1
(e3f3dd1ebe22b64e7405f93b519f2d1930712568), which statically links ggml into
libmagpie-tts.so (ldd shows only system libs).
Wires the full registration: backend-matrix.yml (CPU amd64/arm64, CUDA 12/13,
Intel SYCL f16/f32, Vulkan amd64/arm64, ROCm, NVIDIA L4T + L4T CUDA 13, and
Darwin metal), backend/index.yaml metas and image entries, the root Makefile
build targets, the changed-backends backend-filter path mapping, the bump_deps
auto-bump matrix, a test-extra per-backend smoke job, the /backends/known
pref-only importer entry, the backend capabilities map (TTS + TTSStream, no
voice cloning), and the README / compatibility-table docs rows.
Verified locally: unit + e2e Ginkgo suites pass against the real q8_0 GGUF
(22.05 kHz mono WAV, RMS > 0.01), a live gRPC LoadModel + TTS round-trip
returns valid non-silent audio, and the pre-commit gates (make lint,
make test-coverage-check) pass, run manually with LOCALAI_TEST_HTTP_PORT
overriding the locally-occupied 9090.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* gallery: add magpie-tts-cpp model entries (q8_0 + f16)
Add the Magpie TTS Multilingual 357M GGUFs from mudler/magpie-tts.cpp-gguf to
the model gallery: q8_0 (~624 MB, near-lossless, fastest decode, recommended)
with an f16 (~784 MB) variant, both served by the magpie-tts-cpp backend.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* magpie-tts-cpp: bump pin to rewritten upstream v0.1.1 SHA
Upstream history was rewritten to purge accidentally committed build
artifacts; v0.1.1 now resolves to 6f7696cf.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Remove the versioned pre-commit hook and its installer while retaining CI coverage and conformance checks.
Assisted-by: Codex:gpt-5
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The stall watchdog arms as soon as the response body exists, but the
downloader then re-hashed the entire existing .partial before reading a
single byte from the network. On slow models storage (a CIFS share
reading at ~117MB/s) hashing a multi-GB partial outlasts the 60s stall
window, so the watchdog aborted every healthy resume with 'download
stalled: no data received for 1m0s'. The partial never grew, so every
retry re-paid the same hash and failed identically, wedging the install
permanently (any partial over ~7GB on such storage).
Open the partial and hash it before the HTTP request instead: the
watchdog now only measures actual network idle time, and the origin no
longer sits on an idle connection while the hash runs.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(trl): disable inline GRPO reward code by default (RCE)
POST /api/fine-tuning/jobs accepts reward_functions[].code, an inline Python
body, and compile_inline_reward() execs it against a restricted-builtins
allowlist (_SAFE_BUILTINS). That allowlist is not a security boundary:
().__class__.__bases__[0].__subclasses__() reaches os._wrap_close and thus
os.system, giving arbitrary code execution. The fine-tuning endpoint is
unauthenticated by default, so any caller could run code on the host.
Hardening the allowlist is a losing game against CPython introspection, so
inline reward code is now refused unless the operator explicitly opts in with
LOCALAI_TRL_ALLOW_INLINE_REWARD=true on the backend. Builtin reward functions
are unaffected. The gate lives in build_reward_functions(), the single point
all inline specs flow through. Docs updated to stop describing the allowlist as
a sandbox and to document the opt-in.
Fixes#11015
Signed-off-by: Isabel Wu <231155141+wuisabel-gif@users.noreply.github.com>
Co-authored-by: Isabel Wu <231155141+wuisabel-gif@users.noreply.github.com>
fix(model-artifacts): persist companion artifacts, not just the primary
A managed model can declare companion artifacts (LongCat-Video-Avatar-1.5
pulls its tokenizer, text encoder and VAE from the separate LongCat-Video
base repo via a target: companion artifact). preloadOne resolves the whole
set in memory, but the binding written back to disk carried only the
primary: persistArtifactBinding marshalled []Spec{result.Spec} and replaced
the entire artifacts: list with it, silently dropping every companion.
In a single process the loss is invisible because the in-memory config keeps
the companion. It bites on the next controller restart: the config reloads
from the mangled file with the primary alone, so withCompanionArtifactOptions
finds no resolved companion and synthesizes no base_model option. The remote
longcat-video backend then never receives base_model, falls back to
BASE_MODEL_ID and downloads the repo itself ("Downloading required files for
meituan-longcat/LongCat-Video"), failing the load with "base_model must point
to a LongCat-Video checkpoint".
This is why an explicit base_model:<path> added to the config options works
where the managed companion does not: an explicit option lives in options:,
which is never rewritten, while the managed companion lives in artifacts:,
which the binding overwrote.
Persist the full resolved set (primary + every companion), and widen
bindingNeedsPersistence to compare the whole artifact list so a companion
resolving for the first time still triggers a write. The single-node path is
unaffected: there the in-memory config already carried the companion, and the
staging/ModelPath resolution for a remote worker (nested per-model staged
root, #10949) is unchanged and already correct once the option is generated.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The Python gRPC bindings expose message fields as plain attributes
(request.language, request.caption), not Go/Java-style Get*() accessors.
Because request.language is an empty string when unset, the
request.language or request.GetLanguage() or "en"
expression falls through to request.GetLanguage(), which does not exist
on the generated Python message and raises AttributeError, surfaced to
clients as:
rpc error: code = Unknown desc = Exception calling application: GetLanguage
Every /v1/sound-generation request without an explicit language field
failed. Drop the bogus accessor calls (TTS already uses the plain-field
form a few lines below).
Signed-off-by: Tai An <antai12232931@outlook.com>
* fix(backends): choose the protoc generator from the protobuf runtime, and regenerate stubs after late installs
The vLLM backends still crash on startup with
VersionError: Detected incompatible Protobuf Gencode/Runtime versions when
loading backend.proto: gencode 7.35.0 runtime 6.33.6
despite #10735 and #10944. Three separate defects kept it alive.
1. runProtogen picked the generator from the installed *grpcio* version.
grpcio-tools' version tracks grpcio, but the gencode its bundled protoc
emits tracks *protobuf*, and the two move independently: grpcio-tools
1.82.1 (the version #10735 pins to, matching grpcio 1.82.1) requires
protobuf>=7.35.1 and stamps gencode 7.35.0. Pinning to grpcio could
therefore never constrain the gencode. Constrain the install to the
protobuf already in the venv instead and let the resolver pick the newest
compatible grpcio-tools. That both selects a generator the runtime accepts
and stops protogen from moving the runtime under the backend's other deps.
This is self-correcting, so the hardcoded GRPCIO_TOOLS_VERSION=1.78.0
escape hatch from #10944 is no longer needed and is removed.
2. The stubs were generated too early. Most branches of vllm/install.sh (and
vllm-omni) install vllm *after* installRequirements, and vllm re-resolves
the protobuf runtime as it lands. Stubs generated against the pre-vllm
runtime can end up newer than the runtime that finally ships, which is the
ROCm failure exactly. Regenerate once the dependency set is final.
3. rm -f of the .py sources left __pycache__ behind. CPython validates a .pyc
against source mtime and size, both of which can be unchanged across a
regeneration (the gencode triple is the same width whether it reads 7.35.0
or 6.33.5), so a stale backend_pb2.pyc could shadow the stub just written.
Also fail the build when the generated stub cannot be imported, so a
gencode/runtime mismatch surfaces at image build time instead of reaching
users as an opaque "grpc service not ready".
Verified by driving the real runProtogen through the ROCm install sequence in
a venv harness: before, gencode 7.35.0 against runtime 6.33.6 (reproducing the
reported error verbatim); after, gencode 6.33.5 against runtime 6.33.6 and the
stub imports cleanly.
Closes#10940Closes#10718
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
* fix(backends): regenerate protobuf stubs in the other backends that install after installRequirements
Same defect as the vllm change: installRequirements generates the stubs at the
end of its own run, so any backend that installs further packages afterwards can
have the protobuf runtime moved out from under stubs that were already written.
The gencode stamped into backend_pb2.py then exceeds the runtime that ships and
the backend dies at model load with "grpc service not ready".
fish-speech already had this bug and worked around the symptom: it forces
protobuf>=5.29.0 after installRequirements precisely because "transitive deps
(wandb, tensorboard) may downgrade protobuf to 3.x but our generated
backend_pb2.py requires protobuf 5+". Regenerating after the pin addresses the
cause rather than propping up the runtime to match stale stubs.
Applied to the backends whose post-installRequirements step resolves a
dependency graph and can therefore move protobuf:
fish-speech -e . plus an explicit protobuf install
vibevoice pip install . (with deps)
llama-cpp-quantization gguf / GGUF_PIP_SPEC
trl gguf / GGUF_PIP_SPEC
Deliberately not applied to ace-step and chatterbox (both --no-deps, so the
dependency graph cannot change) or voxcpm (pins setuptools only). gguf does not
depend on protobuf today, but it resolves dependencies, and "this package does
not touch protobuf right now" is exactly the assumption that made the earlier
fix ineffective.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
* fix(backends): resolve the protoc generator in a throwaway env so it cannot edit the backend's pinned deps
Installing grpcio-tools into the backend's own venv to generate the stubs also
drags its dependencies in: grpcio-tools 1.82.1 requires grpcio>=1.82.1, so a
backend that pinned grpcio==1.78.1 silently shipped 1.82.1 instead. Caught by
building the llama-cpp-quantization image and reading the versions back out of
the artifact:
before grpcio 1.82.1 (requirements.txt pins grpcio==1.78.1)
after grpcio 1.78.1 grpcio-tools absent from the venv entirely
Resolve the generator in a throwaway environment instead, still constrained to
the protobuf the backend ships so the gencode stays compatible. The backend's
dependency set is then exactly what its requirements files declared. protoc's
output is plain Python and carries no dependency on the interpreter that
produced it, so generating from a different env is safe; the import check still
runs under the backend's python, since that is the interpreter that has to load
the stubs at model load.
Verified on the rebuilt image: gencode 7.35.0, runtime protobuf 7.35.1, grpcio
back at its pinned 1.78.1, and the shipped stub imports cleanly against 7.35.1.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
* fix(backends): bound the protoc generator by BOTH the installed grpcio and protobuf
The generated stubs impose two independent constraints, and every fix so far,
including the previous commit on this branch, satisfied one while violating the
other:
backend_pb2.py needs protobuf runtime >= gencode
backend_pb2_grpc.py needs installed grpcio >= grpcio-tools
Resolving the generator against protobuf alone picked grpcio-tools 1.82.1 for a
backend holding grpcio at 1.78.1, so the gencode was fine but the gRPC stub was
not:
RuntimeError: The grpc package installed is at version 1.78.1, but the
generated code in backend_pb2_grpc.py depends on grpcio>=1.82.1.
That is also why installing grpcio-tools into the backend venv appeared to work
earlier: it dragged grpcio up to match, which was load-bearing rather than the
regression it looked like. Isolating the generator removed the accidental fix
and exposed the missing constraint.
Bound grpcio-tools from both sides instead and let the resolver find the newest
version satisfying both. The protobuf ceiling makes it back off to an older
generator when the runtime trails, bounding the gencode; the grpcio ceiling
keeps the _grpc stub loadable. Resolved against the four real runtime pairs
observed in built images:
grpcio 1.78.1 / protobuf 7.35.1 -> grpcio-tools 1.78.0, gencode 6.31.1 OK
grpcio 1.78.0 / protobuf 6.33.6 -> grpcio-tools 1.78.0, gencode 6.31.1 OK
grpcio 1.82.1 / protobuf 6.33.6 -> grpcio-tools 1.81.1, gencode 6.33.5 OK
grpcio 1.82.1 / protobuf 7.35.1 -> grpcio-tools 1.82.1, gencode 7.35.0 OK
Also restore the import check to cover backend_pb2_grpc as well as backend_pb2.
Narrowing it to backend_pb2 is why the image build passed while CI failed: the
guard could not see the constraint that was actually broken.
Verified by running the CI sequence locally for llama-cpp-quantization, the
backend whose test failed:
make -C backend/python/llama-cpp-quantization -> exit 0
make -C backend/python/llama-cpp-quantization test -> exit 0, OK
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Bash] [Edit]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update TheTom/llama-cpp-turboquant
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(turboquant): refresh HIP compatibility patch
The updated fork now carries its own HIP-safe peer-copy path, so the old hunk no longer applies. Keep only the event-creation compatibility change that the fork still needs.
Assisted-by: Codex:gpt-5 [Codex]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
materializeLocked built a download task for every file in the resolved
snapshot unconditionally. A completed file is promoted from
.downloads/<hash> into snapshot/<path> and its blob deleted, so on any
re-entry (a controller pod roll, a resubmit, a crash) the new pass built a
task whose .downloads blob no longer existed, re-downloaded the whole file
from Hugging Face, and its AfterDownload even removed the already-complete
snapshot copy first. The only resume that worked was the downloader's
per-file .partial resume for a file caught mid-transfer; completed files
were never skipped.
Production consequence: installing longcat-video-avatar-1.5 (~35 GB after
allow_patterns) on a cluster whose controller rolls hourly (Flux image
automation) never converged across ~14 hours. Each roll restarted from the
first shard; the completed bytes on disk were repeatedly deleted and
re-fetched, and the artifact never promoted. curl of the same files from
inside the pod ran fine, proving the loss was the materializer re-fetching,
not the network.
Before building a task, check whether the file is already materialized and
verified in this staging tree's snapshot/ and, if so, keep it and count it
complete instead of downloading. "Materialized" means a regular file of the
expected size that passes the same verifyDownloadedFile check the download
path uses, so the kept manifest entry is byte-for-byte identical to a fresh
one and integrity is re-checked. The manifest requires a SHA-256 for every
file and non-LFS files carry none to borrow, so a hash is unavoidable for
the manifest anyway; a full re-hash of local disk is still orders of
magnitude cheaper than re-downloading, and the downloader re-verifies any
file it does fetch. Manifest entries are now written at their snapshot index
rather than appended in completion order, so a mix of skipped and downloaded
files keeps the resolved order that committedResult and staging read. The
unconditional root.Remove(destination) now runs only on the fresh-download
path; a kept file survives. Skips are logged at INFO with count and bytes so
an operator can see resume working.
This is the resume-side counterpart to the sibling defects on this path:
read/write error conflation and transient retry (#10985), hash-verify
progress accounting and silent success on an expired deadline (#11026), and
the response-header hang (#11053). The download machinery resumed a single
in-flight file; the materializer above it still threw away every completed
file on restart. It also makes orphan-partial adoption worth its cost:
an adopted tree's completed files were re-downloaded anyway until now.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nodes): never schedule a model onto a node that cannot store it
A worker whose models filesystem was 100% full kept advertising
`status: healthy`, stayed a scheduling candidate, was picked to host a
70 GB video model, accepted the staging request, transferred ~17 GB and
only then failed:
staging .../whisper-large-v3/model.fp32-00001-of-00002.safetensors:
upload to node b7bacbf4-... failed with status 500:
writing file: /models/longcat-video-avatar-1.5/...: no space left on device
The node was at 937G/937G/0-avail. Total elapsed before the truth
surfaced: 16 minutes, for a decision that could never have succeeded.
The worker health signal only ever proved liveness. `/readyz`
(WorkerReadiness/NATSReadiness) checks the NATS link; `status: healthy`
in the registry is driven by heartbeat recency. Node capacity carried
VRAM and RAM but no disk figure at all, and the router compared model
size against VRAM only — nothing anywhere looked at free space on the
filesystem that staging actually writes to.
Report it, then use it:
- Workers now measure the filesystem backing their MODELS directory
(not `/` -- staged weights land in the models path, and that mount is
very often separate) and report `total_disk`/`available_disk` on
registration and on every heartbeat. Free disk moves faster than VRAM
under staging traffic, so the per-heartbeat refresh matters.
- The SmartRouter drops nodes that cannot store the model before it
picks one. The requirement comes from `modelPayloadBytes` -- the same
local paths `stageModelFiles` uploads, already computed for the
size-derived load budget -- plus a 5% / 1 GiB margin, rather than a
fixed percentage of the node's disk. A percentage threshold would take
a small-but-usable node out of rotation for models it could hold, and
on a homogeneous cluster would strand every node at once.
- When no node fits, scheduling fails immediately with an error naming
the requirement and each node's free space, instead of picking one and
discovering it mid-transfer.
Two deliberate non-changes. Low disk does not mark a node `unhealthy`:
the check is per model, so a node too small for one model stays a valid
target for smaller ones. And `total_disk == 0` means "does not report
disk" (pre-upgrade worker, or a failed stat), not "full" -- such nodes
pass through untouched so a rolling upgrade never empties the candidate
pool. A genuinely full node is distinguishable: non-zero total, zero
available. Registry read failures are logged and scheduling continues
unfiltered; a database hiccup must not wedge a cluster.
Free space is surfaced on the node detail page next to VRAM, since the
incident's signature was a node that looked entirely healthy.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
* feat(nodes): make the disk-headroom check operator-controllable
The admission check added in the previous commit had no off switch. A
scheduler-side veto with no escape hatch is a liability: our size
estimate can be wrong (deduplicating or compressing filesystems, a
backend that fetches its own weights rather than loading the staged
copy), and an operator who hits that has no way out but a downgrade.
Add one knob with two surfaces that share a single source of truth:
- `--distributed-disk-headroom-check` / `LOCALAI_DISTRIBUTED_DISK_HEADROOM_CHECK`
(default true), following the `--distributed-prefix-cache` pattern for
a default-on distributed feature.
- `distributed_disk_headroom_check` in the runtime-settings registry, so
it can be flipped without a restart from `POST /api/settings` and from
Settings -> Distributed in the WebUI.
Both write `DistributedConfig.DiskHeadroomDisabled`, and the SmartRouter
reads that member LIVE on every scheduling decision through a closure
over the application config rather than a value snapshotted at
construction. Env/CLI sets the boot value, the runtime setting overrides
it live, last write wins, and there is exactly one member to read.
Snapshotting would have made the runtime toggle a no-op until restart.
Disabled means WARN, not SKIP. Selection goes back to ignoring free disk
-- byte for byte the pre-check behaviour -- but the check still runs, and
when it would have rejected every node it says so, naming the knob that
suppressed it. Going quiet when switched off would reproduce the exact
condition that made the original incident expensive: a cluster doing
something that could not work and saying nothing. Disabling is also
logged once at startup. Warning only on the total-rejection case keeps
it actionable rather than chatty on a heterogeneous cluster.
Also fixes a false positive in the check itself: shared-models mode
(LOCALAI_DISTRIBUTED_SHARED_MODELS) stages nothing at all -- every node
already mounts this models directory at this path -- so demanding the
full checkpoint size of free space per node would have rejected a
cluster that needs no new bytes. The check is skipped there entirely.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Two lifecycle defects observed on a 2-replica distributed cluster.
The install endpoints mint a job UUID, hand the operation to an unbuffered
channel, and answer HTTP 200 immediately. The gallery worker is a single
goroutine that processes operations serially, and the first status write
happens inside modelHandler/backendHandler — i.e. only once the worker
actually starts the work. An operation queued behind a running install
therefore had no status at all: GET /models/jobs/<uuid> answered
"could not find any status for ID" and GET /models/jobs did not list it,
so the endpoint reported success for work nothing could observe. On the
paths that sent directly rather than from a goroutine, the same unbuffered
channel blocked the HTTP handler for the whole duration of the in-flight
install, which is how a replica came to accept no /models/apply at all
while /readyz stayed green.
Admission now goes through EnqueueModelOp/EnqueueBackendOp, which publish a
"queued" status before handing the operation over, so a job ID is queryable
from the instant it is handed out. Delivery selects on the operation's
context, so cancelling a still-queued operation releases the delivery
goroutine instead of stranding it on a send that will never be received,
and an operation the worker never accepts becomes a terminal failure rather
than a silent leak.
The worker also had no panic containment. A panic in any handler propagated
out of the single consumer goroutine and killed the process, taking every
queued operation with it; it is now contained to the operation that caused
it. The two ignored galleryStore.Create errors are logged, and the model and
backend delete endpoints now run under the same ID they hand back — they
previously ran under an empty ID and returned a status URL for a job that
could never have a status.
Second, an operation orphaned by a controller replaced mid-download kept
reporting phase=downloading, processed=false, error=none while nothing was
downloading. The PostgreSQL side does recover on its own (FindDuplicate
ignores rows untouched for 30 minutes and CleanStale marks them failed), but
the reaper only ever corrected the database. The in-memory statuses map that
GET /models/jobs/<id> and /api/operations actually read was never corrected,
so every replica kept serving the frozen tick indefinitely. ReapStaleOperations
now reconciles the in-memory copy with the reap.
Note that operation ownership is still not tracked: gallery_operations has a
FrontendID column that nothing writes, so a live operation and one whose owner
died are distinguished only by a 30-minute staleness timeout. Narrowing that
window needs a lease/heartbeat mechanism and is out of scope here.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Three measured HTTP-layer regressions on a live deployment, fixed together
because they all shape the bytes on the wire.
1. No compression. The server sent no Content-Encoding regardless of what
the client asked for, confirmed with curl straight at 127.0.0.1:8080 so
it was not an ingress artefact. Adds gzip middleware, on by default and
configurable via LOCALAI_DISABLE_HTTP_COMPRESSION and
LOCALAI_HTTP_COMPRESSION_MIN_LENGTH (default 1024 bytes so tiny bodies
are not wastefully wrapped). Streaming routes are skipped explicitly:
an SSE Accept header, a WebSocket upgrade, and the completion / SSE /
log-tail path prefixes, because whether a completion request streams is
decided by the request body, which the middleware runs too early to see.
Already-compressed formats (woff2, png, mp4, ...) are skipped too; gzip
made those marginally larger. Measured over the embedded React build:
JS+CSS 2815 KB raw to 808 KB gzipped (3.48x).
2. No cache headers on content-hashed assets. Vite hashes the filenames,
so a given /assets/ URL can never change content, yet they shipped with
no Cache-Control, ETag or Last-Modified, and the browser re-fetched the
whole bundle on every navigation with no conditional request available.
/assets/* now carries public, max-age=31536000, immutable. index.html
stays no-cache so a deploy is picked up, and the unhashed locale JSONs
get a short TTL rather than the immutable one.
3. Unbounded trace endpoints. /api/traces returned 21,033,606 bytes in
4.65s and /api/backend-traces 3,471,682 bytes in 1.50s, and the admin
UI polls both every few seconds. The ring buffer holds up to 1024
entries, each embedding full input_text payloads. Both list endpoints
now take limit / offset / full, default to 50 entries, and strip the
heavy fields (request and response bodies plus headers for API traces,
body and data for backend traces) unless full=true. Every trace gets a
process-lifetime ID and GET /api/traces/{id} and
/api/backend-traces/{id} serve the full record, which is what the UI
fetches when a row is expanded. The list body stays a JSON array;
paging metadata rides in X-Total-Count, X-Trace-Offset and
X-Trace-Limit. Reproducing the live shape in a test, the polled payload
goes from 21,131,097 bytes to 7,201 bytes.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
build-test.yaml, yaml-check.yml and secscan.yaml had no concurrency block at
all, so every push to a PR stacked another full batch instead of superseding
the previous one. build-test carries a macos-latest job, the scarcest runner
class we use, and secscan fires on every push to every branch because its
`push:` trigger is unfiltered.
build-test and yaml-check use the same group idiom as lint.yml and the other
eleven workflows that already have one: key on the PR number so pushes to a PR
share a group, and cancel only on pull_request. On a master push the key falls
back to github.sha and cancel-in-progress is false, so master runs never cancel
each other -- that is deliberate, since backend.yml builds only the backends a
given commit touched and superseding would drop those builds.
secscan needs a different key: it has no pull_request trigger, so the shared
idiom would fall back to the unique-per-commit sha and dedup nothing. It groups
on github.ref instead, and excludes master from cancellation for the same
per-commit reason. Cancelling a superseded feature-branch scan is safe because
the only output is a SARIF upload and code scanning keeps the latest result
per ref.
No behaviour change on master for any of the three.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
A gallery model install hung for 94 minutes with zero bytes transferred, no
error, no retry and no abort, leaving a partial tree frozen at 18G. The last
log line was the download starting, then silence:
14:06:19 INFO Downloading url=".../LongCat-Video-Avatar-1.5/resolve/<rev>/base_model/diffusion_pytorch_model-000..."
The retry machinery from #10985 was working (two retries fired at 14:05:01 and
14:06:14); the third attempt simply never returned. The install never
completed, the model config was never written, and nothing surfaced the
failure.
The stall watchdog added earlier wraps the response *body*, so it only starts
guarding once downloadClient.Do() has returned. The transport had no
ResponseHeaderTimeout, so a peer that completes the dial and TLS handshake,
reads the request, and then never sends a status line parks Do() for the
process lifetime. IdleConnTimeout governs pooled idle connections, not an
in-flight request. Both the body request and the HEAD that probes for Range
support were unguarded.
Bound the header wait at the transport, not the client: a client-level Timeout
would also bound the body and truncate multi-tens-of-GB downloads. The knob is
opt-in (WithResponseHeaderTimeout) rather than a default in HardenedTransport,
because a streaming endpoint may legitimately withhold headers until it has
something to say, and capping that would break the streaming clients that share
this constructor.
Also fix a classification trap this exposed: net/http reports a
ResponseHeaderTimeout as an error satisfying errors.Is(err,
context.DeadlineExceeded), which IsRetryable read as "the caller gave up" and
refused to retry. An explicit transient marking now outranks the cancellation
sentinels; a caller who genuinely gave up is still caught by the ctx.Err()
check. The resume probe's error is likewise marked transient, so a momentarily
wedged origin no longer turns a resumable download into a hard install failure.
Third defect found in this download path, after #10985 (read vs write errors
conflated) and #11026 (hash verification emitted no progress and an expired
deadline returned success).
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
A backend reinstall could poison every subsequent model load on a worker
node until the worker process was restarted.
gallery.InstallBackend (and gallery.UpgradeBackend) replace a backend by
renaming the live directory to `<name>.install-backup`, moving the staged
directory into place, then deleting the backup. A working directory
follows the inode across a rename, so a backend process that outlives
that swap ends up with a deleted inode as its CWD, and every getcwd(2)
in it fails with ENOENT.
Observed on a Jetson Thor worker in distributed mode after two
successive reinstalls of cuda13-nvidia-l4t-arm64-longcat-video-development.
A later model load failed with:
rpc error: code = Internal desc = failed to load LongCat model: [Errno 2] No such file or directory
The backend's own traceback shows it dying while importing torch, before
touching any model file:
backend.py line 142 in LoadModel
backend.py line 300 in _import_torch
torch/_library/custom_ops.py lib._register_fake(...)
torch/library.py:183 caller_module = inspect.getmodule(frame)
inspect.py:1013 f = getabsfile(module)
inspect.py:983 return os.path.normcase(os.path.abspath(_filename))
<frozen posixpath>, line 415, in abspath
FileNotFoundError: [Errno 2] No such file or directory
os.path.abspath calls os.getcwd() for a relative path. Scanning /proc
inside the worker container found the deleted CWD directly:
pid 23467 CWD DELETED: /backends/cuda13-nvidia-l4t-arm64-longcat-video-development.install-backup (deleted)
Restarting the worker container cleared it (dead CWD count 1 -> 0).
Python backends import torch lazily inside LoadModel, so such a survivor
still answers HealthCheck and keeps its gRPC port. It looks healthy and
only detonates when a model is actually loaded through it.
The install paths already stop running processes before replacing the
directory (installBackend's force branch, upgradeBackend, backend.delete),
but they resolve them by name. That bookkeeping reaps nothing whenever
the recorded name no longer resolves into the install's identity set: a
legacy entry with an empty backendName, backendIdentity degraded to
name-only matching after a ListSystemBackends failure, or an earlier
reinstall having already rewritten the metadata.json that carries the
alias. Any of those leaves a live process whose directory is about to be
unlinked, and nothing downstream notices, because the reuse gate checks
liveness and name -- and the name is precisely what does not change
across a reinstall.
Record the directory each supervised process runs out of, plus that
directory's identity at spawn time, and compare with os.SameFile before
reusing the process. This needs none of the name bookkeeping to have
been correct. Both reuse gates are covered: processMatchesBackend (the
install fast path) and startBackend's own already-running branch, which
now force-stops such a survivor so the fresh spawn chdirs into the newly
installed directory. Processes with no recorded directory are accepted,
so a rollout does not restart every running backend once.
This matters more with #11024 pending: making GPU backends visible to
the upgrade checker will have AutoUpgradeBackends fan upgrades out to
worker nodes at scale, and every one of those is a reinstall. Left as
is, a rare manual-upgrade footgun becomes a fleet-wide one.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The num_ctx clamping specs added in #11032 construct their fixture with
`config.ModelConfig{ContextSize: &existing}`, but ContextSize is not a
direct field of ModelConfig: it belongs to LLMConfig, which ModelConfig
embeds inline. Go allows reading a promoted field but not setting one in
a composite literal, so the test file has never compiled:
helpers_internal_test.go:33:31: unknown field ContextSize in struct
literal of type "github.com/mudler/LocalAI/core/config".ModelConfig
This broke `make lint` on master from bf19758e0 onward, and because the
typecheck failure takes down the whole package it also reds tests-linux
and tests-apple on every PR branched after that commit.
Use the same literal form the rest of the tree already uses for this
field (see core/backend/options_internal_test.go).
Worth noting the specs were not merely uncompiled but inert: #11032 is a
DoS fix (an unauthenticated client raising the context ceiling drives
KV-cache allocation), and its regression guard was never actually
running. Verified the restored specs are functional by stubbing out the
ceiling clamp, which fails the "does not let an oversized num_ctx raise
an existing context ceiling" spec as intended.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
