A replica row in staging or loading holds its slot, because slot
allocation counts every state except unloading. Nothing ever reclaimed
such a row: every reconciler pass and the router's eviction query filter
state = "loaded", and the per-model probe skips rows without an address,
which is exactly what a row that never finished loading has.
So a worker that dropped out mid-transfer left a row that pinned the
only replica slot for that model on that node. Scheduling then found no
free slot and eviction found nothing it was allowed to evict, and the
request failed with "no replica slot on <node> and eviction failed: all
models busy". The state persisted until an operator intervened.
The reconciler now reclaims a row stuck before serving when no load job
is driving it. Ownership is decided by the job's LastProgress heartbeat,
not by elapsed time: staging a large checkpoint legitimately runs for a
long while without touching the replica row, so a deadline would either
be a model-size cliff or reclaim a healthy transfer. That heartbeat is
the same signal job takeover already trusts. Any error reading the job
leaves the slot held, because holding one for another pass costs a
scheduling opportunity while a wrong reclaim restarts a multi-gigabyte
transfer.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
A node's status comes from its HTTP heartbeat. Backend installs travel
over NATS. The two are independent, so a worker that dies stops
answering on the bus at once but stays healthy in the database until its
heartbeat ages out. Inside that window the scheduler picked a node it
could not reach, and the request failed with "no responders available"
rather than moving to a node that was up.
The scheduler now probes the node it selected and, when nothing answers,
marks it unhealthy and selects again. The demotion is what makes the
retry terminate: the next selection reads only healthy nodes. It also
tells the other frontends what this one learned, so the cluster does not
rediscover a dead worker one failed request at a time.
Only nats.ErrNoResponders counts as absent. A worker that answers slowly
stays eligible, because dropping it would cost capacity that is really
there. The probe reuses the models.running subject: a new subject would
go unanswered by workers that have not been upgraded, and every one of
them would then look dead.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
Picks up "surface option errors instead of discarding them". New records
the error from applying its options and Run returns it, so a process
whose state directory cannot be created reports the real cause instead
of failing later inside os.MkdirAll("") with an empty path.
LocalAI already resolves that directory itself, so this covers the other
process.New call sites rather than changing behaviour on the backend
start path.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5
process.New applies its options and discards the error they return. When
WithTemporaryStateDir could not create a directory, StateDir stayed
empty and every later option went unapplied, so the failure surfaced
from Run as "mkdir : no such file or directory" naming no path.
That message cost a full day of diagnosis on a worker whose volume was
full: the real errno was ENOSPC and nothing reported it.
The loader now creates the directory itself and returns the underlying
error with the path attached.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
A request that carries a file stages it to the worker, which writes it
under its staging directory. Nothing removed it afterwards. The frontend
expires ephemeral keys from object storage, but that sweep never covered
a worker's local disk, so every image, audio clip and video a worker
ever served stayed on it.
One worker had accumulated 175 request directories over three months.
The volume reached 100 percent, and from that point every backend start
failed because the process manager could not create a state directory.
The worker now sweeps its ephemeral staging directory on a timer and
once at startup, so files left by a crash are reclaimed too. Staged
model files live beside that directory and are not touched.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
"stale model config revision" reported only that two hashes differed.
It named neither, so an operator could not tell an edited configuration
from a revision that is not reproducible for one unchanged file, and the
failing value appears in no table.
The error now carries the revision the request brought and the one the
controller holds. It still wraps ErrStaleModelConfigRevision, so callers
that classify the error keep working.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
Add the 2B and 4B distilled models alongside the 9B model. Their compact GGUF builds make the Qwen3.8 reasoning distill practical on smaller hosts.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
LiquidAI now publishes official DSpark draft sidecars for its 2.6B target. Pair Q4 and Q8 targets with matching draft choices so LocalAI can use speculative decoding across different memory budgets.
Assisted-by: Codex:gpt-5 [systematic-debugging]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
syncKnownUsecasesFromString rebuilds KnownUsecaseStrings by ranging
GetAllModelConfigUsecases, which is a map. Go randomizes that order per
call, and the field is part of the serialized config, so one unchanged
YAML hashed to a different config revision on every load.
A model that derives a single usecase hid the problem. One that derives
several, such as a chat model with an mmproj, alternated between as many
revisions as there are orderings. The router treats a revision it did
not establish as a config change, so requests failed with "stale model
config revision" until the stored value happened to match again.
Sorting the list makes the revision a function of the file alone.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
The request middleware merges the caller's prediction parameters into
its copy of the model config. core/backend.ModelOptions then hashed
that copy, so the revision identified the request body rather than the
persisted configuration.
EstablishModelConfigRevision stores the first revision it sees and
requires an exact match afterwards. The first request after a restart
therefore pinned the model to its own temperature, top_p and stop
values, and every later request that sent different ones failed with
"stale model config revision". No config edit was involved.
The loader now stamps the revision when it materializes a config,
before any request override reaches it, and ModelOptions reads that
stamp. Model administration keeps hashing the same persisted config, so
both paths agree on one revision per configuration.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
* fix(vram): persist remote probe metadata
The startup warmer repeated remote size and GGUF metadata probes after every restart because both caches lived only in memory. Store successful HTTP probes for 24 hours so frequent restarts reuse the prior results.
Bound the cache, reject invalid records, and purge it when gallery data changes. Local model files continue to bypass persistence.
Assisted-by: Codex:gpt-5
* fix(vram): check temporary file cleanup
The lint gate rejects the unchecked cleanup call in the persistent cache writer.
Assisted-by: Codex:gpt-5.6 [golangci-lint]
* fix(vram): make persistent cache optional
Remote metadata probes can transfer enough data that operators need
control over disk reuse and startup warming. Gallery autoload now gates
both behaviors, and the runtime setting applies changes immediately.
Assisted-by: Codex:gpt-5
* fix(ui): expose gallery startup pre-warm
The existing gallery autoload setting also gates the startup metadata warmer. Name both effects in Settings so operators can find the requested boot control.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* docs: design configurable copy buffering
Document the context-aware copy buffer option and its validation plan.
Assisted-by: Codex:gpt-5
* docs: design durable distributed staging operations
Assisted-by: Codex:gpt-5
* docs: design distributed model config revisions
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
* feat(config): add stable model revisions
Hash typed model configuration and effective protobuf options deterministically for distributed revision comparisons.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
* feat(worker): acknowledge exact model stops
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
* feat(nodes): track model config revisions
Assisted-by: Codex:GPT-5 [apply_patch]
* fix(distributed): retry quarantined model cleanup
Stop quarantined replicas by exact process identity, retain failed cleanup as durable capped retries, and compare-and-delete only the claimed registry row. Process one sufficiently leased row at a time so multiple frontends cannot duplicate slow cleanup work.
Assisted-by: Codex:gpt-5
* fix(distributed): bind loads to config revisions
Assisted-by: Codex: GPT-5 [OpenAI Codex]
* fix(modeladmin): apply config revisions consistently
Route model edits, patches, state changes, deletion, and peer refreshes through the same revision lifecycle. Quarantine stale replicas before exact cleanup and report durable pending cleanup without failing successful config writes.
Assisted-by: Codex: GPT-5 [OpenAI Codex]
* feat(distributed): expose model config revision state
Document replica revision observability and durable cleanup behavior. Keep pending cleanup explicit in model mutation responses and verify endpoint contracts expose revision state without serialized load options.
Assisted-by: Codex:GPT-5 [OpenAI Codex]
* test(distributed): cover model revision convergence
Exercise cross-frontend quarantine, stale replay rejection, exact cleanup retry, worker re-registration, and current-generation replica convergence against the distributed PostgreSQL harness.
Assisted-by: Codex:gpt-5
* fix(distributed): pass config revision CI checks
Keep configured gallery sources out of authoritative runtime snapshots only after validating their real schema, and harden rollback snapshots against symlink races and non-regular files.
Assisted-by: Codex: GPT-5 [OpenAI Codex]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The Qwopus variant pair was added twice, which makes the later pair unreachable and fails the gallery invariant on master.
Keep the first pair that LocalAI already resolves and remove the dead duplicate declarations.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Keep the homepage focused on runtime capabilities and move engine details to their canonical directory. Make installation choices stable and explicit for users across supported hardware.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-bot <306113404+localai-org-bot@users.noreply.github.com>
* docs: design configurable copy buffering
Document the context-aware copy buffer option and its validation plan.
Assisted-by: Codex:gpt-5
* docs: design durable distributed staging operations
Assisted-by: Codex:gpt-5
* fix(distributed): merge durable staging operations
Use active model load jobs as the durable operations baseline and overlay replica-local staging progress without duplication. Preserve tracker-only operations when the registry cannot be read.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The gallery ranker reads serving features from tags instead of backend options. Mark two entries that enable draft MTP so ranking can identify them correctly.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix: GHSA-qwww-vcr4-c8h2 security vulnerability
Automated dependency upgrade by OrbisAI Security
Signed-off-by: anupamme <mediratta@gmail.com>
* fix: upgrade react-router-dom to 7.18.2 to fully remediate GHSA-qwww-vcr4-c8h2
The prior fix pinned react-router@7.18.2 directly but left react-router-dom
at ^7.18.1, which bun resolved to 7.18.1. That package bundles its own
react-router@7.18.1 sub-dep, leaving the vulnerable version in bun.lock
via the react-router-dom/react-router scoped resolution. Pinning
react-router-dom to 7.18.2 and regenerating the lockfile removes all
7.18.1 resolutions.
Assisted-by: Claude Code:claude-sonnet-4-6
Signed-off-by: Anupam Mediratta <mediratta@gmail.com>
---------
Signed-off-by: anupamme <mediratta@gmail.com>
Signed-off-by: Anupam Mediratta <mediratta@gmail.com>
* docs(blog): add the 4.9 release post and its demo clips
The 4.9 cycle changed how you authenticate, how chat handles a history
that no longer fits, and where models and backends live in the UI. The
release notes list every pull request; this post covers the three
changes that alter day-to-day use, and leads with the auth one because
it needs an action before upgrading.
Two clips are recorded from a real session against a local-ai built
from master with the live gallery loaded: model-lifecycle.mp4 walks
the unified models and backends pages, import-model.mp4 shows the
rebuilt import form. Both follow the clip conventions in
.agents/preparing-a-release.md: h264, no audio track, 1000x562, under
30 seconds, and named after the feature so they stay reusable.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Playwright]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(blog): anti-slop pass over the 4.9 post
Ran the post through the humanizer and no-ai-slop rules, calibrated
against what-landed-in-localai-4-8.md. That post is the one #11324
left unchanged, so it sets the rate for this series.
The draft ran denser than it on two constructions: "rather than" at
5.1 per thousand words against 3.5, and "instead of" at 3.1 against
1.6. Both are now at or below the 4.8 rate, 2.7 and 1.5, by rewriting
seven of them as plain statements.
Also cut: "keeping both cost a mode switch", a ledger metaphor for
something that is not money, which is the tell #11324 removed eight
times from the APEX post. "A follow-up fixed the thing that made that
awkward", an unearned framing plus a colon reveal. "This release adds
a different one: compress them", a second colon reveal. And
"byte-structurally identical", a second exactness idiom in a post that
already uses "byte-identical" where the precision carries weight.
Five paragraphs opened with "Two things" or "Two details", so three of
them start differently now. The summary listed three items, which is
the rule of three; it lists four, like the 4.8 summary.
Every figure, PR number, link and media reference is unchanged, checked
by diffing them out of both revisions. Hugo builds clean and the
rendered HTML has no em dashes.
Assisted-by: Claude Code:claude-opus-5 [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Expose the upstream Q8_0 build and the MTP drafter for the existing Gemma 4 12B agentic entry. The variants let LocalAI select higher fidelity or faster speculative decoding from the same model family.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update NVIDIA/NeMo-Speech.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(nemo-speech-cpp): build required sentencepiece
The updated runtime uses SentencePiece for RNNT context biasing even when Flashlight and text normalization are disabled. Build upstream's pinned static archive before configuring the engine so every backend variant has the newly required library and headers.
Assisted-by: Codex:gpt-5
* fix(nemo-speech-cpp): support BSD install
The pinned SentencePiece script uses GNU install -D for license files. macOS provides BSD install, which rejects that flag after the library compiles.
Create the license directory and use the portable install form on Darwin.
Assisted-by: Codex:gpt-5
* fix(nemo-speech-cpp): support CMake 4
Homebrew CMake 4 rejects the old policy floor in the pinned SentencePiece helper. Select the oldest supported policy floor for the Darwin dependency build.
Assisted-by: Codex:gpt-5.6 [Codex CLI]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add the official Q4_K_M and Q8_0 GGUF builds with the shared BF16 vision projector.
Assisted-by: Codex:gpt-5 [Hugging Face API]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add Tencent’s compact translation model in Q4 and Q8 GGUF formats. The linked variants let LocalAI select the best build that fits each host.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
