A quantization job that runs with no client on its progress stream stays
"queued" forever, in the API and in state.json, while the finished artifact
sits on disk. state.json was written once by StartJob, and the only code that
advanced a job afterwards lived inside the stream callback of StreamProgress,
so job state depended on somebody watching it.
The backend's progress stream cannot simply gain a second reader: each job owns
one queue.Queue and QuantizationProgress pops from it, so two consumers split
the updates rather than both seeing them. The stream has to be opened exactly
once per job.
StartJob now starts watchProgress on the application context (the request
context is done as soon as the handler returns). That goroutine is the single
reader: it applies each update to the job -- in the cross-replica store and in
state.json, terminal statuses still winning over late updates -- and republishes
it in-process. StreamProgress becomes a pure reader over that fan-out and no
longer loads a backend or opens a stream. A client attaching to a job that has
already finished, including one hydrated from disk after a restart, gets a final
event built from the stored job instead of blocking.
Two paths used to end a client's stream by breaking the gRPC connection and now
release it explicitly: StopJob kills the backend, so it publishes the stopped
event itself; and a stream that ends without a terminal update means the backend
is gone, so the job is recorded as failed rather than left running forever.
Signed-off-by: Tai An <antai12232931@outlook.com>
The recommended sampling parameters for a model family were applied at
install and then never took effect. Two things went wrong on the way to
disk.
They were written as top level keys. ModelConfig embeds PredictionOptions
under the "parameters" yaml key, so temperature, top_p, top_k, min_p,
repeat_penalty and presence_penalty are only read from there. At the top
level they parse without error and are then ignored for the life of the
model.
They were also merged in after the YAML had already been marshalled. The
only re-marshal sat behind the artifact binding, which an entry carrying
files: never reaches, so for those entries the defaults were computed and
then dropped before anything was written.
Neither failure was visible in normal use. ApplyInferenceDefaults runs
again at load time and fills the same values from the same table, so the
model ends up tuned correctly while the file on disk pins nothing. It
surfaces when someone edits one of those values expecting it to win, or
when a family is absent from inference_defaults.json and there is nothing
to refill from.
Both install paths are covered: an entry carrying files:, and one that
binds a primary artifact instead.
The empty base spec asserted that the authored parameters block landed
verbatim. It now checks the authored keys individually, because the family
defaults are merged into that same block.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
Web Chat Settings left the System Prompt field empty but still treated a
blank/whitespace value as an explicit system turn. That satisfied
tokenizer chat templates' messages[0].role == system check and suppressed
the model YAML system_prompt on fresh chats.
Omit empty/whitespace system messages in the React and Alpine UIs, strip
them server-side, and inject config.SystemPrompt for tokenizer-template
models when the request has no real system turn.
Fixes#11834
Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
* fix(distributed): evict only when a node is known to be full
scheduleNewModel asked the registry for a free replica slot and treated
every error as "this node is full", so a control-plane database slow
enough to time out the lookup evicted a healthy loaded model. The
evicted process died, a peer frontend still holding its address dialled
the dead port and retried, and the model thrashed between nodes. The
comment on the branch already said it meant a full node; the code never
tested for it.
Evict only on ErrNoFreeSlot. Any other error now returns and names the
lookup that failed, so a slow database degrades into a diagnosable
load failure instead of into lost work.
An audit of the rest of the router found one branch of the same shape:
node selection discarded the error from its last-resort finder, so a
database timeout there also produced a nil node and evicted for it.
That path now returns unless the finder said gorm.ErrRecordNotFound,
which is the only answer that means the cluster had no node to give.
No other destructive branch in router.go fires on a generic error.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): checkpoint heartbeat writes instead of writing every beat
Every heartbeat UPDATEd backend_nodes. Six nodes at a ten second beat is
roughly 52,000 writes a day against a six-row table, and that churn is
what turned a blocked autovacuum into a 460 MB table whose six-row scan
cost 867 ms and timed out the queries that place models.
A beat carrying only a fresher timestamp now waits for the checkpoint
interval. Each reported field is compared against the value last
persisted rather than tested for presence, because a worker sends its
disk figures on every beat and presence alone would suppress nothing.
A node's first beat, a changed total VRAM, total disk or GPU vendor,
and a free VRAM, RAM or disk reading that has moved more than 256 MiB
from the persisted value all still write at once. A node that is not
active is never suppressed, because it recovers only when the health
monitor sees a fresh timestamp.
The persisted column is up to one interval stale by design, so the
stale-node threshold moves from 60s to 5m to cover it.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): fail worker readiness when a held backend is unreachable
The readiness gate tracked only the NATS link, so a worker whose backend
processes had died still answered /readyz with 200 and kept receiving
loads. One node did exactly that during an incident: it reported healthy
while its backend port refused connections, and every load routed to it
failed.
Readiness is now the NATS link and, for each backend process the worker
believes it is running, a short dial of its recorded address. A worker
holding no backends stays ready, because idle is a healthy state.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): keep a starting backend out of the readiness dial set
A backend process is inserted into the supervisor map with its gRPC
address already recorded, but the address refuses connections until the
gRPC server binds, which the startup poll allows up to 30 seconds for and
which takes 10 to 15 seconds on a slow node. The new data-path readiness
probe dialled that address straight away, so a worker answered /readyz
with 503 for the whole of every cold backend start. The container
HEALTHCHECK absorbs that, but a Kubernetes readinessProbe at 10s does
not, and the worker would leave rotation each time it loaded a model.
The skip for a stopping process had no counterpart at the other end of
the lifecycle. Backend processes now carry a serving flag, set where the
startup health-check gate succeeds, and the probe dials only processes
that are serving and not yet stopping. backendStartStillValid becomes
markBackendServing: the check and the mark must share one lock hold, so
the flag can only ever land on the entry the key currently owns.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(distributed): export control-plane database health gauges
Four transactions wedged on a corrupt index held the vacuum horizon open
for 42 days. Nothing measured it, so the first symptom anyone saw was
models failing to load six weeks later, by which time a six-row table
had grown to 460 MB.
Export the oldest xmin age, the longest open transaction, and the dead
tuple ratio on the registry tables. The first is the number that would
have caught it: it sits near zero in health and was 21,002,291.
Sampling is scrape-driven behind a cache, and a failed sample reports
the last good values rather than failing the scrape, because these
gauges matter most when the database is already struggling.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): rate-limit failed control-plane database samples
The cache advanced its clock only on a successful sample, so once the
database started failing every scrape retried the query immediately.
That turned the cache off in the one regime it exists for: a retry
storm at scrape cadence aimed at a database already in trouble. A
catalog read that consistently exceeds the 5 second timeout also paid
that cost on every scrape, with all scrapes serialised behind the
sampler mutex.
Time every attempt rather than every success, so failures and timeouts
cost the same interval as good samples. Whether a good sample exists
moves to its own field, keeping the gauges absent until the first
success and holding the last good values through later failures.
Also note in the runbook that pg_stat_activity cannot see prepared
transactions or replication slot xmins, so a healthy-looking xmin age
does not by itself rule out a blocked horizon.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(distributed): pin that a failing database evicts nothing
Exercises the real distributed stack against a control-plane database that
refuses the router's slot lookup, and asserts the scheduler reports the
lookup it could not answer instead of falling through to eviction.
The failure is injected with privileges rather than a statement timeout. A
timeout set with ALTER DATABASE also breaks AutoMigrate, and it leaks into
every later spec in the suite unless it is reset, so the spec would end up
testing the migration rather than the scheduler. Instead the spec creates a
dedicated login role, points a second gorm handle at it, and revokes that
role's SELECT on node_models.replica_index. This has to be a separate role:
the test container's owner is a PostgreSQL superuser, and superusers bypass
every privilege check, so revoking from CURRENT_USER is recorded and then
ignored.
The revoke is scoped to one column on purpose. Revoking the whole table
would also blind node selection, which runs first and has a guard of its
own, so the scheduler would never reach the slot lookup this spec is about.
Leaving every other column readable lets selection succeed and lands the
refusal exactly on NextFreeReplicaIndex, which plucks replica_index. The
grant is restored from BeforeEach via DeferCleanup, so a failing assertion
or a panic cannot hand the next spec a role that cannot read.
Reverting the eviction guard fails this spec, which is the point of it: the
router then reports "no replica slot on keeper and eviction failed" for an
error that was never evidence the node was full. The surviving-row
assertions are secondary under this injection, because the eviction path
reads whole node_models rows and the same revoke blinds it too; a comment
in the spec says so, so nobody mistakes them for the load-bearing ones.
Also documents why the vector store and the control plane must not share a
database: the removable-tuple cutoff is per database, not per table, so one
transaction left open anywhere stops autovacuum reclaiming the node
registry, and a six-row table bloats into hundreds of megabytes. The note
names LOCALAI_AUTH_DATABASE_URL and LOCALAI_AGENT_POOL_DATABASE_URL as the
two knobs that must differ, and the localai_control_plane_oldest_xmin_age
gauge as the way to see it coming.
grep for StaleNodeThreshold and HealthCheckInterval in
core/config/runtime_settings_registry.go returns no matches: the
distributed duration knobs are not exposed as runtime settings, so the new
heartbeat checkpoint interval follows them and needs no registry entry.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): close the review gaps in the heartbeat and health path
The stale-node threshold moved from 60 seconds to 5 minutes in this branch
because checkpointing makes last_heartbeat up to one checkpoint interval
behind by design. Two things were left inconsistent with that. NewHealthMonitor
still fell back to a hardcoded 60 seconds when handed a zero threshold, so any
future caller that stopped passing the configured value would mark every
healthy, beating node offline on every cycle. And the threshold itself had a
flag-name constant but no AppOption, no CLI field and no env binding, so an
operator who widened --node-heartbeat-checkpoint had no way to widen the
threshold to match. The fallback now tracks config.DefaultStaleNodeThreshold,
and --stale-node-threshold / LOCALAI_STALE_NODE_THRESHOLD is wired the same
way its sibling is.
Heartbeat suppression compared the RAW reported free VRAM against the
snapshot, but the column persists capAvailable(raw, ceiling). On any node with
a VRAM budget set, whose actual free VRAM oscillates above that ceiling, every
beat looked material while the persisted value never moved: suppression was
defeated on exactly the nodes an operator had configured, and the write
amplification this branch exists to remove came straight back there. The
comparison and the snapshot now both hold the capped figure, so they measure
the same quantity as the column.
Fixing that needs the ceiling, and reading it cost a SELECT on every beat,
including suppressed ones. The skip decision therefore moved ahead of the
updates map and now reuses the ceiling cached on the last durable write, while
the write path still re-reads it before capping anything. A ceiling that
changed inside the checkpoint window can cost one extra or one late write; it
cannot persist a wrong figure. A suppressed beat now costs no query at all.
Also: the operations section now says to grant pg_read_all_stats to the
LocalAI role, because PostgreSQL blanks backend_xmin and xact_start for
sessions owned by other roles, and the transaction that wedged the horizon in
the incident was a co-located vector store connecting as a different role, so
without the grant the new gauge sees only our own sessions. The compose
healthcheck comment now describes readiness covering the backend data path,
and the control-plane gauge registration records the otel.SetMeterProvider
ordering it depends on.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): resolve the gauge's table names through gorm
The dead-tuple gauge queried pg_stat_user_tables against a hardcoded list
of three table names. Those three do not agree on where their name comes
from: BackendNode and NodeModel take gorm's default pluralisation, while
GalleryOperationRecord overrides TableName, and gallery_operations
already had a constant of its own that the list duplicated.
A literal list keeps compiling after any of that moves, and the query
then matches nothing. The failure is silent and it points the wrong way:
a dead-tuple ratio that matched no rows reports the same numbers as a
cluster with no bloat, so the gauge would look healthiest exactly when it
had stopped working.
Ask gorm what each model is stored as instead, which follows a TableName
override and the default pluralisation alike. A spec pins that the
override really is consulted: naive pluralisation of the type would give
gallery_operation_records, so the resolution cannot quietly stop asking
the model.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
stageDirectory and countStageableFiles already skip them, but
stageOptionDir did not - and it is the path sherpa-onnx voices take for
espeak-ng-data. The receiver writes "<file>.sha256" for every file it
accepts, so staging the sidecars made it write sidecars for those in
turn, one level deeper on every load.
Observed on a live node: "<file>.sha256" repeated eleven times, 5077
junk files out of 7832 in the models dir, and still growing. Staging
never finished, so vits-piper-it_IT-paola-sherpa stayed permanently
"staging on node" and every realtime warmup needing that voice failed
with the session then going silent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0142UfUh8HWxdim5JZqf8Tr6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Install 404s because the gallery still points at mmproj-...-f16.gguf.
HF only ships ...-F16.gguf now, with a different sha256.
Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
OpenAI GA clients send multipart or raw SDP requests. They expect a bare
SDP answer. LocalAI only accepted its legacy JSON envelope, so signaling
failed before media setup.
Keep the JSON contract for existing clients. Accept both GA request
shapes and choose the matching response format.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Node placement and replica rules could only name a model, so an operator
who pinned "llama3" to the GPU tier had to rewrite the rule whenever a
different model took over that job. An alias already gives a stable name
for whichever model serves it, and a rule on that name makes it a
deployment slot: repoint the alias and the placement follows.
A rule keeps the name the operator chose. Reads resolve that name through
the config loader to the model the rule governs, so the reconciler counts,
schedules and trims replicas of the target, and the router finds an
alias-keyed rule from the target it is already routing. An alias that
resolves to nothing governs nothing loadable, so the reconciler skips it
and the write paths refuse it.
A replica is shared by every name that resolves to it, so only one rule
can decide where it runs. The REST and MCP write paths reject a rule whose
target another rule already governs. A pair that arrives some other way,
such as a seed file or an alias repointed onto a model that already has a
rule, resolves in favour of the rule named after the model itself and then
the oldest, and the rest are listed as shadowed.
The eviction guard is the exception: it matches rules to replicas in raw
SQL inside a locking transaction and cannot resolve an alias. It reads a
stored target that the reconciler refreshes each tick, and falls back to
the rule's own name when that target is empty.
Assisted-by: Claude:claude-opus-5 golangci-lint eslint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ui): move node labels into the scheduling selector field
The scheduling page kept a node-label browser open above the rules
whether or not anyone was writing one, while the field that actually
needs labels, the rule's node selector, was two bare text inputs with no
hint of what the cluster reports.
The browser is gone. The selector's key input now completes against the
label keys the cluster uses, and the value input offers only the values
that key takes. The roster already loads for the page, so the
suggestions cost no request, and a roster that fails to load costs the
admin the hints and nothing else.
Suggestions stay suggestions: a key no node reports yet still commits as
typed, which is how an admin writes a rule before labelling the nodes
for it.
Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): size model fit against the cluster, not the frontend
The models page asked the frontend how much memory a model may occupy.
In distributed mode the frontend is usually a GPU-less pod while every
model runs on a worker, so a fleet of GPU nodes was told it could only
run the smallest CPU build. The variant picker's fits flag and its
auto-selection came from the same place, as did the hardware
recommendations.
The registry now reports the largest single healthy backend node. The
largest node, not the fleet total: a model loads into one node, so four
16GB workers are not a home for a 40GB model. An operator-set VRAM
budget caps a node's contribution, because the scheduler refuses a load
above that ceiling anyway, and a GPU node beats a CPU node holding more
system RAM.
GET /api/resources and GET /api/models carry this as an additional
cluster object. Their aggregate and ram fields keep reporting the
frontend's own hardware, which is what the resource monitor shows.
Variant selection judges backends against the union of the capabilities
present in the cluster, the way backend discovery already did.
Every path degrades to the local host: no cluster object in single-node
mode, and none when the registry cannot be read, so a hiccup narrows the
answer back to single-node behaviour rather than marking the whole
catalog too large.
The verdicts now name the node they belong to, since a model fits
somewhere or nowhere.
Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
/api/tags appends :latest to untagged names, but chat and the other
model endpoints looked the tagged name up as-is and 404'd.
Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
The scheduler's liveness probe asks a worker a question over NATS and
reads "no responders" as proof the worker is gone. That is only sound
when every worker in the fleet subscribes to the subject asked.
It asked models.running, which arrived in 4.6. A 4.5 worker is alive and
serving, answers backend.list, and never subscribes to models.running,
so the probe condemned it on every scheduling attempt and marked it
unhealthy. A model pinned to such a node by its selector could then
never be placed at all: on this cluster an embedding model pinned to the
one Apple node was unschedulable for exactly this reason, while that
node's log showed it handling backend.list throughout.
Ask backend.list, which has been in the worker protocol far longer, and
treat a worker that answers anything as alive. Only a node that reports
no responders on every subject is absent, so adding a newer subject here
can never condemn an older worker.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
A model's revision is published by administration and checked against on
every inference request. Those were computed by separate code: the
request path resolves through the loader, while each publisher hashed
whatever ModelConfig it happened to hold. By then SetDefaults had folded
in the GGUF guess and app-level options, so the published value was one
no request would ever carry and the model became unroutable until the
row was deleted by hand.
Fixing the publishers one at a time did not hold. Three rounds each
found another: the startup resync, then a saved edit and a toggle, then
a rename and the peer-change path.
ModelConfigLoader.RevisionFor is now the only way to obtain a revision,
and the raw hash is unexported, so a caller outside this package cannot
hash a config it holds. A publisher and a request agree by construction
rather than by two implementations happening to match.
The request path no longer falls back to hashing its merged config
either: an unstamped config is routed without a revision rather than
with a wrong one.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
When no node the selector allows has a free slot, scheduling falls back
to evicting the least-recently-used idle model. That eviction searched
every healthy node, so it freed a slot on a node the selector forbids
and the model was then placed there: pinned to one class of hardware and
running on another.
An unrelated model pays for it. On this cluster an embedding model
pinned to Apple hardware could not reach its only matching node, so each
attempt evicted a large language model from an Nvidia node, failed to
start there anyway, and left the evicted model to reload. Repeated, that
reads as one replica bouncing between nodes.
Eviction is now restricted to the candidate set the selector produced.
With no selector the candidate set is nil and eviction stays global.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
The abandoned-load sweeper treated a replica row with no load job as
abandoned. Only the request path creates load jobs; the reconciler's own
scale-up loads a replica without one. So any scale-up that ran past the
five-minute grace period was deleted mid-transfer, which for a
multi-gigabyte checkpoint is every time. The replica never finished
anywhere, and the reconciler kept re-placing it, so it looked like one
replica hopping between nodes instead of a model reaching its replica
count.
A row with no job is now reclaimed only once its node stops being
healthy, which is the case the sweeper was written for: a worker that
dropped out mid-transfer. A job that failed or stopped heartbeating
still proves abandonment on its own. Every uncertain case leaves the
slot held.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
Only backend workers subscribe to backend.list. ListBackends asked every
node that was not pending, offline or draining, so an agent worker could
only answer "no responders", which the error handling reads as a node
that has gone away. Every poll of the backends view therefore marked
each agent node unhealthy, and its next heartbeat marked it healthy
again.
While unhealthy the node is not schedulable, so this also cost agent
capacity for as long as each flap lasted.
Skip non-backend workers, as the backend-op fan-out already does for the
same reason. A backend worker that does not answer is still marked
unhealthy: that one really is gone.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
The revision was computed after SetDefaults, which folds in things that
are not persisted configuration: the GGUF guess, the hardware defaults,
and app-level options such as threads.
The GGUF guess is the damaging one. It parses the model file to fill in
values like context size, and when that parse fails it falls back to a
different default. Whether a multi-gigabyte file on network storage
parses at a given moment is not a property of the configuration, so one
unchanged YAML produced two different revisions depending on when it was
read. The controller rejected every request carrying the other one, and
the model stayed unroutable until the stored value happened to match
again. This is why it never reproduced against a model directory with no
weights in it: the guess is skipped there and both values agree.
The app-level defaults are the same class of bug with a slower fuse:
changing threads in the settings UI changed every model's revision and
made every model unroutable.
The revision is now stamped when the file is parsed, before any defaults
are applied, so it is a function of the file alone.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
Two code paths computed a model's revision. Inference resolves the
config through the loader, which applies SetDefaults a second time.
Everything that publishes a revision hashed the stored config instead,
with SetDefaults applied once.
SetDefaults is not idempotent for every model: it re-runs the GGUF guess
and the hardware defaults, both of which read state the stored config
does not carry. Where the two disagree, a publisher wrote a revision no
request would ever carry, and the model became unroutable the moment it
was published. On this cluster the startup resync republished one such
value and every request for that model was then rejected against it.
The publishers now resolve the revision through the loader, exactly as a
request does, so there is one definition rather than two that agree only
when SetDefaults happens to be idempotent. This covers the startup
resync, a saved config edit, and enabling or disabling a model.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
* fix(ci): bound Discord release summaries
The release model can return more than Discord's 2,000-character
message limit. Discord then rejects the entire release notification.
Ask the model for a smaller response and truncate extracted content to
1,800 characters before the notification step. The smaller bound leaves
room below Discord's hard limit when model output varies.
Assisted-by: Codex:gpt-5
* fix(tests): implement node liveness stub
NodeCommandSender now requires PingNode. The endpoint test stub must
implement it before the package can compile.
Assisted-by: Codex:gpt-5 [Codex]
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The resync added in 3953448f6 ran before LoadModelConfigsFromPath, so it
read an empty loader, reconciled nothing and reported success. The
symptom was a stored revision that stayed stale across restarts while
the log showed no complaint, which is exactly what the resync was meant
to prevent.
Move the call after the configs are loaded, and refuse to treat an empty
loader as a clean run: reconciling zero models is indistinguishable from
reconciling correctly, and that is what hid the mis-ordered call.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
The controller pins a model's replicas to a stored revision and rejects
any request carrying a different one. Nothing ever re-derived that value
from the configuration on disk: it moved only on an edit, a gallery
install, or a peer's change broadcast. An inference request may only
establish a revision, never replace one.
So any other way for the two to diverge left the model permanently
unroutable. A configuration edited while a frontend was down lands
there, and so does a change in what the revision is computed over: an
upgrade that alters the hashed form leaves every stored revision
describing a configuration that no longer exists. The only recovery was
deleting the row by hand, which is not something a cluster should need.
Each frontend now reconciles the stored revisions against the loaded
configurations at startup and republishes the ones that disagree. Only
those: republishing quarantines every replica loaded under the old
revision, so doing it for a model that did not drift would unload a
healthy replica for nothing. A model with no stored revision has never
been served and is left for its first request to establish.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
A replica row in staging or loading holds its slot, because slot
allocation counts every state except unloading. Nothing ever reclaimed
such a row: every reconciler pass and the router's eviction query filter
state = "loaded", and the per-model probe skips rows without an address,
which is exactly what a row that never finished loading has.
So a worker that dropped out mid-transfer left a row that pinned the
only replica slot for that model on that node. Scheduling then found no
free slot and eviction found nothing it was allowed to evict, and the
request failed with "no replica slot on <node> and eviction failed: all
models busy". The state persisted until an operator intervened.
The reconciler now reclaims a row stuck before serving when no load job
is driving it. Ownership is decided by the job's LastProgress heartbeat,
not by elapsed time: staging a large checkpoint legitimately runs for a
long while without touching the replica row, so a deadline would either
be a model-size cliff or reclaim a healthy transfer. That heartbeat is
the same signal job takeover already trusts. Any error reading the job
leaves the slot held, because holding one for another pass costs a
scheduling opportunity while a wrong reclaim restarts a multi-gigabyte
transfer.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
A node's status comes from its HTTP heartbeat. Backend installs travel
over NATS. The two are independent, so a worker that dies stops
answering on the bus at once but stays healthy in the database until its
heartbeat ages out. Inside that window the scheduler picked a node it
could not reach, and the request failed with "no responders available"
rather than moving to a node that was up.
The scheduler now probes the node it selected and, when nothing answers,
marks it unhealthy and selects again. The demotion is what makes the
retry terminate: the next selection reads only healthy nodes. It also
tells the other frontends what this one learned, so the cluster does not
rediscover a dead worker one failed request at a time.
Only nats.ErrNoResponders counts as absent. A worker that answers slowly
stays eligible, because dropping it would cost capacity that is really
there. The probe reuses the models.running subject: a new subject would
go unanswered by workers that have not been upgraded, and every one of
them would then look dead.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
A request that carries a file stages it to the worker, which writes it
under its staging directory. Nothing removed it afterwards. The frontend
expires ephemeral keys from object storage, but that sweep never covered
a worker's local disk, so every image, audio clip and video a worker
ever served stayed on it.
One worker had accumulated 175 request directories over three months.
The volume reached 100 percent, and from that point every backend start
failed because the process manager could not create a state directory.
The worker now sweeps its ephemeral staging directory on a timer and
once at startup, so files left by a crash are reclaimed too. Staged
model files live beside that directory and are not touched.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
"stale model config revision" reported only that two hashes differed.
It named neither, so an operator could not tell an edited configuration
from a revision that is not reproducible for one unchanged file, and the
failing value appears in no table.
The error now carries the revision the request brought and the one the
controller holds. It still wraps ErrStaleModelConfigRevision, so callers
that classify the error keep working.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
syncKnownUsecasesFromString rebuilds KnownUsecaseStrings by ranging
GetAllModelConfigUsecases, which is a map. Go randomizes that order per
call, and the field is part of the serialized config, so one unchanged
YAML hashed to a different config revision on every load.
A model that derives a single usecase hid the problem. One that derives
several, such as a chat model with an mmproj, alternated between as many
revisions as there are orderings. The router treats a revision it did
not establish as a config change, so requests failed with "stale model
config revision" until the stored value happened to match again.
Sorting the list makes the revision a function of the file alone.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
The request middleware merges the caller's prediction parameters into
its copy of the model config. core/backend.ModelOptions then hashed
that copy, so the revision identified the request body rather than the
persisted configuration.
EstablishModelConfigRevision stores the first revision it sees and
requires an exact match afterwards. The first request after a restart
therefore pinned the model to its own temperature, top_p and stop
values, and every later request that sent different ones failed with
"stale model config revision". No config edit was involved.
The loader now stamps the revision when it materializes a config,
before any request override reaches it, and ModelOptions reads that
stamp. Model administration keeps hashing the same persisted config, so
both paths agree on one revision per configuration.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
* fix(vram): persist remote probe metadata
The startup warmer repeated remote size and GGUF metadata probes after every restart because both caches lived only in memory. Store successful HTTP probes for 24 hours so frequent restarts reuse the prior results.
Bound the cache, reject invalid records, and purge it when gallery data changes. Local model files continue to bypass persistence.
Assisted-by: Codex:gpt-5
* fix(vram): check temporary file cleanup
The lint gate rejects the unchecked cleanup call in the persistent cache writer.
Assisted-by: Codex:gpt-5.6 [golangci-lint]
* fix(vram): make persistent cache optional
Remote metadata probes can transfer enough data that operators need
control over disk reuse and startup warming. Gallery autoload now gates
both behaviors, and the runtime setting applies changes immediately.
Assisted-by: Codex:gpt-5
* fix(ui): expose gallery startup pre-warm
The existing gallery autoload setting also gates the startup metadata warmer. Name both effects in Settings so operators can find the requested boot control.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* docs: design configurable copy buffering
Document the context-aware copy buffer option and its validation plan.
Assisted-by: Codex:gpt-5
* docs: design durable distributed staging operations
Assisted-by: Codex:gpt-5
* docs: design distributed model config revisions
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
* feat(config): add stable model revisions
Hash typed model configuration and effective protobuf options deterministically for distributed revision comparisons.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
* feat(worker): acknowledge exact model stops
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
* feat(nodes): track model config revisions
Assisted-by: Codex:GPT-5 [apply_patch]
* fix(distributed): retry quarantined model cleanup
Stop quarantined replicas by exact process identity, retain failed cleanup as durable capped retries, and compare-and-delete only the claimed registry row. Process one sufficiently leased row at a time so multiple frontends cannot duplicate slow cleanup work.
Assisted-by: Codex:gpt-5
* fix(distributed): bind loads to config revisions
Assisted-by: Codex: GPT-5 [OpenAI Codex]
* fix(modeladmin): apply config revisions consistently
Route model edits, patches, state changes, deletion, and peer refreshes through the same revision lifecycle. Quarantine stale replicas before exact cleanup and report durable pending cleanup without failing successful config writes.
Assisted-by: Codex: GPT-5 [OpenAI Codex]
* feat(distributed): expose model config revision state
Document replica revision observability and durable cleanup behavior. Keep pending cleanup explicit in model mutation responses and verify endpoint contracts expose revision state without serialized load options.
Assisted-by: Codex:GPT-5 [OpenAI Codex]
* test(distributed): cover model revision convergence
Exercise cross-frontend quarantine, stale replay rejection, exact cleanup retry, worker re-registration, and current-generation replica convergence against the distributed PostgreSQL harness.
Assisted-by: Codex:gpt-5
* fix(distributed): pass config revision CI checks
Keep configured gallery sources out of authoritative runtime snapshots only after validating their real schema, and harden rollback snapshots against symlink races and non-regular files.
Assisted-by: Codex: GPT-5 [OpenAI Codex]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* docs: design configurable copy buffering
Document the context-aware copy buffer option and its validation plan.
Assisted-by: Codex:gpt-5
* docs: design durable distributed staging operations
Assisted-by: Codex:gpt-5
* fix(distributed): merge durable staging operations
Use active model load jobs as the durable operations baseline and overlay replica-local staging progress without duplication. Preserve tracker-only operations when the registry cannot be read.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix: GHSA-qwww-vcr4-c8h2 security vulnerability
Automated dependency upgrade by OrbisAI Security
Signed-off-by: anupamme <mediratta@gmail.com>
* fix: upgrade react-router-dom to 7.18.2 to fully remediate GHSA-qwww-vcr4-c8h2
The prior fix pinned react-router@7.18.2 directly but left react-router-dom
at ^7.18.1, which bun resolved to 7.18.1. That package bundles its own
react-router@7.18.1 sub-dep, leaving the vulnerable version in bun.lock
via the react-router-dom/react-router scoped resolution. Pinning
react-router-dom to 7.18.2 and regenerating the lockfile removes all
7.18.1 resolutions.
Assisted-by: Claude Code:claude-sonnet-4-6
Signed-off-by: Anupam Mediratta <mediratta@gmail.com>
---------
Signed-off-by: anupamme <mediratta@gmail.com>
Signed-off-by: Anupam Mediratta <mediratta@gmail.com>
* test: make coverage failures observable
Keep per-root logs, reject concurrent coverage runs, and avoid relying on /bin/sleep in the worker timeout test.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: parallelize coverage without remote fixtures
Assisted-by: Codex:gpt-5 [apply_patch] [exec_command]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: add offline resource infrastructure
Introduce versioned resource manifests, a checksum-verified CAS preparer, offline test wrappers, and a guarded network transport. Replace live Hugging Face, GitHub, and OCI cases with deterministic fixtures and inject fixture metadata into importer discovery.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: enforce offline resource replay
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: harden offline resource refresh
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: expose slow coverage waits
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: eliminate avoidable wall-clock waits
Inject a clock into Hugging Face retry handling, reuse a process-scoped PostgreSQL container with per-spec schemas in the nodes suite, and poll local import jobs promptly.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: remove repeated fixture startup waits
Share PostgreSQL fixtures across parallel endpoint and agent suite workers, and make the worker Free deadline injectable so the wedged-backend test does not spend five seconds on wall-clock time.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: fix offline resource CI portability
Normalize Docker archive metadata before content addressing, derive archive checksums during explicit refreshes, make network lint portable to macOS, and prepare distributed images before running their offline suite.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* ci: cache Go modules before offline tests
Warm the complete module graph before the Linux and macOS test jobs enter offline replay mode, so tool dependencies such as Ginkgo are not fetched through the guarded proxy.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: drop the static network lint in favour of real isolation
The offline test suite already prevents tests from reaching the network
twice over: run-test-linux-offline.sh puts the test process in a cgroup
and REJECTs egress outside the private ranges, and HardenedTransport
installs testnetwork.LocalGuard to refuse dials that resolve to a public
address. Both fail the test with a precise error at the moment of the
dial.
test-network-lint.sh added neither. Its diff stage defaulted to a HEAD
base, so on a clean checkout it compared the tree against itself and
inspected nothing; the branch's own commits were never examined. It only
produced output when an earlier job step dirtied the tree, and then it
matched a bare https?:// against whatever changed. make react-ui runs
npm install rather than npm ci, so CI rewrote
core/http/react-ui/package-lock.json and the lint reported an npm
registry URL as forbidden test network access:
+ "resolved": "https://registry.npmjs.org/hono/-/hono-4.12.25.tgz",
Its fingerprint stage was self-defeating in a quieter way: hashing the
whole tree's network-mechanism inventory meant every rebase onto a master
that touched any _test.go needed a manual baseline bump, so the check
mostly caught its own staleness.
Remove the script, its make target and the two prerequisite edges, along
with the test-network: fixture markers that existed only to suppress it.
The isolation itself is untouched.
Assisted-by: Claude:claude-opus-5 [go vet]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* ci: keep hidden files in the offline test bundle artifact
Cherry-picked from 15a37b0ac on the remote branch. The offline bundle lives
under .cache/, which actions/upload-artifact skips by default, so the Linux
job packed an artifact missing the very file the next step restores.
The other half of 15a37b0ac moved test-network-lint out of the `test` and
`test-coverage` prerequisite lists into a recipe line, so parallel make could
not fingerprint the tree while generated fixtures were still changing. That
is dropped: the preceding commit removes the lint entirely, and the race it
worked around is one more reason a whole-tree fingerprint was the wrong
mechanism.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* refactor: share bounded exponential backoff
Use overflow-safe saturating arithmetic for retry delays across model import polling, downloads, registration, node operations, and model loading. Keep model import status checks responsive initially while capping their interval at 500ms.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* ci: mirror Jetson Python wheels
Keep the CUDA aarch64 wheel subset in GHCR and serve it as a local PEP 503 index during L4T backend builds, preserving last-known-good packages through upstream outages.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* docs(agents): index the Jetson wheels mirror
Mention the GHCR-hosted L4T wheel mirror in the CI caching guide summary so maintainers can find its outage and cache documentation.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* ci: add defensive build network proxy
Record build destinations and byte counts, retry observable idempotent HTTP downloads, and isolate explorer database tests that race under coverage.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(kokoros): implement updated backend trait
Return unimplemented for image upscaling, matching the backend's other unsupported modalities after the protobuf API update.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(ci): clear recovered proxy errors
Do not mark a request failed when a later safe retry succeeds.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* ci: require HTTPS build interception
Inject a short-lived proxy CA into BuildKit and Dockerfile RUN steps, reject plain HTTP and opaque tunnels, and retain method/status/byte telemetry for verified HTTPS traffic.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(ci): preserve system trust in unproxied builds
Mount the generated interception CA at a dedicated secret path and add it to the trust bundle only in proxy-aware dependency stages. This prevents optional secret mounts from masking the system CA bundle in ordinary backend test builds.
Install the requested Go toolchain before starting the proxy and satisfy cleanup error checks found by CI lint.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(ci): persist build proxy trust
Install the generated proxy CA through the system-managed local certificate directory so ca-certificates upgrades retain it. Avoid turning canceled matrix jobs into proxy cleanup failures.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(ci): trust proxy in nested build scripts
Install the build proxy CA before nested source fetches, route the DS4 package setup through the HTTPS mirror helper, and avoid repeated OCI setup in gallery behavior tests.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(ci): use HTTPS apt sources for Bonsai
Rewrite ARM64 package sources before installing GCC and check gallery fixture cleanup errors so the optimized tests satisfy errcheck.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(privacy-filter): trust build proxy CA
Install the mounted build proxy certificate before privacy-filter's make target fetches its HTTPS sources, for both source and prebuilt builder paths.\n\nAssisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* test: fail on hidden offline egress
Count cgroup-scoped firewall rejects and fail the offline test harness with bounded aggregate diagnostics. Inject the gen-audio GGUF probe so fixture-backed importer tests do not attempt real network access.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(ci): preserve system CA trust
Build a combined runner certificate bundle instead of replacing public roots with the generated proxy CA. Centralize additive container installation in the shared proxy CA helper.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The V8 coverage build inlines every dynamic import, so the stale chunk tests cannot intercept a page chunk. Run those tests against the normal code-split bundle and exclude them from the inlined coverage pass.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
A deploy replaces the whole content-hashed asset set at once. A tab
holding an older index.html, or one whose request lands on a replica
that the rollout has not swapped yet, asks for a page chunk the server
no longer has. The dynamic import rejects and React Router's default
error boundary replaces the app with "Unexpected Application Error!"
until someone reloads by hand.
The router now reloads the page itself when a chunk fails to load.
index.html is served no-cache, so the reload lands on a self-consistent
asset set. A timestamp in sessionStorage bounds this to one reload per
10 seconds, so a chunk that is genuinely gone reaches the error boundary
instead of looping forever.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5[1m]
* feat(config): add context compression policy
Define the opt-in model configuration contract before the chat middleware consumes it. Document each policy field so later request handling does not invent a second schema.\n\nRefs #9534\n\nAssisted-by: Codex:gpt-5
* fix(config): register compression fields
The model editor metadata gate rejects new config fields without descriptions and suitable controls. Register the compression policy so operators can edit its six fields safely.
Assisted-by: Codex:gpt-5 [monitoring-prs]
* feat(chat): compress long contexts
Long conversations currently fail once they reach the model context window. The opt-in policy now summarizes complete older turns before primary inference and preserves the newest tool chains.
Both OpenAI and MCP chat routes share the same transformation. Usage metadata and metrics expose each compression event.
Refs #9534
Assisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
