* fix(ui): move node labels into the scheduling selector field
The scheduling page kept a node-label browser open above the rules
whether or not anyone was writing one, while the field that actually
needs labels, the rule's node selector, was two bare text inputs with no
hint of what the cluster reports.
The browser is gone. The selector's key input now completes against the
label keys the cluster uses, and the value input offers only the values
that key takes. The roster already loads for the page, so the
suggestions cost no request, and a roster that fails to load costs the
admin the hints and nothing else.
Suggestions stay suggestions: a key no node reports yet still commits as
typed, which is how an admin writes a rule before labelling the nodes
for it.
Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): size model fit against the cluster, not the frontend
The models page asked the frontend how much memory a model may occupy.
In distributed mode the frontend is usually a GPU-less pod while every
model runs on a worker, so a fleet of GPU nodes was told it could only
run the smallest CPU build. The variant picker's fits flag and its
auto-selection came from the same place, as did the hardware
recommendations.
The registry now reports the largest single healthy backend node. The
largest node, not the fleet total: a model loads into one node, so four
16GB workers are not a home for a 40GB model. An operator-set VRAM
budget caps a node's contribution, because the scheduler refuses a load
above that ceiling anyway, and a GPU node beats a CPU node holding more
system RAM.
GET /api/resources and GET /api/models carry this as an additional
cluster object. Their aggregate and ram fields keep reporting the
frontend's own hardware, which is what the resource monitor shows.
Variant selection judges backends against the union of the capabilities
present in the cluster, the way backend discovery already did.
Every path degrades to the local host: no cluster object in single-node
mode, and none when the registry cannot be read, so a hiccup narrows the
answer back to single-node behaviour rather than marking the whole
catalog too large.
The verdicts now name the node they belong to, since a model fits
somewhere or nowhere.
Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): follow upstream MTMD APIs
The dependency update adds MTMD initialization options to prompt and
bitmap helpers. The gRPC adapter now passes the server options through
each affected path.
The update also replaces the per-layer MoE regex helper. Preparation
probes both APIs because older forks still reuse this adapter.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
go-m1cpu v0.1.6 runs its cgo initialiser from a package init(), where
getFrequency() dereferences the CFTypeRef returned by
IORegistryEntryCreateCFProperty without a NULL check. On Apple M5 the
pmgr IORegistry node does not expose voltage-states5-sram /
voltage-states1-sram in the shape v0.1.6 expects, so the call returns
NULL and CFDataGetLength(NULL) faults before main() runs. Every command
dies, including local-ai --version.
The package is linked indirectly: cmd/local-ai reaches
gopsutil/v3/{process,disk}, which pull in gopsutil/v3/cpu on darwin,
which calls m1cpu.IsAppleSilicon() and m1cpu.PCoreHz().
v0.2.2 adds the missing NULL guard and moves the IORegistry probe out of
init() behind a lazy sync.Once. The exported Go API is unchanged and the
non-darwin stub is byte-identical, so gopsutil/v3 compiles against it
untouched and no other platform is affected.
Bumping gopsutil/v3 is not an alternative: v3.24.5 is the final v3
release, so the v3 line will never carry this fix.
Fixes#11735
Assisted-by: Claude:claude-opus-5
Signed-off-by: Szymon Podeszwa <2962046+sz-po@users.noreply.github.com>
* ⬆️ Update mudler/vllm.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(vllm-cpp): mirror ABI v23
The new engine pin reports ABI v23 and appends mmproj_path to
vllm_model_params. LocalAI still declares v21, so the build-time ABI
guard rejects every backend build.
Grow the Go mirror by the appended pointer and update its offset checks.
ABI v23 adds a video function but does not change the mirrored text
structs.
Assisted-by: Codex:gpt-5.6 [systematic-debugging]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
/api/tags appends :latest to untagged names, but chat and the other
model endpoints looked the tagged name up as-is and 404'd.
Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
* feat(gallery): add Ornith 1.5 397B variants
Add the official Q4_K_M and Q8_0 GGUF builds with their shared BF16 vision projector.
Assisted-by: Codex:gpt-5
* feat(gallery): resolve Ornith variant ordering\n\nKeep the 35B entries from master next to the 397B variants.\n\nAssisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): adapt to the common JSON API
The llama.cpp bump replaces its nlohmann JSON alias with common_json. Update the gRPC adapter for the new exception, iterator, conversion, and container APIs.
Assisted-by: Codex:gpt-5.6 [systematic-debugging]
* fix(turboquant): adapt the JSON exception type
The shared gRPC source now follows the upstream common_json API. The
TurboQuant fork still exposes nlohmann JSON and cannot compile the new
exception type.
Translate that exception in the fork-specific source patch so both
llama.cpp variants compile from the shared adapter.
Assisted-by: Codex:gpt-5.6 [systematic-debugging]
* fix(bonsai): adapt the JSON exception type
The shared gRPC source uses upstream's common_json wrapper. The Bonsai fork still exposes nlohmann JSON and cannot compile that exception type.\n\nTranslate the exception in the fork-specific preparation step and verify that repeated preparation stays idempotent.\n\nAssisted-by: Codex:gpt-5.6 [systematic-debugging]
* fix(llama-cpp): let prepare register gRPC
The score patch duplicated the gRPC CMake registration that prepare.sh already owns. Its stale context rejects the current upstream tools file on Darwin before compilation starts.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Those backends only forwarded the flag when it was "true", so "false"
never reached apply_chat_template and Qwen3 kept thinking on.
Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
The scheduler's liveness probe asks a worker a question over NATS and
reads "no responders" as proof the worker is gone. That is only sound
when every worker in the fleet subscribes to the subject asked.
It asked models.running, which arrived in 4.6. A 4.5 worker is alive and
serving, answers backend.list, and never subscribes to models.running,
so the probe condemned it on every scheduling attempt and marked it
unhealthy. A model pinned to such a node by its selector could then
never be placed at all: on this cluster an embedding model pinned to the
one Apple node was unschedulable for exactly this reason, while that
node's log showed it handling backend.list throughout.
Ask backend.list, which has been in the worker protocol far longer, and
treat a worker that answers anything as alive. Only a node that reports
no responders on every subject is absent, so adding a newer subject here
can never condemn an older worker.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
A model's revision is published by administration and checked against on
every inference request. Those were computed by separate code: the
request path resolves through the loader, while each publisher hashed
whatever ModelConfig it happened to hold. By then SetDefaults had folded
in the GGUF guess and app-level options, so the published value was one
no request would ever carry and the model became unroutable until the
row was deleted by hand.
Fixing the publishers one at a time did not hold. Three rounds each
found another: the startup resync, then a saved edit and a toggle, then
a rename and the peer-change path.
ModelConfigLoader.RevisionFor is now the only way to obtain a revision,
and the raw hash is unexported, so a caller outside this package cannot
hash a config it holds. A publisher and a request agree by construction
rather than by two implementations happening to match.
The request path no longer falls back to hashing its merged config
either: an unstamped config is routed without a revision rather than
with a wrong one.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
When no node the selector allows has a free slot, scheduling falls back
to evicting the least-recently-used idle model. That eviction searched
every healthy node, so it freed a slot on a node the selector forbids
and the model was then placed there: pinned to one class of hardware and
running on another.
An unrelated model pays for it. On this cluster an embedding model
pinned to Apple hardware could not reach its only matching node, so each
attempt evicted a large language model from an Nvidia node, failed to
start there anyway, and left the evicted model to reload. Repeated, that
reads as one replica bouncing between nodes.
Eviction is now restricted to the candidate set the selector produced.
With no selector the candidate set is nil and eviction stays global.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
The abandoned-load sweeper treated a replica row with no load job as
abandoned. Only the request path creates load jobs; the reconciler's own
scale-up loads a replica without one. So any scale-up that ran past the
five-minute grace period was deleted mid-transfer, which for a
multi-gigabyte checkpoint is every time. The replica never finished
anywhere, and the reconciler kept re-placing it, so it looked like one
replica hopping between nodes instead of a model reaching its replica
count.
A row with no job is now reclaimed only once its node stops being
healthy, which is the case the sweeper was written for: a worker that
dropped out mid-transfer. A job that failed or stopped heartbeating
still proves abandonment on its own. Every uncertain case leaves the
slot held.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
Only backend workers subscribe to backend.list. ListBackends asked every
node that was not pending, offline or draining, so an agent worker could
only answer "no responders", which the error handling reads as a node
that has gone away. Every poll of the backends view therefore marked
each agent node unhealthy, and its next heartbeat marked it healthy
again.
While unhealthy the node is not schedulable, so this also cost agent
capacity for as long as each flap lasted.
Skip non-backend workers, as the backend-op fan-out already does for the
same reason. A backend worker that does not answer is still marked
unhealthy: that one really is gone.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
The revision was computed after SetDefaults, which folds in things that
are not persisted configuration: the GGUF guess, the hardware defaults,
and app-level options such as threads.
The GGUF guess is the damaging one. It parses the model file to fill in
values like context size, and when that parse fails it falls back to a
different default. Whether a multi-gigabyte file on network storage
parses at a given moment is not a property of the configuration, so one
unchanged YAML produced two different revisions depending on when it was
read. The controller rejected every request carrying the other one, and
the model stayed unroutable until the stored value happened to match
again. This is why it never reproduced against a model directory with no
weights in it: the guess is skipped there and both values agree.
The app-level defaults are the same class of bug with a slower fuse:
changing threads in the settings UI changed every model's revision and
made every model unroutable.
The revision is now stamped when the file is parsed, before any defaults
are applied, so it is a function of the file alone.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
Two code paths computed a model's revision. Inference resolves the
config through the loader, which applies SetDefaults a second time.
Everything that publishes a revision hashed the stored config instead,
with SetDefaults applied once.
SetDefaults is not idempotent for every model: it re-runs the GGUF guess
and the hardware defaults, both of which read state the stored config
does not carry. Where the two disagree, a publisher wrote a revision no
request would ever carry, and the model became unroutable the moment it
was published. On this cluster the startup resync republished one such
value and every request for that model was then rejected against it.
The publishers now resolve the revision through the loader, exactly as a
request does, so there is one definition rather than two that agree only
when SetDefaults happens to be idempotent. This covers the startup
resync, a saved config edit, and enabling or disabling a model.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
quay.io redirects blob downloads to pre-signed S3/Akamai URLs that
expire after about 10 minutes. On a slow connection a multi-GiB
backend layer cannot finish inside that window, so the connection
drops mid-stream on every attempt. The retry added for #10577
restarted each attempt from byte zero, which replayed the same
failure until the budget ran out and the install failed with
"unexpected EOF".
A retry now keeps the bytes already on disk and re-requests the
blob with "Range: bytes=N-". Each request goes back to the
registry, so it gets a fresh redirect URL and auth token. The
retry budget only counts attempts that made no forward progress,
so a slow link that keeps advancing keeps downloading. A resumed
file is spliced from separate responses and bypasses the digest
check in layer.Compressed(), so the assembled file is re-verified
against the layer digest before it is trusted; on a mismatch the
download starts over through the verified reader.
Fixes#10577
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): add Qwen3.8 OBLITERATED variants
Add Q4_K_M and Q8_0 llama.cpp builds with the shared BF16 vision projector.
Assisted-by: Codex:gpt-5
* fix(tests): implement node liveness stub
NodeCommandSender now requires PingNode. The endpoint test stub must
implement it before the package can compile.
Assisted-by: Codex:gpt-5 [Codex]
* fix(distributed): restore node liveness tests
The router now probes models.running before it schedules work. The E2E
workers only mocked backend.install, so every test node appeared offline.
The endpoint test double also missed the new PingNode method and stopped
the Linux, Apple, and lint jobs during compilation.
Mock the existing worker reply in both distributed fixtures and keep the
endpoint test double aligned with NodeCommandSender.
Assisted-by: Codex:gpt-5 [golangci-lint]
* fix(tests): check node liveness replies
The liveness test subscriptions ignored setup and reply errors.
Errcheck rejected each branch that carried them.
Assisted-by: Codex:gpt-5 [golangci-lint]
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(distributed): restore node liveness tests
The router now probes models.running before it schedules work. The E2E
workers only mocked backend.install, so every test node appeared offline.
The endpoint test double also missed the new PingNode method and stopped
the Linux, Apple, and lint jobs during compilation.
Mock the existing worker reply in both distributed fixtures and keep the
endpoint test double aligned with NodeCommandSender.
Assisted-by: Codex:gpt-5 [golangci-lint]
* fix(tests): check node liveness replies
The liveness test subscriptions ignored setup and reply errors.
Errcheck rejected each branch that carried them.
Assisted-by: Codex:gpt-5 [golangci-lint]
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
