The spec promised a load-time warning when a chain marks a remote
target warm, where the flag does nothing; the loader now logs it. The
remote-backend test moves into ModelConfig.IsRemoteProxy so the loader
and the failover manager agree on what is remote.
The spec now says what ships: a load blocked by pinned warm targets
proceeds over the limit after eviction retries, without an error that
names them.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Go resends custom headers such as x-api-key when it follows a redirect,
also to another host, so a redirecting upstream could receive the
target's API key elsewhere. The probe client now treats a redirect as
the response, which fails the probe as a non-2xx status.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Transcription-only and sound-detection-only realtime sessions passed a
chain config straight to the model loader. It has no backend, so the
loader fell back to greedy backend auto-detection: slow, and ending in
an unhelpful error. Sound-only sessions are a main use of chains.
The stage routing of the full pipeline moves into a stageRouter that
both realtime model kinds embed. Every stage resolves to the chain's
active target at build time and goes through the failover plan per
call. The session sends failover events for any model with chain
stages, and restarts them when a transcription session.update swaps
the model.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A chain request reached a cloud-proxy target with the client's model,
the chain name, whenever the target set no upstream_model: passthrough
forwards the body's model and translate falls back to it. The upstream
answered 404, which neither retries nor trips, while the liveness
probe, which checks the target's own name, kept passing.
PrepareTarget now sets the upstream model of a remote target to
proxy.upstream_model or the target name, the same name the probe uses.
The request pipeline and realtime chain stages both call it.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A warm target's liveness probe called ModelLoader.Load, which blocked
until the model finished loading (while the warm preload loaded it
too). Tick waited for every probe, so all probing froze, and the probe
then ran HealthCheck on an expired context and tripped the target at
every startup.
The prober now takes a function that returns the running backend
without loading it. A target that is not loaded passes liveness; its
recovery is neither confirmed nor failed and it returns to healthy
after min_dwell, like a cold target. Tick no longer waits for probes:
each probe applies its own result and a target whose probe is running
is skipped.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The request path called HasChains on every request, and with no chains
it scanned the config source each time: the loader's lock plus a copy
and sort of every config, forever, on every installation without
chains. Sync now keeps an atomic flag and HasChains reads only that.
A chain added since the last sync is still served because Plan syncs
on a miss; only in-request retry waits for the next tick (at most 1s).
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A stage that names a chain is resolved on every call, so a switch keeps
the session and its conversation. Clients get localai.model.failover
events at session start and on every switch.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A missing file is not a reliable signal: models download on first use,
some backends need no file, and dotted names like Phi-3.5-mini look like
paths. Marking such a fallback down removed the retry a chain exists
for. Cold targets are now judged only by real requests.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
An admission rejection or a disabled target moves the request to the
next target through Attempt.Skip, which records no failure. A 4xx
response no longer counts as a success. Requests skip body recording
when no chain is configured, and stop it once the model is known not
to be a chain.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The retry wraps SetModelAndConfig, so each attempt binds the request
again from a replayed body. A 5xx of a chain request is held back until
the handler returns, and a streamed response is never retried.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
applyFailoverWarmTargets ran on the manager's single scheduler goroutine
(Sync -> Tick), so a slow or hung PreloadModelByName call froze probing
and fail-back for every chain. Keep the watchdog pin synchronous but run
the preload loop in its own goroutine. Adds a seam (preloadModelByName)
so a unit test can substitute a blocking loader and assert the callback
still returns promptly.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Warm local targets are pinned in the watchdog and preloaded. Switches
and target health are exported as metrics, skipped attempts as traces.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Matches cloud-proxy's resolveAPIKey exactly: os.Getenv + empty check
rather than os.LookupEnv, so a variable that's set but empty errors
instead of probing unauthenticated.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Remote liveness uses /v1/models, which every OpenAI-compatible upstream
serves. Recovery sends one minimal request for the target's usecase.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Idle targets get a liveness probe each interval, recovering targets an
inference probe. Cold local targets are never loaded to be probed.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
recomputeLocked only fired the event on an active-target change or on
entering degraded. When the active target itself recovered while every
target was down, the chain silently left degraded with no event, so
SSE/realtime consumers tracking chain.switched.state got stuck on
"degraded".
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
errcheck flagged two side-effect-only m.Plan calls in tests, and unused
flagged close(), which Task 5's probe scheduler wires in.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Health is tracked per target and the active target per chain. Fail-back
waits for recovery probes and a minimum time on the fallback.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Reject chains whose targets are missing or are chains, at load and on
create or edit, and warn when the targets share no usecase.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
A chain is a model config with an ordered list of target models, probe,
trip and recovery settings. Like an alias it has no backend.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): read metadata for system-path backends, enabling variant aliases
Problem:
- ListSystemBackends only read metadata.json for user-managed backends;
the system-path scan (LOCALAI_BACKENDS_SYSTEM_PATH) was a bare
directory walk with Metadata hardcoded nil
- system-packaged backends (distro packages installing several
accelerator builds of one backend) could not declare aliases or meta
indirection at all, while gallery-installed backends could
- surfaced while packaging LocalAI for Gentoo: the packages install
cpu-/rocm-/vulkan-audio-cpp as system backends aliased to audio-cpp,
which the server ignored
Change:
- scan each root separately, clean the system collection against the
user-managed one, merge, then build and resolve — precedence lives in
one explicit step
- alias candidates carry their own metadata: the resolved alias entry
can never pair one installation's executable with another's metadata,
and it reports the chosen candidate's origin (IsSystem)
- deterministic resolution: entries build in sorted name order and
candidates sort by name at the resolution site, independent of scan
order
Precedence (user-managed always wins):
- a user-managed backend hides a same-named system backend entirely
- a user-managed variant takes over its whole alias family: the alias
resolves among user-managed variants only and the system family's
concrete names disappear — family versions move together, and a stale
system variant may not work with newer models, so it must not stay
reachable
- a system variant's alias never hijacks a name that exists as a
user-managed backend
Tests: Ginkgo regressions for system-path aliasing, same-name hiding,
family takeover, and the full metadata permutation matrix of
cross-root name collisions (both directions, with and without
metadata on each side).
Docs: new "Backend Directory Format" section (run.sh, metadata.json,
alias resolution — previously undocumented for user-managed backends
too) and "System-Provided Backends" with the precedence rules.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
* fix(gallery): preserve managed meta backends
A system alias can replace a user-managed meta backend during discovery.
Protect meta entries with the same precedence guard as concrete backends.
Add a regression test and clarify the documented precedence.
Assisted-by: Codex:GPT-6
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
---------
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(vllm-cpp): unify decision pipeline through Score RPC with vllm_decide ABI v29
Replace the model-specific SystemOne gRPC approach with a generic Score
RPC extension. The pre-existing Score RPC (previously unused by any
backend) now carries question_type and response_json fields:
- question_type="systemone" routes kev/laya decision-pipeline requests
through the unified vllm_decide C ABI (v29), returning the full
response JSON in response_json.
- question_type empty routes cua-s1-forms candidate scoring through the
same vllm_decide ABI, returning CandidateScore probabilities.
The vllm-cpp backend's Score() method calls vllm_decide and dispatches
by architecture internally. The /v1/systemone HTTP endpoint checks
whether the model's backend supports Score; if so, it forwards the raw
request JSON and returns the backend response as-is. Other backends
fall through to the existing NER-based path.
This mirrors the vllm.cpp C ABI refactor (PR #3301) that replaced
vllm_systemone + vllm_score with a single vllm_decide function. The
purego bindings bump abiVersion from 27 to 29 and resolve vllm_decide
and vllm_decide_free symbols.
Also fixes validModelPath to accept cua-s1-forms.json and
rl_agent_config.json alongside config.json, matching the engine's
model_loader.cpp config-filename ordering.
AI-Assisted: true
Assisted-by: Maki:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore: ⬆️ Update mudler/vllm.cpp to e28ec46c6 (fix macOS -Werror build)
Bumps vllm.cpp to e28ec46c6 which fixes a -Wnull-conversion error in
qwen3_5.cpp:12483 that broke the macOS Metal CI build under -Werror.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
llama.cpp picks a new random media marker per backend process. LocalAI
cached the first probe on the model config and skipped later probes when
MediaMarker was non-empty, so after SINGLE_ACTIVE eviction/reload the
prompt still used the stale marker and mtmd_tokenize failed (0 markers
vs 1 bitmap).
Re-probe whenever the model was not already resident before Load, while
still skipping the RPC on warm cache hits.
Fixes#12246
Assisted-by: Cursor:composer-2.5
Signed-off-by: leilei3167 <imleilei123@gmail.com>
* fix(gallery): verification follow-ups for oci:// galleries
Follow-ups from the post-merge review of #12238 and #12239.
Only a policy decision is a refusal now. cosignverify wraps
ErrPolicyRejected around a failed signature check, an identity or
source-repository mismatch, a not_before cutoff and a missing or
unparseable bundle. A TUF, registry or network failure during
verification, or a timeout, is an outage: the gallery falls back to the
copy verified under the current policy, as it does when the registry is
down.
An oci:// gallery with a verification block, or any oci:// gallery under
strict integrity, is no longer answered by an https://, github: or
file:// mirror. Such a mirror is ignored with a warning, because nothing
can check its signature. The index of an HTTP gallery, whose policy only
covers its backend images, is cached under the URL-only name again, so no
unchecked body is stored under a policy-keyed name.
The in-memory index cache key now includes the policy. After a runtime
policy change the index is fetched again, and entries with a relative url
install again.
The registry digest lookups after install and upgrade, and in the
upgrade check, run only for real registry references (new
URI.LooksLikeRegistryOCI), not for ollama:// or ocifile://.
The refusal message names strict integrity when that is the cause, and
the gallery name is no longer repeated.
Specs pin the URL-only cache name for galleries without a policy, a fixed
key for a fixed policy, and that every GalleryVerification field changes
the key. The docs describe refusal, outage, mirrors and strict integrity.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): reset listings on gallery changes, classify referrer outages
Review follow-ups for this PR.
The React UI lists from AvailableGalleryModelsCached, which is keyed by
nothing. A gallery change through the settings API or a
runtime_settings.json edit now drops that listing when the model or
backend gallery configuration differs. Before, the UI kept the old list,
with local paths into the old policy's tree, until the next background
refresh, or for good when the new policy refused the gallery.
In cosignverify, a referrer the registry fails to serve now makes the
lookup an outage whatever other referrers failed and in any order, since
the unread one may be the valid signature. An invalid policy (Validate in
NewVerifier, an unparseable not_before) is ErrPolicyRejected, because no
fetch can make it usable.
The docs say that only an oci:// gallery with a verification block skips
non-OCI mirrors, and list an unusable policy as a refusal.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The unpacked oci:// gallery cache and the last known good copy of the
index were keyed on the gallery URL only. After an operator tightened a
gallery's verification policy (added source_repository, moved not_before
forward), content verified under the older policy, or under none, was
still served for up to an hour from the unpacked cache, and indefinitely
from the last known good copy while fetches failed. A fetch refused by
signature verification also fell back to that last known good copy, so
a refusal became a silent downgrade. Turning strict integrity on did not
stop an unverified cached copy from being served either.
Name both caches by the URL plus a stable hash of the policy. A gallery
without a policy keeps its old URL-only name, so existing caches stay
usable. A fetch refused by the policy, or by strict integrity, is now
reported and never answered with a cached copy; a network failure still
falls back, but only to a copy verified under the current policy. The
strict integrity check runs before the cache is read.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
A backend installed from an oci://host/repo:tag or oci://host/repo@sha256
URI was downloaded correctly, but the digest lookup that follows the
install (and the one after an upgrade) passed the raw URI to the registry
client. The client does not know the oci:// scheme: with a port in the
host it failed to parse the reference, without one it read "oci" as the
registry host and queried https://oci/v2/. The install still succeeded,
so the only trace was a warning and an empty digest in metadata.json,
which made the next upgrade check report an upgrade for no reason.
Add downloader.URI.OCIReference and route every consumer that hands an
OCI URI to a registry client through it: the install and upgrade digest
lookups, the upgrade check, the OCI download path, the oci:// gallery
fetch and the llama.cpp importer.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): add GLiNER2.5 NER via TokenClassify
Wire the vllm-cpp backend to the C ABI NER surface (vllm_gliner_ner,
ABI v27) so LocalAI can serve zero-shot named entity recognition through
the existing TokenClassify gRPC method.
backend.go: TokenClassify method on *VllmCpp calls vllm_gliner_ner with
the text and labels, copies the C-owned entity array into protobuf
TokenClassifyEntity messages, and frees the result.
govllmcpp.go: cNerEntity and cNerResult Go POD mirrors matching the C
structs; vllmGlinerNer and vllmNerResultFree purego bindings; abiVersion
bumped 26 -> 27.
options.go: ner_labels, ner_threshold, ner_max_width parsed from
engine_args.
pkg/grpc: ClassifyModel interface and TokenClassify server handler
(follows the Embedding locking pattern).
core/config: vllm-cpp backend declares MethodTokenClassify and
UsecaseTokenClassify.
docs/content/features/vllm-cpp.md: NER section documenting the
engine_args keys and the host-forward contract.
Assisted-by: MAKI:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): correct NER pointer lint directive
Use the govet directive for the C-owned NER array, matching the other
purego pointer conversions. The array remains valid until its deferred
free; the misspelled directive caused CI to flag this conversion.
Assisted-by: Codex:gpt-6 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): add kev-compatible SystemOne API endpoints
Add POST /v1/systemone, /v1/systemone/permute, and
/v1/systemone/separate to LocalAI, mirroring the kev project's
structured-extraction API. Each endpoint runs zero-shot NER over the
rendered state text and builds kev-compatible answers for three question
types: noul (binary entity presence), choice (pick one option), and
score (pick one level).
The TokenClassifyRequest proto gains a `repeated string labels` field so
each question can supply its own labels at inference time, and
TokenClassifier gains TokenClassifyWithLabels for per-call label
selection. The vllm-cpp backend uses request labels when non-empty,
falling back to configured ner_labels then the built-in defaults.
Helpers (renderState, softmax, choiceConfidence, scoreConfidence, r2)
are ported from kev/api.py and mirrored in vllm.cpp's api_server.cpp so
both servers produce the same answer shape.
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): suppress gosec G404 on seeded permutation RNG
The SystemOne permute endpoint uses math/rand with a caller-supplied
seed for reproducible option permutations, matching kev's random.seed.
gosec flags this as G404 (weak RNG). Add #nosec with a comment naming
the intent: this is reproducibility, not cryptography.
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(vllm-cpp): bump vllm.cpp pin to GLiNER2.5 merge commit
Advance VLLM_CPP_VERSION from f3cd97e to 5058268d, the commit that
landed GLiNER2.5 zero-shot NER support (PR #3224) in vllm.cpp. This
brings the DeBERTa v2 encoder, GLiNER2 boundary head, C ABI NER
functions, and server endpoints into the LocalAI vllm-cpp backend.
The ABI version (27) and Go struct mirrors already match.
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): use instruction text as NER label in SystemOne handler
The SystemOne handler was passing question IDs as NER labels for noul
questions and bare key names for choice questions, so the model never
matched any entities. Port the label mapping from vllm.cpp's
ParseSystemOneBody:
- noul: use the rendered instructions field (with instr alias) as the
NER label, not the question ID
- choice: use optionText(name, desc) — "name: description" or "name"
when the description is null/empty — not the bare key
- score: already correct (rendered criteria text)
- permute: shuffle indices and build parallel key/label arrays so the
NER call uses the optionText labels while the response is keyed by
the original option names
Also add the instructions field to the SystemOneQuestion schema struct
(accepted alongside the instr backward-compat alias).
Verified end-to-end against the real GLiNER2.5 model: noul questions
now find "Apple Inc. is" (organization, 0.999) and "Tim Cook is"
(person, 0.852) where they previously returned zero entities.
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(capabilities): report the per-request context with split KV slots
With parallel slots and kv_unified:false, llama.cpp gives each slot
n_ctx/n_parallel, padded up to a multiple of 256. /models/capabilities
still reported the full n_ctx. A client that budgets a request against
context_size then overflows at a fraction of it.
EffectiveRequestContextSize returns the per-slot size in that case and
the full context otherwise. With the unified KV cache, the grpc-server
default, one request may use all of n_ctx. The capabilities endpoint
and the router's prompt trimmer now use it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(openai): return an HTTP error when a stream fails before any chunk
A streamed chat request set the SSE headers, then waited for the
backend. When the backend failed before the first token, LocalAI sent
a 200 with a `data: {"error":...}` chunk and [DONE]. Clients that do
not parse error chunks saw an empty reply. cogito's LocalAI client was
one of them: nib users got "streaming decision produced no content"
instead of the context overflow that caused it.
Nothing has been written at that point, so the handler now returns the
error as a normal HTTP response. A failure after the first chunk keeps
the in-stream error chunk.
A prompt that exceeds the context is now a 400 on both paths, as in
the OpenAI API and llama-server, and no longer a 500. The message is
kept whole, because clients read the token counts from it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(e2e): check the error from closing the response body
golangci-lint's errcheck flags the unchecked resp.Body.Close in the
new pre-stream error helper.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
feat(kimodocpp): add API and backend request observability
Capture animation requests, phase timings, output metadata, and failures in traces. Record correct API error statuses and cover completed, running, failed, and disabled tracing.
Assisted-by: Codex:GPT-6
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Track successful header authentication before allowing cross-site requests
to bypass CSRF checks. Arbitrary headers on unauthenticated servers and
cookie-authenticated requests no longer grant an exemption.
Share the production CSRF middleware with multipart tests, add regression
coverage for credential sources, and document the exemption behavior.
Assisted-by: Codex:gpt-6 golangci-lint
Signed-off-by: Richard Palethorpe <io@richiejp.com>
On a single-node install nothing in Operate listed the models loaded on
this machine or let an admin stop one. The System page that did was
retired in #11548, and its replacements (the Nodes workbench) only work
in distributed mode. The Nodes page also mis-detected single-node mode:
the cluster routes are not registered there, so /api/nodes answers 404,
but only 503 was treated as "distributed off", which sent every
single-node install to the empty worker-registration card. The rail hid
the entry anyway.
Nodes route on a single node becomes "This machine":
- the Nodes page's VRAM / RAM / CPU / models-disk gauges, fed from this
host by mapping /api/resources onto the worker heartbeat fields
- a memory bar splitting host RAM by running model
- a running-models table (backend, RSS, CPU share, uptime, PID) with
search, sorting, logs and a confirmed Stop
- the distributed setup behind an "Add machines" button
The Operate overview gains a "Running now" preview (heaviest five, with
Stop) on single node and a pointer to Nodes > Running models on a
cluster. The rail shows "This machine" in Runtime with a running count.
Backend, additive only:
- /system: each loaded model carries a `process` block (pid, rss_bytes,
memory_percent, cpu_percent, started_at). A sampler keeps one gopsutil
handle per PID so CPU is the share since the previous poll rather than
the lifetime average; it is omitted on the first reading.
- /api/resources: host `cpu` and models-path `disk`, the same readings
workers send in their heartbeat.
Also fixes the fleet tables widening the page on phones: the headers'
absolutely positioned sr-only labels escaped the scroll wrapper.
Assisted-by: Claude:claude-opus-5 [Playwright]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
An alias config is a pure redirect with no backend of its own, so the
capabilities listing described it from its stub: no capabilities, no
modalities, and the default 4096 context_size. Clients that size their
context budget from this endpoint (nib, for one) then compacted every
turn against a model that really serves 100k.
Resolve the alias and report the target's capabilities, modalities and
context_size under the alias's id. A dangling or chained alias now
reports no enrichment instead of defaults no model runs with.
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(oci): pull ORAS artifacts into a directory
Galleries published to an OCI registry arrive as an ORAS artifact whose
layers carry their tree path in org.opencontainers.image.title. Nothing in
pkg/oci could read a non-image manifest, so add PullArtifact: it lays the
layers out under a destination directory and returns the manifest digest so
callers can pin and verify signatures on it.
A registry is remote input and this writes files, so the manifest is
validated before any byte is fetched. A title that is absolute, escapes the
destination or is missing is refused, an artifact of an unexpected
artifactType is refused, and both the layer count and the total size are
capped with caller-settable limits.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(gallery): fetch and verify galleries published as OCI artifacts
A gallery whose URL is oci://host/repo:tag is now pulled as an ORAS
artifact and its index.yaml read from the unpacked tree, alongside the
existing HTTP and file sources. Mirrors, the per-candidate timeout, the
failure cooldown and the last known good copy work the same for both
schemes, so a gallery can mirror an OCI primary with an HTTP fallback.
When the gallery declares a verification policy the artifact reference is
resolved to its manifest digest, the publisher signature is checked
against that digest, and only then is the same digest pulled. Nothing
unverified is written to disk or parsed. With strict integrity mode on,
an OCI gallery without a verification policy is refused instead of
warned about, which is what the flag already does for backend installs.
The pull lands in a staging directory that is renamed into the cache only
once the whole tree is on disk and the index reads back as an index, so a
failed or interrupted pull leaves nothing a later fetch would serve. A
fetch that finds a fresh cache entry does not contact the registry.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(gallery): resolve entry urls relative to the gallery root
A gallery published as a self-contained tree, which is what an OCI
gallery is, names its base configs by their place in that tree. Such a
url reached the HTTP client verbatim and no entry of the gallery could
be installed.
A scheme-less entry url now resolves against the root the index came
from: the unpacked artifact for an oci:// gallery, and the directory of
the index URL otherwise. The relative path may not climb out of that
root, reusing the guard the artifact puller already applies to layer
titles, and an entry that does is dropped from the listing rather than
failing the whole gallery. An entry that names its own scheme is
untouched, so .ref indirection and local github: resolution behave as
before.
Reading an entry of an OCI gallery also needs the unpacked artifact
directory as its trusted root, since the cache is deliberately a sibling
of the models directory the downloader otherwise confines a file read
to.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(oci): restrict artifact file permissions
Create artifact files with owner-only permissions to satisfy G302.
Document the existing path validation for the G304 scanner finding.
Check the permissions of downloaded layers in the artifact test.
Assisted-by: Codex:gpt-6 gosec
* fix(oci): confine artifact writes to their root
Use root-relative filesystem operations to reject escapes through existing
directory symlinks. Retain owner-only permissions and add a symlink escape
regression test, without suppressing the path traversal finding.
Assisted-by: Codex:gpt-6
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Report input tokens and frame-step output units in response metadata and
record them through the existing usage accounting pipeline. Preserve the
accounting rule and model-specific dimensions as JSON without extending
the gRPC schema for each modality.
Expose animation usage only under metadata.usage, validate counts before
recording, and document the response contract and loaded-model location.
Add coverage for transport, defaults, failures, persistence, and recording
requests once with statistics enabled or disabled.
Assisted-by: Codex:GPT-6
Signed-off-by: Richard Palethorpe <io@richiejp.com>
GetImageDigest calls name.ParseReference, which cannot parse the oci://
scheme. Self-hosted registries must set the scheme to be recognised by
LooksLikeOCI, so without stripping it they silently lose upgrade detection.
Strips the prefix at the call site, matching what the other two call
sites already do (pkg/downloader/uri.go, gallery/importers/llama-cpp.go).
Supersedes #12120 (DCO not signed by contributor, branch had unrelated
history needing rebase).
Assisted-by: MAKI:regolo/glm5.2
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The global ::selection used --color-primary-light (14% primary), which
composites to about 1.1:1 against the dark page ground — selected text
was nearly indistinguishable from unselected. Give selection its own
token in both palettes and align the CodeMirror themes with the same
strengths.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
Expose a negative_prompt string parameter on the image endpoints,
matching Stable Diffusion WebUI / vLLM-Omni conventions. When both the
negative_prompt parameter and a '|'-suffixed negative prompt in the
main prompt are present, they are joined with a comma so callers can
keep a global negative prompt in negative_prompt and add per-image
negative tags after '|'.
Assisted-by: Pi: DeepSeek V4 Pro
Signed-off-by: Fedor Zuev <Fedor.Zuev@gmail.com>
* fix(vulkan): preserve host ICD discovery for packaged backends
Add bundled Mesa manifests through VK_ADD_DRIVER_FILES instead of replacing the system driver list. Merge inherited and model-specific additive paths while preserving explicit operator overrides, with regression coverage.
Assisted-by: Codex:gpt-5 golangci-lint
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(3d): add Kimodo CPU and Vulkan animation backend
Introduce a distinct animation capability and model-described 3D operations, with a typed /3d/animate API, RPC transport, distributed media staging, permissions, and tracing.
Add a persistent kimodo.cpp adapter, skeleton GLB export, CPU/Vulkan packages, model and backend galleries, importer support, CI builds, and documentation. Adapt Studio inputs to each model and provide real-time skeleton playback, seeking, and history.
Cover backend validation, packaging, API behavior, importer inventories, distributed staging, and Studio workflows. Validate real-model CPU/Vulkan generation and deploy the integration to the local QA instance.
Assisted-by: Codex:gpt-5 golangci-lint
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(kimodocpp): adopt monolithic encoders and resident inference
Update upstream for resident weights, packed execution paths, and cached motion graphs. Default to all 32 text layers while retaining configurable streaming and legacy bundle support.
Use monolithic Q8_0 encoders by default and offer all six published quantizations through the gallery and importer. Refresh pinned hashes, tests, and documentation; remove the obsolete thread patch and ensure cached source checkouts follow the upstream pin.
Validated CPU and Vulkan generation, lower-bit streaming, gallery/importer suites, packaging, lint, and cold/warm Studio generation on localai-dev.
Assisted-by: Codex:gpt-5 golangci-lint
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(ui): restore node operation controls
The node restructure hid backend logs and split related controls
across inconsistent layouts.
Restore contextual log actions and align the detail page with the
fleet dashboard. Make multi-node selection clear and accessible.
Assisted-by: Codex:gpt-5 Playwright ESLint
* feat: expose context_size in /v1/models/capabilities
The /v1/models/capabilities endpoint already enriches the OpenAI models
listing with capabilities and modalities. Add context_size to that
enrichment so clients (e.g. nib) can size their context budget for
auto-compaction and pruning without guessing.
The value comes from backend.EffectiveContextSize, which resolves the
configured context_size or falls back to DefaultContextSize (4096) —
the same number the backend itself runs with. It is omitempty so a
loose file with no config surfaces 0 (unknown) rather than a misleading
default, and existing clients that don't know the field see no change.
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The node detail redesign moved backend operations into an action menu.
Four existing specs still search for the removed direct buttons, so the
UI E2E workflow fails consistently on master.
Open the backend action menu before checking or activating its items.
Assisted-by: Codex:gpt-5
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The node restructure hid backend logs and split related controls
across inconsistent layouts.
Restore contextual log actions and align the detail page with the
fleet dashboard. Make multi-node selection clear and accessible.
Assisted-by: Codex:gpt-5 Playwright ESLint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(credentials): parse and match download credential rules
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): keep secrets out of parse errors and tighten URL matching
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(credentials): resolve secrets lazily and authenticate HTTP per hop
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): redact secrets in nested and store formatting
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(credentials): add registry keychain and oras credential adapters
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): match repository rules for Docker Hub in the oras adapter
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(downloads): authenticate HTTP downloads and gallery reads from the credentials store
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(oci): authenticate registry pulls, resumes, blobs and cosign from the credentials store
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(cli): load download credentials from --credentials-file
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(credentials): correct the local-network registry rules
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): keep secrets out of match and YAML parse errors
A match that fails to parse is no longer quoted in the Parse error, since
it may be a URL with a token in it. Userinfo is detected before the scheme
check, so ftp://user:token@host is refused as userinfo, and a match with a
query string or fragment is refused because it can never apply and a query
string is where signed URLs carry their token. Every YAML decode error is
now redacted, not only type errors: quoted scalars such as a secret under a
mismatched !!int tag are replaced and unquoted map keys are cut off.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(downloads): make auth errors name the real cause and never retry unresolved secrets
AuthError now appends its cause, so a registry's DENIED or UNAUTHORIZED
detail reaches the operator. HTTP auth errors print only the status text in
place of the cause, because the downloader builds that cause from the
requested URL, which can carry a signed query string. Registry pulls say
that docker config credentials were tried too, and a download that carried
a caller-provided credential (WithBearerToken, or an explicit authorization
on gallery reads) reports that credential as rejected instead of blaming
the store.
The Range probe for a leftover partial file now returns an unresolved
secret as a permanent error, like the download request already did.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): keep oras pulls anonymous on a broken docker helper and close bodies
When docker config names a credsStore helper that cannot run, the oras
credential func now logs at debug and returns no credential, so public
pulls keep working as they did before the adapter existed. The transport
closes the request body when a rule's secret cannot be resolved, as the
RoundTripper contract requires. The redirect spec now uses a custom header
rule on the origin, which net/http would not strip on its own, to prove the
transport does not carry credentials to the next hop.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(oci): cover FetchImageBlob authentication against a private registry
FetchImageBlob now has a spec that pulls a layer blob by digest from a
basic-auth registry through the oras credential adapter, and one that shows
the same fetch fails when no rule matches. oras only speaks HTTPS here, so
the registry serves TLS and the spec points http.DefaultTransport, which
retry.DefaultClient falls back to, at the test server's client for its
duration instead of adding a transport seam to production code.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(credentials): document auth error wording, ollama manifests and registry tokens
The errors section now lists the registry and provided-credential messages
and says the server's reason is appended. ollama:// manifests are fetched
without credentials, so only blob downloads use the file. GHCR, Docker Hub
and Quay need basic auth with the token as password, and match rules must
not carry a query string or fragment. The backend gallery docs and the
container troubleshooting section now point to the private sources page.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): document trusted file path
The credentials path comes from operator configuration. Mark the file
read with a scoped G304 explanation to resolve the gosec false positive.
Assisted-by: Codex:gpt-6 gosec
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(cli): benchmark configured text models
Compare model aliases through a running LocalAI server. Report full
request latency and server-reported throughput with raw JSON samples.
Keep warmups separate and fail before writing results on request errors.
Assisted-by: Codex:GPT-6
* fix(cli): satisfy benchmark error checks
Explicitly discard errors from buffered report writes, HTTP response cleanup, and test server writes to pass errcheck without changing behavior.
Assisted-by: Codex:gpt-6 golangci-lint
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The LocalAGI bump in #11985 changed state.NewAgentPool to take a
SkillsProvider and a PoolLimits value. The call site here was not
updated, so master stopped compiling and every Go job went red.
Pass the limits explicitly, mirroring LocalAGI's own defaults, so the
pool prunes conversation dumps and scheduler run history instead of
growing without end.
Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
