Upgrade gguf-parser-go to v0.26.3 so string lengths are checked against the remaining file size before allocation. Master AIO CI crashed in the background gallery warmer when v0.25.0 tried to allocate several terabytes; panic recovery cannot catch a fatal runtime OOM.
Check both overflow-sized and file-exceeding strings through the remote reader, and document the size-only estimate fallback.
Assisted-by: Codex:gpt-6
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Fence model load jobs by generation, lease them on the database clock, bound the work on the worker with operations and a process-group watchdog, and add one stop path with a load-cancel API. See the pull request for the design, the rolling upgrade notes and the test evidence.
Assisted-by: Claude Code:claude-sonnet-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(diarization): return sound events with include_sounds
A client that wants text, speakers, voice prints and sound events had to
make a diarization call and a separate sound call. Add an include_sounds
request field to /v1/audio/diarization that adds a sounds array of closed
events {start, end, label, confidence}, in seconds.
The parakeet-cpp backend runs a tagger-only scene stream over the clip,
the same stream and thresholds the live path uses, so a clip gives the
same events offline and live. A model with no sound_model companion, or a
backend that does not report sound events, fails with 501 and the stable
code include_sounds_unsupported instead of an empty list. The proto
carries sounds_included so an empty list still means "nothing heard".
The localai-proxy backend forwards the field. Swagger, docs and the
e2e mock backend are updated.
Assisted-by: Claude:claude-sonnet-5-5 [protoc swag go]
* feat(gallery): add parakeet-cpp-multilingual-diarization-speakers-sounds
Same as parakeet-cpp-multilingual-diarization-speakers (TDT 0.6B v3,
Nemotron-3-Diarization, WeSpeaker) plus a CED-Tiny sound_model, so one
model name serves /v1/audio/diarization with include_text,
include_speaker_profiles and include_sounds. It declares the
sound_classification usecase like the realtime scene entries.
Assisted-by: Claude:claude-sonnet-5-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(xsysinfo): read process VRAM without proc children
Kernels without CONFIG_PROC_CHILDREN have no task children file, so
ProcessVRAM dropped every DRM reading. When that file is missing, walk
child processes from /proc/<pid>/stat ppid links instead. Other read
errors still drop the reading.
Fixes#12481
Signed-off-by: leilei3167 <imleilei123@gmail.com>
* docs(system): describe the proc children fallback
Document VRAM reporting on kernels without CONFIG_PROC_CHILDREN.
Assisted-by: Codex:GPT-6
---------
Signed-off-by: leilei3167 <imleilei123@gmail.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(oci): stage image downloads in a configured directory, not in /tmp
Problem:
- the image tar and its compressed layers staged in os.TempDir()
- /tmp is commonly a RAM-backed tmpfs far smaller than a backend image
- big installs failed with a full /tmp or silently ate RAM
Change:
- staging path resolved like the other storage paths
(LOCALAI_DOWNLOAD_STAGING_PATH, default ${basepath}/downloading),
threaded to the extractor as a download option
- every download works in its own subdirectory holding its tar and
layers: nothing shared between concurrent downloads, one removal
cleans a download up, a crash leaves one self-contained orphan
- callers that pass no staging directory keep the OS temp behavior
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
* fix(oci): handle staging cleanup errors
Log staging cleanup failures to satisfy errcheck. Test extraction with an
unavailable OS temp directory and verify that staging is removed.
Assisted-by: Codex:GPT-6
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
---------
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
Remove watchdog tracking when backends stop, crash, or fail to start.
Validate eviction addresses under the model lifecycle lock so delayed
shutdowns cannot stop a replacement backend. Preserve replacement size
estimates when removing an old address.
Add lifecycle regression tests and document shutdown behavior.
Fixes#12331
Assisted-by: Codex:gpt-6
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The HuggingFace tree API returns each entry's path relative to the repo
root ("assets/plots", not "plots"). The recursive listing prefixed the
parent directory again, so it requested "assets/assets/plots", got a
404, and failed the whole listing. The importer then treated the URI as
a non-HF repo and no importer matched.
This broke the import of GGUF repos that keep per-quant subfolders next
to a nested assets tree, such as
ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF.
The test mock now returns root-relative directory paths like the real
API and routes on the exact tree path, so a doubled path 404s.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
utils.VerifyPath joins its argument onto the base path, so a path that
the caller already joined always passes. Several callers gave it joined
paths, and their checks could not fail:
- modeladmin (config view, patch, edit, pin and state): the config file
path from the loader. A config loaded from outside the models
directory (--models-config-file) could be pinned, and the pin wrote
the outside file. The patch and state paths stopped later, in the
mutation snapshot, with a different error.
- core/backend/tts.go: the model path joined onto the models path.
- The trellis2cpp and stablediffusion-ggml backends: option paths
(*_path) joined onto the model path. A "../" value outside the model
directory was accepted.
Add utils.VerifyResolvedPath for a full path. modeladmin and tts use
it. The backends now check the relative option value before they join
it. A rename in modeladmin checks the new relative name.
For models from a config file outside the models directory, the admin
API and web UI now return ErrPathNotTrusted for view, edit, pin, and
enable or disable. The docs describe this.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
listModelFiles gave utils.VerifyPath paths that it had already joined
onto the models directory. VerifyPath joins its argument onto the base
again, so an absolute path always passes and none of the four checks
could fail. Model deletion then removed files outside the models
directory:
- A model name such as "../outside/victim" removed
outside/victim.yaml. The in-process MCP delete_model tool passes the
name from the tool call without a check.
- A gallery file that lists a files: entry with "../" removed that
file.
listModelFiles now gives VerifyPath the relative names.
InTrustedRoot also looped forever when a relative path was outside a
relative root. filepath.Dir stops at "." for a relative path, and the
loop waited for "/". The loop now stops when Dir returns its input.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(functions): honor function_arguments_key when building the tool grammar
All four call sites of `Functions.ToJSONStructure(name, args string)` pass
`FunctionsConfig.FunctionNameKey` as *both* arguments, so
`FunctionArgumentsKey` never reaches the grammar generator.
`ToJSONStructure` writes the two properties into the same map:
property[nameKey] = FunctionName{Const: function.Name}
property[argsKey] = Argument{...}
When `nameKey == argsKey` the second assignment overwrites the first, so a
model configured with `function_name_key` gets a grammar carrying only the
arguments object -- the `{"const": "<function name>"}` constraint is gone and
the grammar can no longer express which function was called.
With `function_name_key: function`, the generated property set collapses from
{"function": {"const": "get_weather"}, "arguments": {...}}
to
{"function": {"type": "object", "properties": {...}}}
Setting only `function_arguments_key` is equally broken in the other
direction: the grammar keeps emitting `arguments` while `ParseFunctionCall`
(pkg/functions/parse.go) looks up the configured key, so the parsed call comes
back with its arguments empty.
The default configuration is unaffected -- with both keys empty
`ToJSONStructure` falls back to `name`/`arguments` for both parameters, which
is why this went unnoticed.
The existing `ToJSONStructure()` unit test already calls the helper with two
distinct keys, so only the call sites were wrong. Extend that test with a case
that keeps both custom keys distinct and asserts the two properties survive.
Signed-off-by: Anai-Guo <antai12232931@outlook.com>
* test(functions): cover configured grammar keys
Route grammar construction through FunctionsConfig so the regression test
covers the key wiring used by every endpoint.
Assisted-by: Codex:gpt-5
* chore: empty commit to trigger workflow approval
Signed-off-by: Tai An <antai12232931@outlook.com>
---------
Signed-off-by: Anai-Guo <antai12232931@outlook.com>
Signed-off-by: Tai An <antai12232931@outlook.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The proxy sent every chat and completion request with
context.Background, and the gRPC server gave rich backends no context
at all. When a client disconnected, or failover gave up on the target,
the upstream kept generating to the end, which costs tokens on a paid
or shared upstream. A silent upstream held the backend forever.
Add the optional AIModelRichContext interface. The gRPC server prefers
it and passes the call's context, like the Score and Rerank
extensions. The proxy implements it, so the upstream request ends with
the gRPC call.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
* feat(system): report per-model DRM VRAM
Expose optional resident device memory for local backend process trees.
Deduplicate DRM clients and omit unsupported or incomplete readings.
Document accounting limits and preserve a measured zero in JSON.
Closes#11970.
Assisted-by: Codex:gpt-6
* fix(system): document trusted procfs reads
Scope G304 annotations to paths built from the fixed procfs root,
integer process IDs, and kernel directory entries. These reads accept
no user-controlled path components.
Assisted-by: Codex:GPT-6 gosec
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
A request with narrower allow_patterns hashes to a different CacheKey
than an already-committed broader sibling, so committedResult misses and
materializeLocked re-fetches files the sibling already holds. After the
own-tree reuseMaterializedFile miss, consult committed sibling trees
for the same Source (type+endpoint+repo+revision), re-verify the file
through verifyDownloadedFile (full SHA-256, never size-only), and
hard-link it into the writer's staging snapshot (copy fallback only on
EXDEV). Each file is matched individually against the sibling's
manifest, so a broader request can never inherit a narrower sibling's
gaps as if complete.
The sibling manifest set is loaded and source-matched once per
materialization (files indexed by path) instead of once per staged
file, so a models volume with 20 committed artifacts and a 300-file
snapshot does one manifest pass rather than ~6000 reads and JSON
parses. The sibling-reuse behavior cases live in the package's
registered Ginkgo suite so repository test conventions apply.
Refs #11047
Signed-off-by: supermario_leo <leo.stack@outlook.com>
Code scanning reports alerts on every line of a touched file. Mark the
gRPC auth env var name and the mock backend's staged-path reads as
reviewed, and log the error when evicting a model after its connection
fails instead of dropping it.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Review asked for constants instead of the "cloud-proxy" and
"localai-proxy" literals. Add CloudProxyBackend and LocalAIProxyBackend next
to the other backend-name constants in pkg/model (WhisperBackend,
TransformersBackend, ...), which core/config already imports, and use them
in every production check: the proxy options builder, IsRemoteProxy,
IsCloudProxyBackendPassthrough, the PII defaults, the localai-proxy backend
hook and loader warning, and the PII middleware metadata in the routes and
the in-process MCP client.
The proxy options builder now calls IsRemoteProxy() instead of repeating
the two-backend check, so the set of proxy backends is defined once.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): unify decision pipeline through Score RPC with vllm_decide ABI v29
Replace the model-specific SystemOne gRPC approach with a generic Score
RPC extension. The pre-existing Score RPC (previously unused by any
backend) now carries question_type and response_json fields:
- question_type="systemone" routes kev/laya decision-pipeline requests
through the unified vllm_decide C ABI (v29), returning the full
response JSON in response_json.
- question_type empty routes cua-s1-forms candidate scoring through the
same vllm_decide ABI, returning CandidateScore probabilities.
The vllm-cpp backend's Score() method calls vllm_decide and dispatches
by architecture internally. The /v1/systemone HTTP endpoint checks
whether the model's backend supports Score; if so, it forwards the raw
request JSON and returns the backend response as-is. Other backends
fall through to the existing NER-based path.
This mirrors the vllm.cpp C ABI refactor (PR #3301) that replaced
vllm_systemone + vllm_score with a single vllm_decide function. The
purego bindings bump abiVersion from 27 to 29 and resolve vllm_decide
and vllm_decide_free symbols.
Also fixes validModelPath to accept cua-s1-forms.json and
rl_agent_config.json alongside config.json, matching the engine's
model_loader.cpp config-filename ordering.
AI-Assisted: true
Assisted-by: Maki:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore: ⬆️ Update mudler/vllm.cpp to e28ec46c6 (fix macOS -Werror build)
Bumps vllm.cpp to e28ec46c6 which fixes a -Wnull-conversion error in
qwen3_5.cpp:12483 that broke the macOS Metal CI build under -Werror.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): verification follow-ups for oci:// galleries
Follow-ups from the post-merge review of #12238 and #12239.
Only a policy decision is a refusal now. cosignverify wraps
ErrPolicyRejected around a failed signature check, an identity or
source-repository mismatch, a not_before cutoff and a missing or
unparseable bundle. A TUF, registry or network failure during
verification, or a timeout, is an outage: the gallery falls back to the
copy verified under the current policy, as it does when the registry is
down.
An oci:// gallery with a verification block, or any oci:// gallery under
strict integrity, is no longer answered by an https://, github: or
file:// mirror. Such a mirror is ignored with a warning, because nothing
can check its signature. The index of an HTTP gallery, whose policy only
covers its backend images, is cached under the URL-only name again, so no
unchecked body is stored under a policy-keyed name.
The in-memory index cache key now includes the policy. After a runtime
policy change the index is fetched again, and entries with a relative url
install again.
The registry digest lookups after install and upgrade, and in the
upgrade check, run only for real registry references (new
URI.LooksLikeRegistryOCI), not for ollama:// or ocifile://.
The refusal message names strict integrity when that is the cause, and
the gallery name is no longer repeated.
Specs pin the URL-only cache name for galleries without a policy, a fixed
key for a fixed policy, and that every GalleryVerification field changes
the key. The docs describe refusal, outage, mirrors and strict integrity.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): reset listings on gallery changes, classify referrer outages
Review follow-ups for this PR.
The React UI lists from AvailableGalleryModelsCached, which is keyed by
nothing. A gallery change through the settings API or a
runtime_settings.json edit now drops that listing when the model or
backend gallery configuration differs. Before, the UI kept the old list,
with local paths into the old policy's tree, until the next background
refresh, or for good when the new policy refused the gallery.
In cosignverify, a referrer the registry fails to serve now makes the
lookup an outage whatever other referrers failed and in any order, since
the unread one may be the valid signature. An invalid policy (Validate in
NewVerifier, an unparseable not_before) is ErrPolicyRejected, because no
fetch can make it usable.
The docs say that only an oci:// gallery with a verification block skips
non-OCI mirrors, and list an unusable policy as a refusal.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
A backend installed from an oci://host/repo:tag or oci://host/repo@sha256
URI was downloaded correctly, but the digest lookup that follows the
install (and the one after an upgrade) passed the raw URI to the registry
client. The client does not know the oci:// scheme: with a port in the
host it failed to parse the reference, without one it read "oci" as the
registry host and queried https://oci/v2/. The install still succeeded,
so the only trace was a warning and an empty digest in metadata.json,
which made the next upgrade check report an upgrade for no reason.
Add downloader.URI.OCIReference and route every consumer that hands an
OCI URI to a registry client through it: the install and upgrade digest
lookups, the upgrade check, the OCI download path, the oci:// gallery
fetch and the llama.cpp importer.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): add GLiNER2.5 NER via TokenClassify
Wire the vllm-cpp backend to the C ABI NER surface (vllm_gliner_ner,
ABI v27) so LocalAI can serve zero-shot named entity recognition through
the existing TokenClassify gRPC method.
backend.go: TokenClassify method on *VllmCpp calls vllm_gliner_ner with
the text and labels, copies the C-owned entity array into protobuf
TokenClassifyEntity messages, and frees the result.
govllmcpp.go: cNerEntity and cNerResult Go POD mirrors matching the C
structs; vllmGlinerNer and vllmNerResultFree purego bindings; abiVersion
bumped 26 -> 27.
options.go: ner_labels, ner_threshold, ner_max_width parsed from
engine_args.
pkg/grpc: ClassifyModel interface and TokenClassify server handler
(follows the Embedding locking pattern).
core/config: vllm-cpp backend declares MethodTokenClassify and
UsecaseTokenClassify.
docs/content/features/vllm-cpp.md: NER section documenting the
engine_args keys and the host-forward contract.
Assisted-by: MAKI:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): correct NER pointer lint directive
Use the govet directive for the C-owned NER array, matching the other
purego pointer conversions. The array remains valid until its deferred
free; the misspelled directive caused CI to flag this conversion.
Assisted-by: Codex:gpt-6 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): add kev-compatible SystemOne API endpoints
Add POST /v1/systemone, /v1/systemone/permute, and
/v1/systemone/separate to LocalAI, mirroring the kev project's
structured-extraction API. Each endpoint runs zero-shot NER over the
rendered state text and builds kev-compatible answers for three question
types: noul (binary entity presence), choice (pick one option), and
score (pick one level).
The TokenClassifyRequest proto gains a `repeated string labels` field so
each question can supply its own labels at inference time, and
TokenClassifier gains TokenClassifyWithLabels for per-call label
selection. The vllm-cpp backend uses request labels when non-empty,
falling back to configured ner_labels then the built-in defaults.
Helpers (renderState, softmax, choiceConfidence, scoreConfidence, r2)
are ported from kev/api.py and mirrored in vllm.cpp's api_server.cpp so
both servers produce the same answer shape.
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): suppress gosec G404 on seeded permutation RNG
The SystemOne permute endpoint uses math/rand with a caller-supplied
seed for reproducible option permutations, matching kev's random.seed.
gosec flags this as G404 (weak RNG). Add #nosec with a comment naming
the intent: this is reproducibility, not cryptography.
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(vllm-cpp): bump vllm.cpp pin to GLiNER2.5 merge commit
Advance VLLM_CPP_VERSION from f3cd97e to 5058268d, the commit that
landed GLiNER2.5 zero-shot NER support (PR #3224) in vllm.cpp. This
brings the DeBERTa v2 encoder, GLiNER2 boundary head, C ABI NER
functions, and server endpoints into the LocalAI vllm-cpp backend.
The ABI version (27) and Go struct mirrors already match.
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): use instruction text as NER label in SystemOne handler
The SystemOne handler was passing question IDs as NER labels for noul
questions and bare key names for choice questions, so the model never
matched any entities. Port the label mapping from vllm.cpp's
ParseSystemOneBody:
- noul: use the rendered instructions field (with instr alias) as the
NER label, not the question ID
- choice: use optionText(name, desc) — "name: description" or "name"
when the description is null/empty — not the bare key
- score: already correct (rendered criteria text)
- permute: shuffle indices and build parallel key/label arrays so the
NER call uses the optionText labels while the response is keyed by
the original option names
Also add the instructions field to the SystemOneQuestion schema struct
(accepted alongside the instr backward-compat alias).
Verified end-to-end against the real GLiNER2.5 model: noul questions
now find "Apple Inc. is" (organization, 0.999) and "Tim Cook is"
(person, 0.852) where they previously returned zero entities.
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Use -n 1 for a single JSON sample instead of -s 1, which sets a 1 ms
refresh period and runs until interrupted on current intel_gpu_top.
Fixes#12205
Signed-off-by: leilei3167 <imleilei123@gmail.com>
* feat(oci): pull ORAS artifacts into a directory
Galleries published to an OCI registry arrive as an ORAS artifact whose
layers carry their tree path in org.opencontainers.image.title. Nothing in
pkg/oci could read a non-image manifest, so add PullArtifact: it lays the
layers out under a destination directory and returns the manifest digest so
callers can pin and verify signatures on it.
A registry is remote input and this writes files, so the manifest is
validated before any byte is fetched. A title that is absolute, escapes the
destination or is missing is refused, an artifact of an unexpected
artifactType is refused, and both the layer count and the total size are
capped with caller-settable limits.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(gallery): fetch and verify galleries published as OCI artifacts
A gallery whose URL is oci://host/repo:tag is now pulled as an ORAS
artifact and its index.yaml read from the unpacked tree, alongside the
existing HTTP and file sources. Mirrors, the per-candidate timeout, the
failure cooldown and the last known good copy work the same for both
schemes, so a gallery can mirror an OCI primary with an HTTP fallback.
When the gallery declares a verification policy the artifact reference is
resolved to its manifest digest, the publisher signature is checked
against that digest, and only then is the same digest pulled. Nothing
unverified is written to disk or parsed. With strict integrity mode on,
an OCI gallery without a verification policy is refused instead of
warned about, which is what the flag already does for backend installs.
The pull lands in a staging directory that is renamed into the cache only
once the whole tree is on disk and the index reads back as an index, so a
failed or interrupted pull leaves nothing a later fetch would serve. A
fetch that finds a fresh cache entry does not contact the registry.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(gallery): resolve entry urls relative to the gallery root
A gallery published as a self-contained tree, which is what an OCI
gallery is, names its base configs by their place in that tree. Such a
url reached the HTTP client verbatim and no entry of the gallery could
be installed.
A scheme-less entry url now resolves against the root the index came
from: the unpacked artifact for an oci:// gallery, and the directory of
the index URL otherwise. The relative path may not climb out of that
root, reusing the guard the artifact puller already applies to layer
titles, and an entry that does is dropped from the listing rather than
failing the whole gallery. An entry that names its own scheme is
untouched, so .ref indirection and local github: resolution behave as
before.
Reading an entry of an OCI gallery also needs the unpacked artifact
directory as its trusted root, since the cache is deliberately a sibling
of the models directory the downloader otherwise confines a file read
to.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(oci): restrict artifact file permissions
Create artifact files with owner-only permissions to satisfy G302.
Document the existing path validation for the G304 scanner finding.
Check the permissions of downloaded layers in the artifact test.
Assisted-by: Codex:gpt-6 gosec
* fix(oci): confine artifact writes to their root
Use root-relative filesystem operations to reject escapes through existing
directory symlinks. Retain owner-only permissions and add a symlink escape
regression test, without suppressing the path traversal finding.
Assisted-by: Codex:gpt-6
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
A registry without the OCI 1.1 referrers API leaves the referrers index to
the signing client, and cosign fills each entry's artifactType from the
manifest's config media type rather than from its artifactType. CNCF
distribution 3.0.0 has no referrers route at all, so on that registry every
correctly signed image reads as unsigned: the verifier filtered the index
entries by artifactType and matched nothing.
The verifier now asks the referrer manifest what it is when no entry
advertises itself, checking the manifest's own artifactType and its first
layer. The fast path is unchanged, the second pass runs only when the first
finds nothing, and it is bounded so a heavily referenced image cannot turn
one verification into an unbounded walk.
Found by a real release: the images and signatures were correct and no
client could verify them.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Report input tokens and frame-step output units in response metadata and
record them through the existing usage accounting pipeline. Preserve the
accounting rule and model-specific dimensions as JSON without extending
the gRPC schema for each modality.
Expose animation usage only under metadata.usage, validate counts before
recording, and document the response contract and loaded-model location.
Add coverage for transport, defaults, failures, persistence, and recording
requests once with statistics enabled or disabled.
Assisted-by: Codex:GPT-6
Signed-off-by: Richard Palethorpe <io@richiejp.com>
On shared-memory AMD APUs (e.g. Strix Halo), rocm-smi VRAM is only the
BIOS UMA carve-out while usable HIP memory lives in GTT. Query
--showmeminfo all and fold GTT when the VRAM/GTT ratio matches an APU,
preserving dedicated dGPU accounting.
Fixes#12058
Signed-off-by: lei_lei <imleilei123@gmail.com>
Gallery entries that ship archives (for example Piper voices) leave the
downloaded .tar.bz2 and its .sha256 checksum in the models directory.
ListFilesInModelPath reported them as loose models, so they showed up in
/v1/models and in the UI as models that cannot be loaded.
Add both suffixes to the skip list, next to the existing .tar.gz entry.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vulkan): preserve host ICD discovery for packaged backends
Add bundled Mesa manifests through VK_ADD_DRIVER_FILES instead of replacing the system driver list. Merge inherited and model-specific additive paths while preserving explicit operator overrides, with regression coverage.
Assisted-by: Codex:gpt-5 golangci-lint
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(3d): add Kimodo CPU and Vulkan animation backend
Introduce a distinct animation capability and model-described 3D operations, with a typed /3d/animate API, RPC transport, distributed media staging, permissions, and tracing.
Add a persistent kimodo.cpp adapter, skeleton GLB export, CPU/Vulkan packages, model and backend galleries, importer support, CI builds, and documentation. Adapt Studio inputs to each model and provide real-time skeleton playback, seeking, and history.
Cover backend validation, packaging, API behavior, importer inventories, distributed staging, and Studio workflows. Validate real-model CPU/Vulkan generation and deploy the integration to the local QA instance.
Assisted-by: Codex:gpt-5 golangci-lint
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(kimodocpp): adopt monolithic encoders and resident inference
Update upstream for resident weights, packed execution paths, and cached motion graphs. Default to all 32 text layers while retaining configurable streaming and legacy bundle support.
Use monolithic Q8_0 encoders by default and offer all six published quantizations through the gallery and importer. Refresh pinned hashes, tests, and documentation; remove the obsolete thread patch and ensure cached source checkouts follow the upstream pin.
Validated CPU and Vulkan generation, lower-bit streaming, gallery/importer suites, packaging, lint, and cold/warm Studio generation on localai-dev.
Assisted-by: Codex:gpt-5 golangci-lint
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(credentials): parse and match download credential rules
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): keep secrets out of parse errors and tighten URL matching
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(credentials): resolve secrets lazily and authenticate HTTP per hop
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): redact secrets in nested and store formatting
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(credentials): add registry keychain and oras credential adapters
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): match repository rules for Docker Hub in the oras adapter
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(downloads): authenticate HTTP downloads and gallery reads from the credentials store
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(oci): authenticate registry pulls, resumes, blobs and cosign from the credentials store
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(cli): load download credentials from --credentials-file
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(credentials): correct the local-network registry rules
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): keep secrets out of match and YAML parse errors
A match that fails to parse is no longer quoted in the Parse error, since
it may be a URL with a token in it. Userinfo is detected before the scheme
check, so ftp://user:token@host is refused as userinfo, and a match with a
query string or fragment is refused because it can never apply and a query
string is where signed URLs carry their token. Every YAML decode error is
now redacted, not only type errors: quoted scalars such as a secret under a
mismatched !!int tag are replaced and unquoted map keys are cut off.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(downloads): make auth errors name the real cause and never retry unresolved secrets
AuthError now appends its cause, so a registry's DENIED or UNAUTHORIZED
detail reaches the operator. HTTP auth errors print only the status text in
place of the cause, because the downloader builds that cause from the
requested URL, which can carry a signed query string. Registry pulls say
that docker config credentials were tried too, and a download that carried
a caller-provided credential (WithBearerToken, or an explicit authorization
on gallery reads) reports that credential as rejected instead of blaming
the store.
The Range probe for a leftover partial file now returns an unresolved
secret as a permanent error, like the download request already did.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): keep oras pulls anonymous on a broken docker helper and close bodies
When docker config names a credsStore helper that cannot run, the oras
credential func now logs at debug and returns no credential, so public
pulls keep working as they did before the adapter existed. The transport
closes the request body when a rule's secret cannot be resolved, as the
RoundTripper contract requires. The redirect spec now uses a custom header
rule on the origin, which net/http would not strip on its own, to prove the
transport does not carry credentials to the next hop.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(oci): cover FetchImageBlob authentication against a private registry
FetchImageBlob now has a spec that pulls a layer blob by digest from a
basic-auth registry through the oras credential adapter, and one that shows
the same fetch fails when no rule matches. oras only speaks HTTPS here, so
the registry serves TLS and the spec points http.DefaultTransport, which
retry.DefaultClient falls back to, at the test server's client for its
duration instead of adding a transport seam to production code.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(credentials): document auth error wording, ollama manifests and registry tokens
The errors section now lists the registry and provided-credential messages
and says the server's reason is appended. ollama:// manifests are fetched
without credentials, so only blob downloads use the file. GHCR, Docker Hub
and Quay need basic auth with the token as password, and match rules must
not carry a query string or fragment. The backend gallery docs and the
container troubleshooting section now point to the private sources page.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): document trusted file path
The credentials path comes from operator configuration. Mark the file
read with a scoped G304 explanation to resolve the gosec false positive.
Assisted-by: Codex:gpt-6 gosec
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Do not seed streaming reasoning state when the latest prompt thinking marker is already followed by its matching closing marker. This keeps direct Gemma 4 output in content when its template disables thinking with a preclosed channel.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Saved profiles previously resolved to one audio path and transcript, so
cloning backends could not use several examples of one personality.
Store ordered audio and transcript pairs while preserving the legacy
first-reference fields. Fish Speech and audio.cpp receive all pairs,
including on distributed workers. Other backends retain their
single-reference behavior.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Broadcast forced-disturb events and successful scale-up resets so every frontend shares the same rolling autoscale signal. Deduplicate NATS echoes, expose an origin-only Prometheus counter, and document cluster behavior.
Closes#10083
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Backend processes shared the host temporary directory, so crashes could leave request images and audio behind until the filesystem filled. Give each process a locked LocalAI-owned runtime, remove scratch on exit, and sweep only marked abandoned runtimes at the next start.
Also close known request error-path leaks in the Python media backends, CrispASR, LongCat Video, and stable-diffusion.cpp.
Assisted-by: Codex:gpt-5
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* docs: design ephemeral staging retention
High-frequency camera and audio inputs can fill a worker before the current six-hour cleanup window expires.
Define a one-hour retention policy that preserves recently modified request payloads.
Assisted-by: Codex:gpt-6
* docs: make ephemeral staging request-owned
Time-based retention can still fill a worker under bursty or high-rate input. Define request-lifecycle cleanup with capacity reservation and crash recovery.
Assisted-by: Codex:gpt-6
* feat(distributed): release exact staged keys
Request inputs need transport-neutral cleanup after backend calls. Add authenticated exact-key deletion for HTTP and coordinated cache eviction before shared-object deletion for S3/NATS.
Preserve URL metacharacters as filename data, reject unsafe keys, and remove upload sidecars while pruning empty request directories.
Assisted-by: Codex:gpt-6
* fix(distributed): release staged request inputs
Ephemeral inputs remained on workers after inference completed. Release each exact key after synchronous and streaming calls, including partial staging failures.
Use a bounded cleanup context so caller cancellation cannot suppress release. Preserve caller requests and backend results when cleanup fails.
Assisted-by: Codex:gpt-6
* feat(worker): bound ephemeral staging capacity
Concurrent staging can otherwise exceed its byte limit or consume reserved filesystem headroom. Explicit states keep bytes charged through each reservation, write, and commit transition.
Use a synchronized waiter count to prove Commit blocks until bounded writers close, and retain committed baselines across re-reservation.
Assisted-by: Codex:gpt-6
* feat(worker): enforce ephemeral staging bounds
Share capacity accounting across HTTP and S3 request inputs so workers
reject uploads before exhausting their filesystem. Reconcile exact release
and crash recovery with the same guard.
Assisted-by: Codex:gpt-6
* fix(distributed): make staged release race-safe
Pin each release path component before removing request-owned inputs and sidecars. Stop pruning when a directory identity changes.
Assisted-by: Codex:gpt-6
* fix(worker): retain staged input ownership
Keep committed request inputs protected from age recovery until exact release ends their ownership. Startup-scanned files remain reclaimable and can acquire ownership through reservation.
Assisted-by: Codex:gpt-6
* fix(worker): claim cached ephemeral inputs
Keep startup-scanned cache hits owned while inference uses them and reconcile their actual size against capacity.
Assisted-by: Codex:gpt-6
* fix(distributed): enforce staging admission
Propagate multimodal staging failures before inference and claim matching ephemeral HTTP cache entries. Fall back to PUT when an older worker does not support claims.
Assisted-by: Codex:gpt-6
* fix(distributed): close staging accounting gaps
Keep unknown-length reservations charged until bytes reach disk and bound NATS release waits by the lifecycle cleanup deadline.
Assisted-by: Codex:gpt-6
* fix(distributed): restage swept cache hits
Treat files removed between cache probing and ownership claims as misses so HTTP and S3 workers can stage them again.
Assisted-by: Codex:gpt-6
* fix(distributed): release staged inputs by request
Release every input from one inference with one fixed-size worker coordination request. Fence request ingress against cleanup, bound staging capacity and cleanup state, and retain exact-key release for rolling upgrades.
Assisted-by: Codex:gpt-6
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(downloader): make file:// installs reachable again
DownloadFileWithContext already has a branch that copies from a local
file, but it could never run. Before reaching it the function decides
whether the destination is fetchable with
} else if !os.IsNotExist(err) || !URI(url).LooksLikeHTTPURL() {
and LooksLikeHTTPURL is http(s) only, so any URI resolving to a local
path is rejected there. Falling through requires the destination to be
missing AND the source to be an HTTP URL, which a file:// source never
is -- leaving the local-source branch below unreachable.
A first import always has a missing destination, so importing
file:///path/to/model.gguf always failed, with an error that listed
file:// among the supported schemes (#11701).
Name the local-source condition once as URI.hasLocalSource and use it
both to admit the destination and to pick the source, so the two cannot
drift apart again.
Signed-off-by: Tai An <antai12232931@outlook.com>
Node placement and replica rules could only name a model, so an operator
who pinned "llama3" to the GPU tier had to rewrite the rule whenever a
different model took over that job. An alias already gives a stable name
for whichever model serves it, and a rule on that name makes it a
deployment slot: repoint the alias and the placement follows.
A rule keeps the name the operator chose. Reads resolve that name through
the config loader to the model the rule governs, so the reconciler counts,
schedules and trims replicas of the target, and the router finds an
alias-keyed rule from the target it is already routing. An alias that
resolves to nothing governs nothing loadable, so the reconciler skips it
and the write paths refuse it.
A replica is shared by every name that resolves to it, so only one rule
can decide where it runs. The REST and MCP write paths reject a rule whose
target another rule already governs. A pair that arrives some other way,
such as a seed file or an alias repointed onto a model that already has a
rule, resolves in favour of the rule named after the model itself and then
the oldest, and the rest are listed as shadowed.
The eviction guard is the exception: it matches rules to replicas in raw
SQL inside a locking transaction and cannot resolve an alias. It reads a
stored target that the reconciler refreshes each tick, and falls back to
the rule's own name when that target is empty.
Assisted-by: Claude:claude-opus-5 golangci-lint eslint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
quay.io redirects blob downloads to pre-signed S3/Akamai URLs that
expire after about 10 minutes. On a slow connection a multi-GiB
backend layer cannot finish inside that window, so the connection
drops mid-stream on every attempt. The retry added for #10577
restarted each attempt from byte zero, which replayed the same
failure until the budget ran out and the install failed with
"unexpected EOF".
A retry now keeps the bytes already on disk and re-requests the
blob with "Range: bytes=N-". Each request goes back to the
registry, so it gets a fresh redirect URL and auth token. The
retry budget only counts attempts that made no forward progress,
so a slow link that keeps advancing keeps downloading. A resumed
file is spliced from separate responses and bypasses the digest
check in layer.Compressed(), so the assembled file is re-verified
against the layer digest before it is trusted; on a mismatch the
download starts over through the verified reader.
Fixes#10577
Assisted-by: Claude Code:claude-fable-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
process.New applies its options and discards the error they return. When
WithTemporaryStateDir could not create a directory, StateDir stayed
empty and every later option went unapplied, so the failure surfaced
from Run as "mkdir : no such file or directory" naming no path.
That message cost a full day of diagnosis on a worker whose volume was
full: the real errno was ENOSPC and nothing reported it.
The loader now creates the directory itself and returns the underlying
error with the path attached.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]
* fix(vram): persist remote probe metadata
The startup warmer repeated remote size and GGUF metadata probes after every restart because both caches lived only in memory. Store successful HTTP probes for 24 hours so frequent restarts reuse the prior results.
Bound the cache, reject invalid records, and purge it when gallery data changes. Local model files continue to bypass persistence.
Assisted-by: Codex:gpt-5
* fix(vram): check temporary file cleanup
The lint gate rejects the unchecked cleanup call in the persistent cache writer.
Assisted-by: Codex:gpt-5.6 [golangci-lint]
* fix(vram): make persistent cache optional
Remote metadata probes can transfer enough data that operators need
control over disk reuse and startup warming. Gallery autoload now gates
both behaviors, and the runtime setting applies changes immediately.
Assisted-by: Codex:gpt-5
* fix(ui): expose gallery startup pre-warm
The existing gallery autoload setting also gates the startup metadata warmer. Name both effects in Settings so operators can find the requested boot control.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* docs: design configurable copy buffering
Document the context-aware copy buffer option and its validation plan.
Assisted-by: Codex:gpt-5
* docs: design durable distributed staging operations
Assisted-by: Codex:gpt-5
* docs: design distributed model config revisions
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
* feat(config): add stable model revisions
Hash typed model configuration and effective protobuf options deterministically for distributed revision comparisons.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
* feat(worker): acknowledge exact model stops
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
* feat(nodes): track model config revisions
Assisted-by: Codex:GPT-5 [apply_patch]
* fix(distributed): retry quarantined model cleanup
Stop quarantined replicas by exact process identity, retain failed cleanup as durable capped retries, and compare-and-delete only the claimed registry row. Process one sufficiently leased row at a time so multiple frontends cannot duplicate slow cleanup work.
Assisted-by: Codex:gpt-5
* fix(distributed): bind loads to config revisions
Assisted-by: Codex: GPT-5 [OpenAI Codex]
* fix(modeladmin): apply config revisions consistently
Route model edits, patches, state changes, deletion, and peer refreshes through the same revision lifecycle. Quarantine stale replicas before exact cleanup and report durable pending cleanup without failing successful config writes.
Assisted-by: Codex: GPT-5 [OpenAI Codex]
* feat(distributed): expose model config revision state
Document replica revision observability and durable cleanup behavior. Keep pending cleanup explicit in model mutation responses and verify endpoint contracts expose revision state without serialized load options.
Assisted-by: Codex:GPT-5 [OpenAI Codex]
* test(distributed): cover model revision convergence
Exercise cross-frontend quarantine, stale replay rejection, exact cleanup retry, worker re-registration, and current-generation replica convergence against the distributed PostgreSQL harness.
Assisted-by: Codex:gpt-5
* fix(distributed): pass config revision CI checks
Keep configured gallery sources out of authoritative runtime snapshots only after validating their real schema, and harden rollback snapshots against symlink races and non-regular files.
Assisted-by: Codex: GPT-5 [OpenAI Codex]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
