The unpacked oci:// gallery cache and the last known good copy of the
index were keyed on the gallery URL only. After an operator tightened a
gallery's verification policy (added source_repository, moved not_before
forward), content verified under the older policy, or under none, was
still served for up to an hour from the unpacked cache, and indefinitely
from the last known good copy while fetches failed. A fetch refused by
signature verification also fell back to that last known good copy, so
a refusal became a silent downgrade. Turning strict integrity on did not
stop an unverified cached copy from being served either.
Name both caches by the URL plus a stable hash of the policy. A gallery
without a policy keeps its old URL-only name, so existing caches stay
usable. A fetch refused by the policy, or by strict integrity, is now
reported and never answered with a cached copy; a network failure still
falls back, but only to a copy verified under the current policy. The
strict integrity check runs before the cache is read.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The gallery agent started adding documentation sections for individual
models (NeoHorse, Maple-Preview, Hy-MT2, Instella-MoE, Spark-X2.5,
Occamy, etc.) to the model-gallery page. This clutters the general
gallery documentation with model-specific install instructions and
descriptions that belong in the gallery index or model cards, not in
the feature docs.
Remove all per-model sections. The page now covers only the gallery
infrastructure: how galleries work, how to add them, the API, variants,
and the stable-diffusion/whisper examples that were already there.
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): add GLiNER2.5 NER via TokenClassify
Wire the vllm-cpp backend to the C ABI NER surface (vllm_gliner_ner,
ABI v27) so LocalAI can serve zero-shot named entity recognition through
the existing TokenClassify gRPC method.
backend.go: TokenClassify method on *VllmCpp calls vllm_gliner_ner with
the text and labels, copies the C-owned entity array into protobuf
TokenClassifyEntity messages, and frees the result.
govllmcpp.go: cNerEntity and cNerResult Go POD mirrors matching the C
structs; vllmGlinerNer and vllmNerResultFree purego bindings; abiVersion
bumped 26 -> 27.
options.go: ner_labels, ner_threshold, ner_max_width parsed from
engine_args.
pkg/grpc: ClassifyModel interface and TokenClassify server handler
(follows the Embedding locking pattern).
core/config: vllm-cpp backend declares MethodTokenClassify and
UsecaseTokenClassify.
docs/content/features/vllm-cpp.md: NER section documenting the
engine_args keys and the host-forward contract.
Assisted-by: MAKI:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): correct NER pointer lint directive
Use the govet directive for the C-owned NER array, matching the other
purego pointer conversions. The array remains valid until its deferred
free; the misspelled directive caused CI to flag this conversion.
Assisted-by: Codex:gpt-6 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vllm-cpp): add kev-compatible SystemOne API endpoints
Add POST /v1/systemone, /v1/systemone/permute, and
/v1/systemone/separate to LocalAI, mirroring the kev project's
structured-extraction API. Each endpoint runs zero-shot NER over the
rendered state text and builds kev-compatible answers for three question
types: noul (binary entity presence), choice (pick one option), and
score (pick one level).
The TokenClassifyRequest proto gains a `repeated string labels` field so
each question can supply its own labels at inference time, and
TokenClassifier gains TokenClassifyWithLabels for per-call label
selection. The vllm-cpp backend uses request labels when non-empty,
falling back to configured ner_labels then the built-in defaults.
Helpers (renderState, softmax, choiceConfidence, scoreConfidence, r2)
are ported from kev/api.py and mirrored in vllm.cpp's api_server.cpp so
both servers produce the same answer shape.
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): suppress gosec G404 on seeded permutation RNG
The SystemOne permute endpoint uses math/rand with a caller-supplied
seed for reproducible option permutations, matching kev's random.seed.
gosec flags this as G404 (weak RNG). Add #nosec with a comment naming
the intent: this is reproducibility, not cryptography.
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(vllm-cpp): bump vllm.cpp pin to GLiNER2.5 merge commit
Advance VLLM_CPP_VERSION from f3cd97e to 5058268d, the commit that
landed GLiNER2.5 zero-shot NER support (PR #3224) in vllm.cpp. This
brings the DeBERTa v2 encoder, GLiNER2 boundary head, C ABI NER
functions, and server endpoints into the LocalAI vllm-cpp backend.
The ABI version (27) and Go struct mirrors already match.
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm-cpp): use instruction text as NER label in SystemOne handler
The SystemOne handler was passing question IDs as NER labels for noul
questions and bare key names for choice questions, so the model never
matched any entities. Port the label mapping from vllm.cpp's
ParseSystemOneBody:
- noul: use the rendered instructions field (with instr alias) as the
NER label, not the question ID
- choice: use optionText(name, desc) — "name: description" or "name"
when the description is null/empty — not the bare key
- score: already correct (rendered criteria text)
- permute: shuffle indices and build parallel key/label arrays so the
NER call uses the optionText labels while the response is keyed by
the original option names
Also add the instructions field to the SystemOneQuestion schema struct
(accepted alongside the instr backward-compat alias).
Verified end-to-end against the real GLiNER2.5 model: noul questions
now find "Apple Inc. is" (organization, 0.999) and "Tim Cook is"
(person, 0.852) where they previously returned zero entities.
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:regolo/glm5.2 [maki]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* chore(model gallery): 🤖 add new models via gallery agent
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* feat(gallery): add NeoHorse-1-9B GGUF variants
Add the official Q4_K_M, Q5_K_M, and Q8_0 builds with revision-pinned
weights and verified SHA256 values.
Assisted-by: Codex:gpt-6
* chore(model gallery): 🤖 add new models via gallery agent
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* feat(gallery): add Qwen3.8 35B Distill variants
Add Q4_K_M, Q5_K_M, and Q8_0 builds with the vision projector.
Pin publisher revisions and document installation and variant selection.
Assisted-by: Codex:gpt-6
* feat(gallery): add ByteShape Qwen3.8 variants
Offer five ShapeLearn GGUF builds with a vision projector and MTP.
Pin downloads to a verified HF revision and document variant selection.
Assisted-by: Codex:gpt-6
* feat(gallery): add Flash Next GSQ-RCO variants
Offer Q2_0, IQ2_XS, and IQ3_XXS builds with both model shards and the
vision projector. Pin and verify download hashes and document how to
select each variant.
Assisted-by: Codex:GPT-6
* feat(gallery): add Occamy-1.0 GGUF variants
Add the publisher's Q4_K_M and Q8_0 builds with the F16 vision projector.
Link the builds as variants and pin downloads to a verified revision.
Document installation and the source tokenizer's NFC requirement.
Assisted-by: Codex:gpt-6
* fix(gallery): set MiniCPM5 context at the top level
The Q4 and Q8 overrides place context_size inside parameters, where
PredictionOptions ignores it. Move it beside parameters so both
builds use the intended 8,192-token context, matching F16.
Assisted-by: Codex:GPT-6
* feat(gallery): add Hy-MT2 7B GGUF variants
Offer the official Q4_K_M, Q6_K, and Q8_0 builds for translation.
Pin the downloads and document installation and translation prompts.
Assisted-by: Codex:gpt-6
* feat(gallery): add Maple-Preview GGUF variants
Offer four ternary builds through the existing llama.cpp backend.
Use the publisher's CPU settings and embedded chat template.
Pin downloads and verify SHA256 values against two HF metadata sources.
Document installation and explicit variant selection.
Assisted-by: Codex:gpt-6
* feat(gallery): add Qwen3.8 Cyber GGUF variants
Offer IQ4_XS and Q8_0 builds with the matching BF16 vision projector.
Pin download revisions and document automatic and explicit selection.
Assisted-by: Codex:GPT-6
* feat(gallery): add official NeoHorse 4B variants
Offer the official Q5_K_M and BF16 GGUF builds alongside the existing
community quantizations. Pin both downloads and document variant selection.
Assisted-by: Codex:gpt-6
* chore(model gallery): 🤖 add new models via gallery agent
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* chore(model gallery): 🤖 add new models via gallery agent
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The ROCm/hipblas image does not set HSA_OVERRIDE_GFX_VERSION,
ROCBLAS_USE_HIPBLASLT, HSA_XNACK, or HSA_ENABLE_SDMA. Remove the
misleading parenthetical so readers know to pass them explicitly.
Fixes#12071
Assisted-by: Cursor:Composer
Signed-off-by: lei_lei <imleilei123@gmail.com>
Co-authored-by: mudler-agent <mudler-bot@c3os.io>
Models whose options carry no explicit backend: open their session on the
CPU backend even in accelerator images. The gallery entries carry
backend:best since #11892; this covers hand-written model configurations
the same way, per deployment: the environment variable supplies the
fallback, an explicit backend: option always wins (merged beside the
existing threads and maingpu fallbacks), and validation reuses the
option parser.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
feat(kimodocpp): add API and backend request observability
Capture animation requests, phase timings, output metadata, and failures in traces. Record correct API error statuses and cover completed, running, failed, and disabled tracing.
Assisted-by: Codex:GPT-6
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(stablediffusion-ggml): bump sd.cpp for Qwen-Image 2.1, fix RPC build
Bump stable-diffusion.cpp to c678dfe70, which adds Qwen-Image 2.1
support (leejet/stable-diffusion.cpp#1994).
The same range pulls a ggml update that adds GGML_OP_SAGE_ATTN but
does not update the GGML_OP_COUNT static_assert in ggml-rpc.h. We build
with SD_RPC=ON and upstream CI does not, so every backend image failed
to compile (see #12170).
Add a sync-rpc-op-count step after checkout. It sets the ggml-rpc.h
assert to the count that ggml.c asserts. The new op is appended before
GGML_OP_COUNT, so existing op ids on the wire do not change, and the
RPC handshake compares only major and minor versions. When upstream
fixes the header, the step does nothing, so future automated bumps
stay green.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): add Qwen-Image 2.1 GGUF for stablediffusion-ggml
Add qwen-image-2.1-q4_k-ggml, with qwen-image-2.1-q8_0-ggml as a
variant, from leejet/Qwen-Image-2.1-GGUF. The config follows the
upstream stable-diffusion.cpp recipe: Qwen3-VL-8B-Instruct text
encoder, the Qwen-Image 2.1 VAE, cfg scale 6, euler sampler.
The bundle also pulls the Qwen3-VL mmproj as llm_vision_path, so that
image editing with reference images works. The text encoder and mmproj
use the same filenames and checksums as the qwen3-vl-8b-instruct
entry, so both entries share the files on disk.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Track successful header authentication before allowing cross-site requests
to bypass CSRF checks. Arbitrary headers on unauthenticated servers and
cookie-authenticated requests no longer grant an exemption.
Share the production CSRF middleware with multipart tests, add regression
coverage for credential sources, and document the exemption behavior.
Assisted-by: Codex:gpt-6 golangci-lint
Signed-off-by: Richard Palethorpe <io@richiejp.com>
On a single-node install nothing in Operate listed the models loaded on
this machine or let an admin stop one. The System page that did was
retired in #11548, and its replacements (the Nodes workbench) only work
in distributed mode. The Nodes page also mis-detected single-node mode:
the cluster routes are not registered there, so /api/nodes answers 404,
but only 503 was treated as "distributed off", which sent every
single-node install to the empty worker-registration card. The rail hid
the entry anyway.
Nodes route on a single node becomes "This machine":
- the Nodes page's VRAM / RAM / CPU / models-disk gauges, fed from this
host by mapping /api/resources onto the worker heartbeat fields
- a memory bar splitting host RAM by running model
- a running-models table (backend, RSS, CPU share, uptime, PID) with
search, sorting, logs and a confirmed Stop
- the distributed setup behind an "Add machines" button
The Operate overview gains a "Running now" preview (heaviest five, with
Stop) on single node and a pointer to Nodes > Running models on a
cluster. The rail shows "This machine" in Runtime with a running count.
Backend, additive only:
- /system: each loaded model carries a `process` block (pid, rss_bytes,
memory_percent, cpu_percent, started_at). A sampler keeps one gopsutil
handle per PID so CPU is the share since the previous poll rather than
the lifetime average; it is omitted on the first reading.
- /api/resources: host `cpu` and models-path `disk`, the same readings
workers send in their heartbeat.
Also fixes the fleet tables widening the page on phones: the headers'
absolutely positioned sr-only labels escaped the scroll wrapper.
Assisted-by: Claude:claude-opus-5 [Playwright]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(oci): pull ORAS artifacts into a directory
Galleries published to an OCI registry arrive as an ORAS artifact whose
layers carry their tree path in org.opencontainers.image.title. Nothing in
pkg/oci could read a non-image manifest, so add PullArtifact: it lays the
layers out under a destination directory and returns the manifest digest so
callers can pin and verify signatures on it.
A registry is remote input and this writes files, so the manifest is
validated before any byte is fetched. A title that is absolute, escapes the
destination or is missing is refused, an artifact of an unexpected
artifactType is refused, and both the layer count and the total size are
capped with caller-settable limits.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(gallery): fetch and verify galleries published as OCI artifacts
A gallery whose URL is oci://host/repo:tag is now pulled as an ORAS
artifact and its index.yaml read from the unpacked tree, alongside the
existing HTTP and file sources. Mirrors, the per-candidate timeout, the
failure cooldown and the last known good copy work the same for both
schemes, so a gallery can mirror an OCI primary with an HTTP fallback.
When the gallery declares a verification policy the artifact reference is
resolved to its manifest digest, the publisher signature is checked
against that digest, and only then is the same digest pulled. Nothing
unverified is written to disk or parsed. With strict integrity mode on,
an OCI gallery without a verification policy is refused instead of
warned about, which is what the flag already does for backend installs.
The pull lands in a staging directory that is renamed into the cache only
once the whole tree is on disk and the index reads back as an index, so a
failed or interrupted pull leaves nothing a later fetch would serve. A
fetch that finds a fresh cache entry does not contact the registry.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(gallery): resolve entry urls relative to the gallery root
A gallery published as a self-contained tree, which is what an OCI
gallery is, names its base configs by their place in that tree. Such a
url reached the HTTP client verbatim and no entry of the gallery could
be installed.
A scheme-less entry url now resolves against the root the index came
from: the unpacked artifact for an oci:// gallery, and the directory of
the index URL otherwise. The relative path may not climb out of that
root, reusing the guard the artifact puller already applies to layer
titles, and an entry that does is dropped from the listing rather than
failing the whole gallery. An entry that names its own scheme is
untouched, so .ref indirection and local github: resolution behave as
before.
Reading an entry of an OCI gallery also needs the unpacked artifact
directory as its trusted root, since the cache is deliberately a sibling
of the models directory the downloader otherwise confines a file read
to.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(oci): restrict artifact file permissions
Create artifact files with owner-only permissions to satisfy G302.
Document the existing path validation for the G304 scanner finding.
Check the permissions of downloaded layers in the artifact test.
Assisted-by: Codex:gpt-6 gosec
* fix(oci): confine artifact writes to their root
Use root-relative filesystem operations to reject escapes through existing
directory symlinks. Retain owner-only permissions and add a symlink escape
regression test, without suppressing the path traversal finding.
Assisted-by: Codex:gpt-6
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Report input tokens and frame-step output units in response metadata and
record them through the existing usage accounting pipeline. Preserve the
accounting rule and model-specific dimensions as JSON without extending
the gRPC schema for each modality.
Expose animation usage only under metadata.usage, validate counts before
recording, and document the response contract and loaded-model location.
Add coverage for transport, defaults, failures, persistence, and recording
requests once with statistics enabled or disabled.
Assisted-by: Codex:GPT-6
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Use a regular (non-editable) pip/uv install so the package lands in the
venv site-packages. An editable finder records the builder source path,
which breaks after the backend is copied out of the image (#9162).
Adds a regression test (scripts/build/vllm-omni-install_test.sh) that
verifies imports survive relocation with a regular install and fail with
an editable install.
Supersedes #12040 (DCO not signed by contributor).
Assisted-by: MAKI:regolo/glm5.2
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Show partial layer offload and CPU expert placement for llama-cpp.
Correct the documented gpu_layers default to match the backend config.
Refs #10557
Assisted-by: Codex:gpt-6
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Expose a negative_prompt string parameter on the image endpoints,
matching Stable Diffusion WebUI / vLLM-Omni conventions. When both the
negative_prompt parameter and a '|'-suffixed negative prompt in the
main prompt are present, they are joined with a comma so callers can
keep a global negative prompt in negative_prompt and add per-image
negative tags after '|'.
Assisted-by: Pi: DeepSeek V4 Pro
Signed-off-by: Fedor Zuev <Fedor.Zuev@gmail.com>
Consolidates all 25 pending gallery bot PRs into a single merge to resolve
the conflict cascade — every PR branched from a different point in master
and they all touch gallery/index.yaml, so merging them individually was
blocked by constant conflicts.
Changes:
- gallery/index.yaml: +739 lines (new model entries and fixes)
- docs/content/features/model-gallery.md: +116 lines (new model docs)
- docs/content/features/audio-cpp.md: +12 lines (Sortformer checksum fix)
Entry count: 1597 -> 1890 (293 new entries, no duplicates, YAML validated).
Supersedes: #11986#11992#11994#11996#11999#12002#12017#12019#12021#12025#12027#12029#12032#12036#12037#12038#12041#12042#12043#12047#12050#12064#12065#12066#12118
Assisted-by: MAKI:regolo/glm5.2
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vulkan): preserve host ICD discovery for packaged backends
Add bundled Mesa manifests through VK_ADD_DRIVER_FILES instead of replacing the system driver list. Merge inherited and model-specific additive paths while preserving explicit operator overrides, with regression coverage.
Assisted-by: Codex:gpt-5 golangci-lint
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(3d): add Kimodo CPU and Vulkan animation backend
Introduce a distinct animation capability and model-described 3D operations, with a typed /3d/animate API, RPC transport, distributed media staging, permissions, and tracing.
Add a persistent kimodo.cpp adapter, skeleton GLB export, CPU/Vulkan packages, model and backend galleries, importer support, CI builds, and documentation. Adapt Studio inputs to each model and provide real-time skeleton playback, seeking, and history.
Cover backend validation, packaging, API behavior, importer inventories, distributed staging, and Studio workflows. Validate real-model CPU/Vulkan generation and deploy the integration to the local QA instance.
Assisted-by: Codex:gpt-5 golangci-lint
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(kimodocpp): adopt monolithic encoders and resident inference
Update upstream for resident weights, packed execution paths, and cached motion graphs. Default to all 32 text layers while retaining configurable streaming and legacy bundle support.
Use monolithic Q8_0 encoders by default and offer all six published quantizations through the gallery and importer. Refresh pinned hashes, tests, and documentation; remove the obsolete thread patch and ensure cached source checkouts follow the upstream pin.
Validated CPU and Vulkan generation, lower-bit streaming, gallery/importer suites, packaging, lint, and cold/warm Studio generation on localai-dev.
Assisted-by: Codex:gpt-5 golangci-lint
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
The node restructure hid backend logs and split related controls
across inconsistent layouts.
Restore contextual log actions and align the detail page with the
fleet dashboard. Make multi-node selection clear and accessible.
Assisted-by: Codex:gpt-5 Playwright ESLint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(credentials): parse and match download credential rules
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): keep secrets out of parse errors and tighten URL matching
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(credentials): resolve secrets lazily and authenticate HTTP per hop
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): redact secrets in nested and store formatting
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(credentials): add registry keychain and oras credential adapters
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): match repository rules for Docker Hub in the oras adapter
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(downloads): authenticate HTTP downloads and gallery reads from the credentials store
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(oci): authenticate registry pulls, resumes, blobs and cosign from the credentials store
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(cli): load download credentials from --credentials-file
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(credentials): correct the local-network registry rules
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): keep secrets out of match and YAML parse errors
A match that fails to parse is no longer quoted in the Parse error, since
it may be a URL with a token in it. Userinfo is detected before the scheme
check, so ftp://user:token@host is refused as userinfo, and a match with a
query string or fragment is refused because it can never apply and a query
string is where signed URLs carry their token. Every YAML decode error is
now redacted, not only type errors: quoted scalars such as a secret under a
mismatched !!int tag are replaced and unquoted map keys are cut off.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(downloads): make auth errors name the real cause and never retry unresolved secrets
AuthError now appends its cause, so a registry's DENIED or UNAUTHORIZED
detail reaches the operator. HTTP auth errors print only the status text in
place of the cause, because the downloader builds that cause from the
requested URL, which can carry a signed query string. Registry pulls say
that docker config credentials were tried too, and a download that carried
a caller-provided credential (WithBearerToken, or an explicit authorization
on gallery reads) reports that credential as rejected instead of blaming
the store.
The Range probe for a leftover partial file now returns an unresolved
secret as a permanent error, like the download request already did.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): keep oras pulls anonymous on a broken docker helper and close bodies
When docker config names a credsStore helper that cannot run, the oras
credential func now logs at debug and returns no credential, so public
pulls keep working as they did before the adapter existed. The transport
closes the request body when a rule's secret cannot be resolved, as the
RoundTripper contract requires. The redirect spec now uses a custom header
rule on the origin, which net/http would not strip on its own, to prove the
transport does not carry credentials to the next hop.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(oci): cover FetchImageBlob authentication against a private registry
FetchImageBlob now has a spec that pulls a layer blob by digest from a
basic-auth registry through the oras credential adapter, and one that shows
the same fetch fails when no rule matches. oras only speaks HTTPS here, so
the registry serves TLS and the spec points http.DefaultTransport, which
retry.DefaultClient falls back to, at the test server's client for its
duration instead of adding a transport seam to production code.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(credentials): document auth error wording, ollama manifests and registry tokens
The errors section now lists the registry and provided-credential messages
and says the server's reason is appended. ollama:// manifests are fetched
without credentials, so only blob downloads use the file. GHCR, Docker Hub
and Quay need basic auth with the token as password, and match rules must
not carry a query string or fragment. The backend gallery docs and the
container troubleshooting section now point to the private sources page.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): document trusted file path
The credentials path comes from operator configuration. Mark the file
read with a scoped G304 explanation to resolve the gosec false positive.
Assisted-by: Codex:gpt-6 gosec
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(cli): benchmark configured text models
Compare model aliases through a running LocalAI server. Report full
request latency and server-reported throughput with raw JSON samples.
Keep warmups separate and fail before writing results on request errors.
Assisted-by: Codex:GPT-6
* fix(cli): satisfy benchmark error checks
Explicitly discard errors from buffered report writes, HTTP response cleanup, and test server writes to pass errcheck without changing behavior.
Assisted-by: Codex:gpt-6 golangci-lint
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The Ministral 3 14B Reasoning entry inherits a Mistral 0.3 prompt and
JSON parser. Its name-first tool calls can therefore reach clients as
plain text.
Use the embedded template and llama.cpp's native tool parser. Document
migration for installed configurations, which gallery updates do not
rewrite.
Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* [gallery] feat: add Orukeet to the existing NeMo speech backend
Assisted-by: Codex:gpt-6
Signed-off-by: Nathan Roll <nathan@oruk.ai>
* docs(nemo): remove model-specific instructions
Keep the backend guide focused on model families, as requested by mudler.
Remove the gallery limitation instead of restoring an outdated claim.
Assisted-by: Codex:GPT-6
Signed-off-by: Nathan Roll <nathan@oruk.ai>
---------
Signed-off-by: Nathan Roll <nathan@oruk.ai>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
grpc::ServerWriter::Write() returns false once the peer is gone, and
PredictStream ignored that result at every call site. The handler kept
pulling decoded tokens and writing them into a dead stream, so the
llama.cpp slot stayed busy until the generation ended on its own terms.
A model configured with max_tokens 0 and a large context ends on its own
terms only at the context limit. On a 35B model at ~41 t/s a 120k context
is about fifty minutes, and a slot held that long is a slot every other
request for that model queues behind. Two abandoned requests were enough
to make a node with free VRAM and a healthy control plane serve nothing:
new requests timed out waiting for a slot, each timeout abandoned another
generation, and the node fell further behind the longer it ran.
Track the peer instead. The first failed write retires it for good, since
a stream never recovers, and the RPC's own cancellation flag folds into
the same predicate so the loop has one condition to test. Returning early
is what frees the slot: ~server_response_reader() posts
SERVER_TASK_TYPE_CANCEL for whatever is still decoding.
TTSStream already checked Write(); this brings PredictStream in line.
Cancellation stays cooperative and is checked between decoded results, so
a batch already in flight may finish before the request stops.
Assisted-by: Claude:claude-opus-5
pinned: true was only honoured by the per-node watchdog. Every distributed
eviction path was pinned-blind: the router's LRU eviction (EvictLRU,
evictLRUAndFreeNode) and the replica reconciler's idle scale-down would
happily unload a pinned model — and since eviction is gated on
in_flight = 0, a pinned model became eviction-eligible the instant each
response completed. Under capacity pressure that surfaces as the backend
being freed immediately after every request (#11101).
Wire the model config loader into the router and reconciler through a new
PinnedModelResolver seam (mirroring ConcurrencyConflictResolver):
- EvictLRU passes the pinned set into FindLRUModel's query so the
next-oldest unpinned model is selected instead of the attempt failing
- evictLRUAndFreeNode filters pinned models inside its locked selection
- scaleDownIdle skips pinned models entirely: trimming to the floor still
means requests beyond the survivor's capacity pay a cold reload
Deliberate teardown (admin unload, model delete, node drain) intentionally
still applies to pinned models, as does dead-row reaping (state correction,
not eviction).
Regression specs verified to fail with the exclusion disabled.
Addresses the cluster-side eviction gap in #11101
Assisted-by: Claude Code:claude-fable-5 [Claude Code]
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
backend.stop was the one lifecycle subject a worker never answered. The
controller published and returned nil as soon as the local publish
succeeded, so a stop that killed nothing, and a stop that failed
outright, were indistinguishable from one that worked.
The unload endpoint calls model.unload and then StopBackend. Only the
first is acknowledged, so the endpoint answered 200 while the backend
kept running and held its VRAM, and its own "backend stop failed" branch
could never run. The worker logged the failure and nobody saw it.
Give the subject a reply. The worker now enumerates the process keys it
terminated and reports any per-process error, so StopBackend fails when
the stop failed. Resolving to nothing stays a success: stopping a backend
that is not running leaves the caller in the state it asked for, and
eviction paths stop already-gone models routinely. The empty list is what
says nothing matched, and ReportsStoppedProcesses is what makes that
emptiness trustworthy, the same way BackendDeleteReply handles it.
A worker built before this reply still receives the request and still
stops the backend, it only stays silent, so a timeout degrades to the old
assumption rather than failing every stop on a fleet mid-upgrade. Only
silence degrades: a transport error is still reported, because
UnloadRemoteModel skips its registry cleanup for a node it could not
reach and needs to keep hearing about that.
Assisted-by: Claude:claude-opus-5 golangci-lint
Adds FunASR/SenseVoice as a Python backend for speech-to-text with
support for CPU, CUDA 12/13, ROCm, Intel SYCL, L4T, and Apple MPS.
Co-authored-by: xingyifeng <xingyifeng@users.noreply.github.com>
Saved profiles previously resolved to one audio path and transcript, so
cloning backends could not use several examples of one personality.
Store ordered audio and transcript pairs while preserving the legacy
first-reference fields. Fish Speech and audio.cpp receive all pairs,
including on distributed workers. Other backends retain their
single-reference behavior.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Let studio users guide speech delivery for backends that support request instructions. Blank guidance stays out of requests and media history.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Clients cannot discover the named voices that an installed TTS model accepts without consulting backend-specific documentation. Expose voice metadata through the audio API and let custom model configs declare their own catalog.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The pinned audio.cpp revision supports HIP, but LocalAI neither builds a ROCm image nor accepts its backend option. AMD hosts therefore fall back to the CPU image.
Build and publish the HIP variant, connect it to AMD capability selection, and accept both upstream HIP names.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Expose diffusers audio pipelines through the existing sound-generation RPC. AudioLDM2 can now return PCM WAV output from the model gallery without a separate backend.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a NATS event contract and exact-residency provider for backend KV cache reports. Keep guessed request observations as the default routing source while maintaining the reported index for future producers.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a Darwin-only MLX-Video backend for LTX-2 and converted Wan checkpoints, expose it through the existing video API, and wire packaging, discovery, tests, docs, and an example.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Document how to connect LocalAI to the upstream RKLLM OpenAI-compatible server through cloud-proxy, including supported SoCs and current limitations.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
