* fix(capabilities): report the per-request context with split KV slots
With parallel slots and kv_unified:false, llama.cpp gives each slot
n_ctx/n_parallel, padded up to a multiple of 256. /models/capabilities
still reported the full n_ctx. A client that budgets a request against
context_size then overflows at a fraction of it.
EffectiveRequestContextSize returns the per-slot size in that case and
the full context otherwise. With the unified KV cache, the grpc-server
default, one request may use all of n_ctx. The capabilities endpoint
and the router's prompt trimmer now use it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(openai): return an HTTP error when a stream fails before any chunk
A streamed chat request set the SSE headers, then waited for the
backend. When the backend failed before the first token, LocalAI sent
a 200 with a `data: {"error":...}` chunk and [DONE]. Clients that do
not parse error chunks saw an empty reply. cogito's LocalAI client was
one of them: nib users got "streaming decision produced no content"
instead of the context overflow that caused it.
Nothing has been written at that point, so the handler now returns the
error as a normal HTTP response. A failure after the first chunk keeps
the in-stream error chunk.
A prompt that exceeds the context is now a 400 on both paths, as in
the OpenAI API and llama-server, and no longer a 500. The message is
kept whole, because clients read the token counts from it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(e2e): check the error from closing the response body
golangci-lint's errcheck flags the unchecked resp.Body.Close in the
new pre-stream error helper.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
feat(kimodocpp): add API and backend request observability
Capture animation requests, phase timings, output metadata, and failures in traces. Record correct API error statuses and cover completed, running, failed, and disabled tracing.
Assisted-by: Codex:GPT-6
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(stablediffusion-ggml): bump sd.cpp for Qwen-Image 2.1, fix RPC build
Bump stable-diffusion.cpp to c678dfe70, which adds Qwen-Image 2.1
support (leejet/stable-diffusion.cpp#1994).
The same range pulls a ggml update that adds GGML_OP_SAGE_ATTN but
does not update the GGML_OP_COUNT static_assert in ggml-rpc.h. We build
with SD_RPC=ON and upstream CI does not, so every backend image failed
to compile (see #12170).
Add a sync-rpc-op-count step after checkout. It sets the ggml-rpc.h
assert to the count that ggml.c asserts. The new op is appended before
GGML_OP_COUNT, so existing op ids on the wire do not change, and the
RPC handshake compares only major and minor versions. When upstream
fixes the header, the step does nothing, so future automated bumps
stay green.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): add Qwen-Image 2.1 GGUF for stablediffusion-ggml
Add qwen-image-2.1-q4_k-ggml, with qwen-image-2.1-q8_0-ggml as a
variant, from leejet/Qwen-Image-2.1-GGUF. The config follows the
upstream stable-diffusion.cpp recipe: Qwen3-VL-8B-Instruct text
encoder, the Qwen-Image 2.1 VAE, cfg scale 6, euler sampler.
The bundle also pulls the Qwen3-VL mmproj as llm_vision_path, so that
image editing with reference images works. The text encoder and mmproj
use the same filenames and checksums as the qwen3-vl-8b-instruct
entry, so both entries share the files on disk.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Track successful header authentication before allowing cross-site requests
to bypass CSRF checks. Arbitrary headers on unauthenticated servers and
cookie-authenticated requests no longer grant an exemption.
Share the production CSRF middleware with multipart tests, add regression
coverage for credential sources, and document the exemption behavior.
Assisted-by: Codex:gpt-6 golangci-lint
Signed-off-by: Richard Palethorpe <io@richiejp.com>
`make protoc` fetched the protoc release with `curl -L` and no `-f`. When
GitHub answered with an error, curl saved the error page as protoc.zip and
exited 0, and the build then failed at unzip with "cannot find zipfile
directory", which points at the archive rather than at the download.
This happened for real: GitHub answered a CI cluster with a 504 for release
downloads for over an hour, and every backend build that runs
`make protogen-go` failed with the unzip message.
`-f` makes curl fail on the HTTP status and write nothing, so the error names
the real cause. `--retry-all-errors` also retries HTTP errors, which rides
out a short blip; a longer outage still fails, now with the right message.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
On a single-node install nothing in Operate listed the models loaded on
this machine or let an admin stop one. The System page that did was
retired in #11548, and its replacements (the Nodes workbench) only work
in distributed mode. The Nodes page also mis-detected single-node mode:
the cluster routes are not registered there, so /api/nodes answers 404,
but only 503 was treated as "distributed off", which sent every
single-node install to the empty worker-registration card. The rail hid
the entry anyway.
Nodes route on a single node becomes "This machine":
- the Nodes page's VRAM / RAM / CPU / models-disk gauges, fed from this
host by mapping /api/resources onto the worker heartbeat fields
- a memory bar splitting host RAM by running model
- a running-models table (backend, RSS, CPU share, uptime, PID) with
search, sorting, logs and a confirmed Stop
- the distributed setup behind an "Add machines" button
The Operate overview gains a "Running now" preview (heaviest five, with
Stop) on single node and a pointer to Nodes > Running models on a
cluster. The rail shows "This machine" in Runtime with a running count.
Backend, additive only:
- /system: each loaded model carries a `process` block (pid, rss_bytes,
memory_percent, cpu_percent, started_at). A sampler keeps one gopsutil
handle per PID so CPU is the share since the previous poll rather than
the lifetime average; it is omitted on the first reading.
- /api/resources: host `cpu` and models-path `disk`, the same readings
workers send in their heartbeat.
Also fixes the fleet tables widening the page on phones: the headers'
absolutely positioned sr-only labels escaped the scroll wrapper.
Assisted-by: Claude:claude-opus-5 [Playwright]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
An alias config is a pure redirect with no backend of its own, so the
capabilities listing described it from its stub: no capabilities, no
modalities, and the default 4096 context_size. Clients that size their
context budget from this endpoint (nib, for one) then compacted every
turn against a model that really serves 100k.
Resolve the alias and report the target's capabilities, modalities and
context_size under the alias's id. A dangling or chained alias now
reports no enrichment instead of defaults no model runs with.
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat(oci): pull ORAS artifacts into a directory
Galleries published to an OCI registry arrive as an ORAS artifact whose
layers carry their tree path in org.opencontainers.image.title. Nothing in
pkg/oci could read a non-image manifest, so add PullArtifact: it lays the
layers out under a destination directory and returns the manifest digest so
callers can pin and verify signatures on it.
A registry is remote input and this writes files, so the manifest is
validated before any byte is fetched. A title that is absolute, escapes the
destination or is missing is refused, an artifact of an unexpected
artifactType is refused, and both the layer count and the total size are
capped with caller-settable limits.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(gallery): fetch and verify galleries published as OCI artifacts
A gallery whose URL is oci://host/repo:tag is now pulled as an ORAS
artifact and its index.yaml read from the unpacked tree, alongside the
existing HTTP and file sources. Mirrors, the per-candidate timeout, the
failure cooldown and the last known good copy work the same for both
schemes, so a gallery can mirror an OCI primary with an HTTP fallback.
When the gallery declares a verification policy the artifact reference is
resolved to its manifest digest, the publisher signature is checked
against that digest, and only then is the same digest pulled. Nothing
unverified is written to disk or parsed. With strict integrity mode on,
an OCI gallery without a verification policy is refused instead of
warned about, which is what the flag already does for backend installs.
The pull lands in a staging directory that is renamed into the cache only
once the whole tree is on disk and the index reads back as an index, so a
failed or interrupted pull leaves nothing a later fetch would serve. A
fetch that finds a fresh cache entry does not contact the registry.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(gallery): resolve entry urls relative to the gallery root
A gallery published as a self-contained tree, which is what an OCI
gallery is, names its base configs by their place in that tree. Such a
url reached the HTTP client verbatim and no entry of the gallery could
be installed.
A scheme-less entry url now resolves against the root the index came
from: the unpacked artifact for an oci:// gallery, and the directory of
the index URL otherwise. The relative path may not climb out of that
root, reusing the guard the artifact puller already applies to layer
titles, and an entry that does is dropped from the listing rather than
failing the whole gallery. An entry that names its own scheme is
untouched, so .ref indirection and local github: resolution behave as
before.
Reading an entry of an OCI gallery also needs the unpacked artifact
directory as its trusted root, since the cache is deliberately a sibling
of the models directory the downloader otherwise confines a file read
to.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(oci): restrict artifact file permissions
Create artifact files with owner-only permissions to satisfy G302.
Document the existing path validation for the G304 scanner finding.
Check the permissions of downloaded layers in the artifact test.
Assisted-by: Codex:gpt-6 gosec
* fix(oci): confine artifact writes to their root
Use root-relative filesystem operations to reject escapes through existing
directory symlinks. Retain owner-only permissions and add a symlink escape
regression test, without suppressing the path traversal finding.
Assisted-by: Codex:gpt-6
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
A registry without the OCI 1.1 referrers API leaves the referrers index to
the signing client, and cosign fills each entry's artifactType from the
manifest's config media type rather than from its artifactType. CNCF
distribution 3.0.0 has no referrers route at all, so on that registry every
correctly signed image reads as unsigned: the verifier filtered the index
entries by artifactType and matched nothing.
The verifier now asks the referrer manifest what it is when no entry
advertises itself, checking the manifest's own artifactType and its first
layer. The fast path is unchanged, the second pass runs only when the first
finds nothing, and it is bounded so a heavily referenced image cannot turn
one verification into an unbounded walk.
Found by a real release: the images and signatures were correct and no
client could verify them.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ci): sign backends in the format we verify
Published backend images carry legacy simplesigning signatures: config
application/vnd.dev.cosign.artifact.sig.v1+json, layer
application/vnd.dev.cosign.simplesigning.v1+json, attached as referrers with
no artifactType. pkg/oci/cosignverify accepts only new-format Sigstore
bundles and refuses the legacy format on purpose, so LOCALAI_REQUIRE_BACKEND_INTEGRITY
and any gallery verification: block reject our own backends.
The cause is one missing flag. backend-signing.md has prescribed
`--new-bundle-format` all along; backend_merge.yml never passed it, and the
comment beside the cosign pin claims v2.4.1 emits the current bundle format
by default, which it does not. The flag arrives in v2.5.0, so the pin moves
to v2.6.5 and both sign calls pass it.
Verified against a published image today
(quay.io/go-skynet/local-ai-backends:latest-gpu-nvidia-cuda-12-llama-cpp):
both referrers are simplesigning, so nothing verifies.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* test(ci): require the Sigstore bundle flags
The signing test rejected the flags required by the pinned cosign release.
Require a bundle flag for each registry while retaining recursion and OCI
referrer checks.
Assisted-by: Codex:gpt-6
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Report input tokens and frame-step output units in response metadata and
record them through the existing usage accounting pipeline. Preserve the
accounting rule and model-specific dimensions as JSON without extending
the gRPC schema for each modality.
Expose animation usage only under metadata.usage, validate counts before
recording, and document the response contract and loaded-model location.
Add coverage for transport, defaults, failures, persistence, and recording
requests once with statistics enabled or disabled.
Assisted-by: Codex:GPT-6
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Use a regular (non-editable) pip/uv install so the package lands in the
venv site-packages. An editable finder records the builder source path,
which breaks after the backend is copied out of the image (#9162).
Adds a regression test (scripts/build/vllm-omni-install_test.sh) that
verifies imports survive relocation with a regular install and fail with
an editable install.
Supersedes #12040 (DCO not signed by contributor).
Assisted-by: MAKI:regolo/glm5.2
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Show partial layer offload and CPU expert placement for llama-cpp.
Correct the documented gpu_layers default to match the backend config.
Refs #10557
Assisted-by: Codex:gpt-6
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
GetImageDigest calls name.ParseReference, which cannot parse the oci://
scheme. Self-hosted registries must set the scheme to be recognised by
LooksLikeOCI, so without stripping it they silently lose upgrade detection.
Strips the prefix at the call site, matching what the other two call
sites already do (pkg/downloader/uri.go, gallery/importers/llama-cpp.go).
Supersedes #12120 (DCO not signed by contributor, branch had unrelated
history needing rebase).
Assisted-by: MAKI:regolo/glm5.2
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
