The Vite build emitted path-absolute asset URLs (base: '/'). index.html
entry scripts and the favicon were rewritten to include the reverse-proxy
prefix in serveIndex, but two reference kinds are not in index.html and so
bypassed that rewrite:
- CSS `url()` font references (e.g. Font Awesome .woff2), which the browser
resolves relative to the stylesheet and which `<base href>` never affects
- lazily-imported route chunks, whose preload base came from the absolute
Vite base
Under a subpath mount (X-Forwarded-Prefix: /llm/) both were fetched from the
origin root, 404ing — missing-glyph "tofu" icons and broken lazy-loaded pages.
Switch Vite to a relative base ('./') so every generated URL resolves against
the file that references it: CSS fonts and route chunks now load from
`/llm/assets/...`, and index.html's now-relative entry refs resolve via the
`<base href>` serveIndex already injects on every response. Root deployments
are unaffected. The existing path-absolute rewrite in app.go still covers the
public `/favicon.svg`.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The managed-artifact materializer stages a HuggingFace snapshot into a
directory (.artifacts/huggingface/<key>/snapshot/). That is the right load
target for directory-consuming backends (transformers, vLLM, diffusers, ...),
but PrimaryArtifactSpec inferred a managed artifact from ANY HuggingFace-shaped
model reference regardless of backend. A single-file backend such as llama.cpp
or whisper was therefore handed the snapshot directory instead of the weight
file and failed to load it.
The /import-model importer already guards this with a backend allow-list
(managedArtifactBackends), but the loader-side inference did not. Move the
allow-list into core/config as IsManagedArtifactBackend and apply it in
PrimaryArtifactSpec: only directory-consuming backends may have an artifact
inferred from a bare reference; every other backend stays on the legacy
download-to-file path. An explicit artifacts: block still bypasses the gate,
where single-file snapshot resolution handles the load path.
The importer now shares the same predicate, so both paths agree on which
backends auto-materialize.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(model-artifacts): load single-file HF snapshots from the file, not the dir
The managed Hugging Face artifact materializer (#10825) always pointed
backends at the snapshot *directory*
(.artifacts/huggingface/<key>/snapshot). For a single-file model
reference such as huggingface://nomic-ai/nomic-embed-text-v1.5-GGUF/nomic-embed-text-v1.5.f16.gguf,
the GGUF lives *inside* that directory, so llama.cpp was handed a
directory and failed with "gguf_init_from_reader: failed to read magic".
This has kept the tests-aio job red on master since the feature merged
(the embeddings e2e tests could not load text-embedding-ada-002).
Record the single file of a one-file snapshot as Resolved.PrimaryFile and
have ModelFileName() resolve to snapshot/<PrimaryFile> when it is set.
Multi-file snapshots (e.g. transformers repos consumed as a directory)
keep pointing at the snapshot directory. PrimaryFile is derived from the
resolved contents and is deliberately excluded from the artifact cache
key. estimateModelSizeBytes now derives the snapshot directory from the
cache key instead of ModelFileName(), so its manifest lookup is unaffected
by the file-vs-directory resolution.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The artifact progress bridge mapped every PhaseVerifying event to a flat
95%. The materializer emits PhaseVerifying once per file (from each file's
AfterDownload hook) and downloads run sequentially, so the first small file
to finish pinned the bar at 95% - and, because progress is monotonic, it
stayed at 95% for the entire remaining download (e.g. a 70GB checkpoint
reporting 95% at 410MB / 69.7GB).
Track per-file verify proportionally to the running aggregate bytes, the
same way downloading does. CurrentBytes already reflects "completed files +
this file", so the percentage advances honestly. The flat 95%/99% is now
reserved for the genuinely once-per-install Committing/Persisting phases.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The Operate → Traces "API Traces" panel already recorded who made each
request (user_id/user_name) but never showed it, and did not capture the
caller's network identity at all. Operators asked to see the requesting
user (#10886) and the client IP + user agent (#10887) so a trace can be
attributed to who/what issued it.
Backend: add ClientIP and UserAgent to APIExchange and populate them from
echo's c.RealIP() (honours X-Forwarded-For / X-Real-IP behind a trusted
proxy) and the request's User-Agent header. Both are omitempty and the
/api/traces swagger response is map[string]any, so this is additive.
UI: add a sortable "User" column to the API traces table and a metadata
block (User / Client IP / User Agent) at the top of the expanded row
detail. Fields render only when present, so older buffered traces and
unauthenticated/local requests degrade cleanly.
Adds an e2e spec covering the new column value and the expanded metadata.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The Operate -> Traces table rendered the request time with the time of day
only, so entries that span more than one day were ambiguous. Add a
formatDateTime helper (localized date + existing time-with-millis) and use it
for the Traces "Time" column, keeping the cell on a single line. The shared
formatTimestamp used by the log views is unchanged.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
One agent turn runs several internal LLM generations (tool selection,
reasoning, final answer) that all emit stream_event deltas over the same
per-agent SSE channel. The chat page accumulated every 'content' delta
into a single live bubble and ignored the 'done' boundary events, so the
internal generations' text (e.g. the English tool-selection rationale)
merged with — and visually corrupted — the streamed final answer.
Reset the accumulated content/reasoning on 'done': each generation gets
a clean live bubble, and the authoritative full answer still arrives via
the final json_message event as before.
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
StopP2P() read and wrote a.p2pCtx/a.p2pCancel without holding
a.p2pMutex, and StartP2P() reassigned both fields with no lock at
all -- including when RestartP2P() calls it from a background
goroutine after releasing the mutex. Both paths are reachable from
POST /api/settings (empty p2p_token -> StopP2P, non-empty ->
RestartP2P), so concurrent requests race on the same fields.
Take a.p2pMutex in StopP2P and around the field publication in
StartP2P, factor the shared teardown into stopP2PLocked() so
RestartP2P reuses it, and route the goroutine error path through
StopP2P instead of touching a.p2pCancel unlocked.
Signed-off-by: Anai-Guo <antai12232931@anaiguo.com>
Co-authored-by: Anai-Guo <antai12232931@anaiguo.com>
* feat(settings): add declarative runtime-settings field registry
One fieldSpec row per RuntimeSettings field, with a reflection
completeness spec so a field added without a registry row is a red
test instead of a silently-dropped setting (the #10845 bug class).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* refactor(settings): drive ToRuntimeSettings/ApplyRuntimeSettings from the field registry
Behavior-preserving: ~350 hand-written per-field lines become two loops
over runtimeSettingsFields, gated by a To->Apply->To round-trip spec.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* feat(settings): baseline-driven startup merge for persisted runtime settings
ApplyRuntimeSettingsAtStartup compares the live config against
DefaultRuntimeBaseline (option-less-run defaults incl. kong-injected
flag defaults) instead of per-field == 0 guards. Fixes persisted
lru_eviction_max_retries, tracing_max_items, agent_job_retention_days,
memory_reclaimer_threshold, galleries and autoload flags being
silently ignored at boot.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* fix(settings): registry-driven startup merge, applied before consumers
loadRuntimeSettingsFromFile becomes a thin wrapper over
ApplyRuntimeSettingsAtStartup and runs at the top of New(), before
model configs capture app-level defaults. WithThreads stops eagerly
resolving 0 so a persisted thread count survives restart while
LOCALAI_THREADS still wins (#10845); the physical-core fallback moves
after the merge.
Also: run.go now injects the memory-reclaimer threshold unconditionally
so the option-less boot matches DefaultRuntimeBaseline and a UI-saved
threshold survives restart.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* refactor(settings): file watcher delegates to the registry merge; shared API-key merge
Manual edits to runtime_settings.json now behave like a boot-time load
(env still wins) instead of the inverted diverged-from-startup guard
that ignored most manual edits. MergeAPIKeys dedups env keys in one
place for the endpoint and the watcher.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* docs(settings): document unified runtime-settings precedence
Document the single env/CLI > runtime_settings.json > defaults rule,
applied identically at boot, on POST /api/settings, and on manual file
edits, plus the two known limitations (default-valued env vars are
indistinguishable from unset; API-changed fields hot-apply on the next
restart only). Also add a completion debug log when the watcher applies
runtime_settings.json.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* test(settings): reset the global VRAM cap leaked by the round-trip spec
The round-trip spec applies vram_budget=12GiB, whose post-loop hook
installs a process-global default cap; without a reset every spec
ordered after it runs under that phantom budget. Also drop a stale
enumeration in the ApplyRuntimeSettings doc comment.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(audio-transform): serialize WebSocket writes to avoid concurrent-write panic
AudioTransformStreamEndpoint writes to the same Gorilla WebSocket connection
from two goroutines: the backend-forwarding goroutine emits binary PCM frames
(and can call sendWSError on a backend recv error), while the read loop calls
sendWSError for malformed mid-stream JSON or a backend send failure. Gorilla
WebSocket permits only one concurrent writer, so these writers race and can
panic with "concurrent write to websocket connection", resetting the client
session; a -race build reports the data race directly.
Wrap the connection in a lockedConn that serializes WriteMessage behind a
mutex, mirroring the existing lockedConn used by the openresponses WebSocket
endpoint. Reads stay on the single read loop, so only writes need the lock.
Fixes#10844
Signed-off-by: Tai An <antai12232931@outlook.com>
* chore: empty commit to re-trigger checks
Signed-off-by: Anai-Guo <antai12232931@anaiguo.com>
---------
Signed-off-by: Tai An <antai12232931@outlook.com>
Signed-off-by: Anai-Guo <antai12232931@anaiguo.com>
Co-authored-by: Anai-Guo <antai12232931@anaiguo.com>
feat(bonsai): add PrismML llama.cpp fork backend + Bonsai gallery models
Adds a new `bonsai` backend that runs the PrismML fork of llama.cpp
(github.com/PrismML-Eng/llama.cpp, `prism` branch), which ships the Q1_0
(1-bit) and Q2_0 (ternary / 1.58-bit) weight-quantization kernels used by the
Bonsai and Ternary-Bonsai models. Stock llama.cpp cannot decode these quants.
Modeled on the turboquant backend: reuses backend/cpp/llama-cpp/grpc-server.cpp
against the fork's libllama via a thin wrapper Makefile, so the sub-2-bit models
are served with the same OpenAI-compatible API. No grpc-server allow-list patch
is needed (bonsai adds weight quants, transparent to the server, not KV-cache
types), and the reused server compiles cleanly against the fork with no skew
patches (validated locally via a CPU docker build; patches/ is present but empty
for any future re-pin skew).
Backend wiring: backend/cpp/bonsai/, .docker/bonsai-compile.sh,
backend/Dockerfile.bonsai, top-level Makefile targets, backend-matrix.yml build
rows (CPU, CUDA 12/13, L4T, SYCL f32/f16, Vulkan, ROCm/hipblas), backend/index.yaml
meta-backend + per-platform images, and a nightly bump_deps entry tracking the
`prism` branch.
Gallery: 8 entries across 4 families - bonsai-8b-1bit, ternary-bonsai-8b (+g64,
+pq2), bonsai-27b-1bit (vision), ternary-bonsai-27b (+pq2, +g64, vision). The 27B
models wire the mmproj vision tower; the DSpark speculative drafter GGUFs are not
wired (custom semi-autoregressive drafter, not a standard llama.cpp draft model).
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(watchdog): guard StopWatchdog with watchdogMutex to prevent double close
StopWatchdog checked, closed and cleared a.watchdogStop without holding
a.watchdogMutex, while startWatchdog and RestartWatchdog reassign and close the
same channel under that lock.
POST /api/settings dispatches to StopWatchdog or RestartWatchdog depending on
ApplicationConfig.WatchdogShouldRun(), so both are reachable concurrently. Two
callers can observe a non-nil watchdogStop and both close it, which panics with
'close of closed channel' and takes the server down.
Take the mutex, matching the other two writers. StopWatchdog is only called from
the settings handler, which holds no lock, so this cannot deadlock.
Fixes#10841
Co-authored-by: Anai Guo <antai12232931@anaiguo.com>
Finetune() compiled every model cutstrings/extract_regex entry via regexp.Compile
and called xlog.Fatal on failure, which terminates the entire local-ai process.
A single model config with an invalid regex (e.g. cutstrings: ["("]) turns one
/v1/chat/completions request into a process-level denial of service.
Log the compile error and skip the offending pattern instead. The mutex is
released before continuing, and skipping avoids dereferencing the nil regexp
that removing the fatal would otherwise leave behind.
Fixes#10843
Signed-off-by: Tai An <antai12232931@outlook.com>
ApplyRuntimeSettings persists the performance settings (threads,
context_size, f16) on the live /api/settings path, but the startup
loader loadRuntimeSettingsFromFile never read them back, so a value
saved via the Middleware UI was silently ignored on the next restart:
the model booted with the CLI/physical-core default and GET /api/settings
echoed that default instead of the saved value (#10845).
Threads needs special handling: unlike context_size/f16, WithThreads
eagerly resolves an unset (0) value to xsysinfo.CPUPhysicalCores() at
option-apply time, so options.Threads is never 0 in the loader and the
usual "== default" heuristic cannot tell an env/CLI value from the
physical-core fallback. Detect LOCALAI_THREADS/THREADS explicitly so the
env still wins over the persisted file value.
Signed-off-by: Anai-Guo <Anai-Guo@users.noreply.github.com>
Co-authored-by: Anai-Guo <Anai-Guo@users.noreply.github.com>
* feat(vram): add vrambudget primitive for per-node VRAM caps
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): apply default VRAM budget in xsysinfo aggregate getters
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): wire LOCALAI_VRAM_BUDGET flag to xsysinfo default budget
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): persist VRAM budget via runtime settings with live apply
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(vram): reset process-global VRAM budget after runtime-settings spec
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): add VRAM budget field to Settings page
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): store and enforce per-node VRAM budget in the node registry
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): apply per-node VRAM budget in router hardware defaults
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): report worker VRAM budget in node registration
The distributed worker now reports its operator-set VRAM budget string
(LOCALAI_VRAM_BUDGET) to the server on registration. The worker keeps
reporting RAW total/available VRAM and never sets the xsysinfo
process-global budget (that stays standalone-only); the server resolves
and enforces the budget uniformly (Task 6).
Also closes a Task 6 gap: on re-registration, a struct Updates zero-skips
an empty budget, so a worker that dropped LOCALAI_VRAM_BUDGET left the
stale cap in place. For non-admin-override nodes the budget columns are
now force-written (map Updates) even when empty, so removing the env var
clears the cap; admin overrides are preserved unchanged.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* style(vram): drop em dash from worker-clear comment
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): add node VRAM budget admin endpoints
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): add node VRAM budget control to the node UI
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): expose set_node_vram_budget MCP admin tool
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(vram): document LOCALAI_VRAM_BUDGET and node VRAM budget UI
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vram): avoid double-applying VRAM budget in GetResourceAggregateInfo
The GPU-branch aggregate returned by GetResourceInfo is sourced from
GetGPUAggregateInfo, which already caps total/free/used against the
process-wide VRAM budget. GetResourceAggregateInfo then applied the
budget a second time. For an absolute budget this is idempotent, but for
a percentage budget b.Apply resolves the ceiling as a fraction of its
input total, so a second pass yields P*(P*T) instead of P*T and distorts
UsagePercent (read by the memory reclaimer in pkg/model/watchdog.go).
Remove the redundant second application so the budget is applied exactly
once, against the raw physical totals, upstream in GetGPUAggregateInfo.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vram): implement SetNodeVRAMBudget on mcp assistant test stub
The LocalAIClient interface gained SetNodeVRAMBudget; the stubClient in
core/http/endpoints/mcp used by the assistant tests is a separate
implementer and needs the method too (broke golangci-lint typecheck and
both test jobs).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* test(core/http): make the suite's HTTP port overridable
app_test.go and openresponses_test.go hardcoded 127.0.0.1:9090. When
another service already listens on 9090 the suite does not fail fast:
the server goroutine logs the bind error and the specs then poll
whatever is squatting the port until Eventually times out. On machines
where 9090 is permanently taken this makes the pre-commit coverage gate
impossible to pass.
Introduce testHTTPAddr, defaulting to 127.0.0.1:9090 (what CI has
always used) and overridable via LOCALAI_TEST_HTTP_PORT for local runs.
Assisted-by: Claude:claude-fable-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): make per-node backend upgrade actually upgrade
The node detail page's Upgrade button reused the node-scoped install
path (POST /api/nodes/:id/backends/install). That fires NATS
backend.install with force=false, and the worker's install handler is
deliberately "ensure installed": when the backend binary already exists
on disk it short-circuits without touching the gallery. Since only an
installed backend can be upgraded, the whole chain was a guaranteed
successful no-op - the UI then toasted "backend upgraded" without even
waiting for the async job.
Route upgrades through the real force-reinstall path instead:
- BackendManager.UpgradeBackend now receives the ManagementOp (like
InstallBackend already did) so implementations can honor
op.TargetNodeID.
- DistributedBackendManager.UpgradeBackend scopes the backend.upgrade
fan-out to op.TargetNodeID when set, and errors when the target node
does not report the backend as installed.
- New POST /api/nodes/:id/backends/upgrade endpoint enqueues an
Upgrade=true node-scoped op (async 202 + jobID, mirroring install).
- NodeDetail UI calls the new endpoint and reports the dispatch
("Upgrading ... on this node...") instead of claiming success; the
Operations panel tracks the actual job.
Verified against a live local cluster (NATS + Postgres + two workers):
the target worker stops the running process, force-reinstalls from the
gallery and re-downloads the OCI image; the second worker receives no
backend.upgrade event; upgrading a backend missing from the target node
fails the job with a clear error.
Assisted-by: Claude:claude-fable-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
SetDefaults injected the llama.cpp server options cache_reuse
(ApplyServingDefaults) and parallel (ApplyHardwareDefaults, re-applied
per selected node by the distributed router) onto every model config
regardless of backend. Every other backend ignores options it does not
understand, so this was harmless until longcat-video, which strictly
validates its options and fails LoadModel with
"unknown model option(s): cache_reuse, parallel".
Gate both injections behind a new UsesLlamaCppServingOptions allow-list
(llama-cpp plus the empty/auto-detect case that resolves to llama.cpp
from a GGUF file, mirroring how llamaCppDefaults is registered). This
follows the existing UsesLlamaSamplerDefaults precedent for llama-only
defaults. The typed NBatch field is deliberately left alone: it is a
proto field every backend simply ignores, which is why batch never
triggered the error.
Also harden the longcat-video backend to warn-and-ignore unknown model
options and request params through a testable select_known_options
helper, matching the other LocalAI Python backends, so a future
server-injected option cannot break loading again.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
handleRegenerate rebuilt the outbound message from the display-only
message.files metadata ({name, type: 'file'|'image'|..., content}),
which doesn't carry the base64/textContent payload sendMessage's
file-building loop expects. As a result, regenerating any answer whose
own question had an attachment silently dropped that attachment from
the resent message. This wasn't fork-specific, but forking a chat and
then regenerating an earlier (now non-last) answer is the natural way
to hit it.
Fix by reusing the original message's already-assembled `content`
verbatim (it already has the file text / image_url / audio_url /
video_url parts embedded from the first send) instead of trying to
reconstruct it from lossy display metadata.
Fixes#10806
Assisted-by: Claude:claude-sonnet-5
Signed-off-by: ajuijas <189517297+ajuijas@users.noreply.github.com>
Co-authored-by: ajuijas <189517297+ajuijas@users.noreply.github.com>
- Replace hardcoded text with useTranslation hook in UI components
- Add localization support for both English (en) and Indonesian (id) locales
Signed-off-by: Dedy F. Setyawan <dedyfajars@gmail.com>
* feat(ui): add voice library workflow
Give administrators a production-ready flow to record or upload consented reference audio, manage reusable profiles, inspect API usage, discover compatible models, and hand a saved voice directly to text-to-speech.
Assisted-by: Codex:gpt-5
* feat(voice): add managed voice cloning profiles
Make reusable reference voices manageable through the admin API instead of requiring model-directory and YAML edits. Discover compatible installed and gallery models from server-side backend capabilities, retain explicit model configuration controls, and stage saved references for supported backends.
Expose profile management through REST and MCP, document backend-specific behavior, and cover the workflow from profile creation through real Qwen3-TTS synthesis. Harden the agent-job HTTP test against completion racing cancellation.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(backends): add LongCat video and avatar generation
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] [web]
* refactor(config): declare model I/O modalities
Make model configs declare input and output modalities so capability discovery no longer branches on backend or checkpoint names. Complete the LongCat gallery and user documentation, make the SDPA patch apply to the pinned upstream revision, and stabilize the Agent Jobs race exposed by the required hook.
Assisted-by: Codex:GPT-5 [web]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
When a model config has no explicit backend, the model loader greedily
probes every installed backend and binds to the first Load that
succeeds. opus and local-store were the only in-tree backends with no
model artefact to validate, so they accepted anything — an LLM
installed after them could silently bind to the audio codec or the
vector store and then fail at inference with "unimplemented"
(see #9287).
opus now accepts only its own name (what the realtime WebRTC path
sends) or none. local-store namespaces are arbitrary (router caches,
biometrics, user-named stores), so core's StoreBackend now marks
genuine store loads with a store:// prefix on the gRPC model name and
the backend refuses names without it; core and backend ship from the
same release, so the convention upgrades in lockstep.
Also repair the bit-rotted 'make test-stores' bootstrap (the suite
never registered external backends, so BACKENDS_PATH was dead weight)
and add the Load-validation rule to the adding-backends checklist.
Related: #9287
Assisted-by: Claude:claude-fable-5 golangci-lint
Signed-off-by: Richard Palethorpe <io@richiejp.com>
C++/ggml transcription + speaker diarization + timestamps backend. Purego
dlopens libmoss-transcribe.so (ggml statically linked) from moss-transcribe.cpp
and serves offline AudioTranscription, parsing the [start][Sxx]text[end] output
into segments with nanosecond timestamps. Adds the importer (surfaces in
GET /backends/known), backend-matrix (Linux + Darwin/metal), backend/index.yaml,
and a gallery entry (default q5_k GGUF from mudler/moss-transcribe.cpp-gguf).
Local L0 smoke (go build + go test ./... = 16 pass, golangci-lint 0 issues)
passed against the real libmoss-transcribe.so. The pre-commit coverage gate
(full pkg/core + tests/e2e) could not run in the authoring sandbox (no live
models, port 9090 held); CI must enforce it before merge.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(schema): accept reasoning_content as inbound alias for reasoning
Interleaved-thinking clients (cogito, vLLM/DeepSeek-style) emit reasoning_content
on assistant turns. Accept it as an inbound alias so reasoning survives the
tool-result loop; canonical reasoning wins when both are present. Emission is
unchanged (still reasoning).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(schema): pin interleaved reasoning+tool_calls round-trip
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(openai): pin reachedTokenBudget truncation detection
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(anthropic): add thinking and signature fields to content blocks
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(anthropic): parse inbound thinking blocks into reasoning
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(anthropic): emit thinking blocks with synthetic signature on tool turns
Extract buildAnthropicContentBlocks so non-streaming content assembly is
unit-testable, and prepend a thinking block (with an opaque synthetic
signature) before text/tool_use blocks when the request opts into thinking.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(anthropic): stream thinking_delta and signature_delta before tool_use
Extract anthropicStreamSequence so the streaming block order is unit-testable,
and emit content_block_start(thinking) -> thinking_delta -> signature_delta ->
content_block_stop before the tool_use block sequence when thinking is enabled.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: add interleaved thinking with tool calls guide
Add a features guide describing interleaved thinking: an assistant turn
carrying reasoning and tool_calls together, the reasoning-round-trip
contract (including the reasoning_content inbound alias and Anthropic
thinking blocks with a synthetic signature), per-backend enablement
(reasoning_format for llama.cpp, reasoning_parser/tool_call_parser for
vLLM/SGLang plus the vLLM auto-config hook), a worked request/response
example, and known limitations. Cross-link from model-configuration,
text-generation, and openai-functions.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The OIDC verifier was built with a bare oidc.Config{ClientID: ...}, so
go-oidc applied its default of accepting RS256-signed ID tokens only. An
identity provider configured with an EC signing key (e.g. Authentik) issues
ES256-signed tokens, and the callback failed verification with:
failed to verify ID token: oidc: malformed jwt: unexpected signature
algorithm "HS256"; expected ["RS256"]
surfacing to the user as HTTP 500 "failed to fetch user info" (#10677; the
underlying cause became visible after the logging fix in #10679).
Set SupportedSigningAlgs to the standard asymmetric algorithms
(RS256/384/512, ES256/384/512, PS256/384/512, EdDSA). All are verified
against the provider's published JWKS. HS256 is intentionally excluded: it
is symmetric and would validate against the client secret, a different and
security-sensitive trust model.
Tested with a functional spec that signs an ES256 ID token and confirms it
verifies with the configured algorithms and is rejected under go-oidc's
RS256-only default (using oidc.StaticKeySet, no network).
Closes#10677
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Backend log capture into the per-model BackendLogStore (which feeds the
UI "Backend Logs" page and /api/backend-logs) was opt-in and off by
default in single mode, while worker/distributed mode force-enables it
via SetBackendLoggingEnabled(true). There was no CLI flag either, so the
only way to populate the store was the Settings UI toggle - and the page
was silently empty out of the box. Distributed "just worked"; single
mode looked broken.
Default EnableBackendLogging to true in NewApplicationConfig so single
mode matches worker mode. The store is a small in-memory ring buffer, so
the cost is negligible.
Now that the default is on, loadRuntimeSettingsFromFile's usual
"only flip false->true" merge would ignore a persisted false and revert
the UI toggle-off on every restart. There is no env var/CLI flag for
this setting, so an explicit persisted value is now authoritative in
both directions, letting the toggle-off survive a restart.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(transcription): honor model-config language/translate, add language form field
The /v1/audio/transcriptions endpoint read only input.Language /
input.Translate from the parsed request, and the request middleware never
populates those from a multipart upload -- nor did it read a `language`
form field. As a result the model config's parameters.language /
parameters.translate (a valid PredictionOptions field under `parameters:`)
were silently ignored, and multilingual models like canary defaulted to
translating into English even when the YAML set language: ru,
translate: false (#10655).
Resolve both with clear precedence: the request form field wins, then any
language on the parsed request, then the model config default. This also
makes the endpoint honor OpenAI's `language` form parameter, which was
not read before.
Applies to both the streaming and non-streaming paths (the resolved
values are built into the shared TranscriptionRequest). Note this ensures
the language/translate flags reach the backend; whether a given engine
acts on them is up to the backend.
Closes#10655
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
A client that keeps polling a model whose load fails (e.g. a backend that
crashes deterministically on init) triggered a fresh backend start on
every request: request -> load -> crash in ~10s -> 500, repeat on the
next poll. Each attempt could leak GPU/CUDA state, and under
LOCALAI_SINGLE_ACTIVE_BACKEND it kept stealing the active slot from
healthy models. The existing loading-coalesce map only dedups
*concurrent* loads, so sequential polls were never covered.
Track load failures per modelID in ModelLoader. After a load fails,
refuse fresh load triggers for that model until a cooldown elapses,
returning a typed ModelLoadCooldownError that the HTTP layer maps to 503
with a Retry-After header. The cooldown grows exponentially per
consecutive failure (base, doubling, capped at 5m) and resets on a
successful load. The coalesced follower-retry of an in-flight burst
bypasses the gate, so a genuinely concurrent burst still gets its one
retry -- only new, independent triggers are refused, matching the
report's "refuse new load-triggers" wording.
Configurable via --model-load-failure-cooldown /
LOCALAI_MODEL_LOAD_FAILURE_COOLDOWN (default 10s, 0 disables), plumbed
through ApplicationConfig and applied unconditionally at startup.
Closes#10719
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
InstallBackend extracted the artifact directly into the target directory
with no pre-clean, so a reinstall overlaid the new files onto the old
ones. Files present in a previous version but absent in the new artifact
(a stale .so, an orphaned package dir) survived and could shadow the new
build at import time -- e.g. an old vllm shared object lingering next to
a freshly pulled one. Only a failed download cleaned the directory.
Stage the download/extraction into a `<name>.install-tmp` dir, validate
run.sh is present, write metadata, then atomically swap it into place
(rename current -> .install-backup, staging -> current, drop backup),
rolling back on failure. This mirrors the atomic swap UpgradeBackend
already performs, so install and upgrade now leave identical on-disk
state with no orphaned files.
Reported as part of #10720.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(llama-cpp): cap single-pass embedding batch to fit VRAM
Embedding/score/rerank all decode or pool the whole input in one physical
batch, so EffectiveBatchSize sized the batch to the full context window. For
a large context that makes n_ubatch huge, and the per-device CUDA compute
buffer (forward-graph scratch, ~n_ubatch * n_ctx, NOT split across GPUs)
balloons into multi-GiB: a large-context embedding model then aborts on load
(exitCode=-1) even with plenty of free VRAM. Reproduced with qwen3-embedding-4b
(context 40960 -> n_batch 40960 -> abort) and qwen3-embedding-0.6b
(n_batch 8192); pinning batch:512 avoided it.
This is the same root cause as issue #10485 (a large context turns the batch
into multi-GiB of scratch that must fit on a SINGLE card), but the single-pass
path bypassed the VRAM headroom guard the config layer already had — it
returned the unbounded context as the batch with no GPU awareness.
Make the single-pass batch VRAM-aware: cap it to the largest batch whose
compute buffer fits the per-device VRAM headroom, clamped to
[DefaultPhysicalBatch, ctx], reusing the existing computeBufferBytesPerCell and
headroom-divisor math (no duplication). Unknown per-device VRAM (0) stays
conservative (DefaultPhysicalBatch, not the context) so a detection gap can't
OOM. The GPU is resolved through an injectable package var (config.LocalGPU,
backed by sync.Once-cached xsysinfo detection) so the per-request router call
stays cheap and tests inject a deterministic device. Explicit batch: still
wins. An input longer than the cap can no longer be pooled in one pass — the
accepted tradeoff, since a batch that OOMs the device processes nothing.
Assisted-by: Claude:claude-opus-4-8 golangci-lint go-test
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(config): single-pass batch follows context on unknown VRAM
The single-pass (embedding/score/rerank) batch cap must only shrink the batch
when the per-device VRAM ceiling is KNOWN. On unknown VRAM (CPU-only or a GPU
detection gap) SinglePassBatchForContext returned DefaultPhysicalBatch, which
under-sized the batch below the context — over-trimming score/embed/rerank
inputs (the modelTokenTrim middleware regression) with no OOM benefit on CPU
where the compute buffer lives in system RAM. Return the full context instead,
preserving the original single-pass behavior; the VRAM cap stays a downward
safety that only engages when VRAM is known.
Assisted-by: Claude:claude-opus-4-8 [go-test go-vet]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
When a model is imported without an explicit context_size, the GGUF
importer defaulted the model's context to its full trained window
(n_ctx_train). For long-context models (128k / 256k / 1M) that KV cache
cannot fit a consumer GPU, so the backend aborts on load (exitCode=-1)
even though the model file is perfectly fine. Reproduced live:
gemma-4-26b-a4b-it-qat-q4_0 defaulted to context=262144 and
qwythos-9b-claude-mythos-5-1m to 1048576, both aborting on a 20 GB card.
Instead of chasing the trained max, auto-derive a conservative default:
min(trainedMax, DefaultAutoContextSize=8192). A small model keeps its
trained window; a long-context model caps at 8k and users opt into more
via context_size. This cap applies always, including CPU / unknown-VRAM
hosts, so it never regresses those paths.
Per-device VRAM is used only as a DOWNWARD safety: when a per-device
ceiling is detected (xsysinfo.MinPerGPUVRAM) and even the 8k cap would
not fit it with headroom, step down through candidate contexts to the
largest that fits, floored at DefaultContextSize. When VRAM is unknown
(0) or no GPU is detected we do NOT clamp — the bug is GPU OOM and the
8k cap is already safe, so detection gaps must not shrink the window.
The footprint estimate reuses gpustack/gguf-parser-go's
EstimateLLaMACppRun at a given context with all layers offloaded, taking
the per-device NonUMA VRAM figure. The estimate and VRAM detection are
package vars so tests inject deterministic values. Explicit context_size
always wins (guessGGUFFromFile only acts when it is nil).
Assisted-by: Claude:claude-opus-4-8 [golangci-lint go-test]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The `--generated-content-path` and `--upload-path` defaults were the fixed
shared locations `/tmp/generated/content` and `/tmp/localai/upload`. On any
multi-user host these collide across accounts: macOS routes `/tmp` to the
shared `/private/tmp` for every user, so whichever account starts LocalAI
first creates the parent with 0750 perms and every other account then fails
startup with:
unable to create ImageDir: "mkdir /tmp/generated/content: permission denied"
unable to create UploadDir: "mkdir /tmp/localai/upload: permission denied"
The same happens on Linux once a stale root-owned `/tmp/generated` (e.g. from
a prior `sudo` run) is left behind. This bites the desktop launcher and any
app embedding the raw binary (Wingman, nib-desktop), which start `local-ai
run` with no path flags.
Default both paths under the OS temp dir (`os.TempDir()`, honoring `$TMPDIR`;
already per-user on macOS) namespaced by the current UID
(`TMPDIR/localai-<uid>/...`), so accounts never collide while the paths stay
ephemeral. Wired via new kong vars in main.go so every consumer of the raw
binary inherits the fix. All content subdirs (audio, images) derive from
`GeneratedContentDir`, so they are fixed transitively.
As defense in depth, the launcher also anchors these two paths under its own
per-user data directory (mirroring the #10610 fix for data/config), extracted
into a testable `BuildRunArgs`.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(reasoning): don't persist request-scoped reasoning_effort into model config
When a model sets `reasoning_effort: none` (or any default) in its YAML
without an explicit `reasoning.disable`, ApplyReasoningEffort resolves that
default at request time and sets ReasoningConfig.DisableReasoning on the
request-scoped config copy. The post-load thinking/marker probe then wrote
that request-scoped value back into the loader's persistent config via
UpdateModelConfig, making it look as though the operator had explicitly set
reasoning.disable=true. From then on, per-request `reasoning_effort` overrides
were silently ignored (an explicit operator disable wins over a request
asking to think).
DetectThinkingSupportFromBackend only fills reasoning slots that are still
nil, so a slot already set here came from ApplyReasoningEffort, not the probe.
Snapshot which slots were nil before the probe and only persist those, so the
probe's genuine backend detection is still saved while request-time reasoning
effort never leaks into the persistent config.
Fixes#10622
Signed-off-by: Tai An <antai12232931@outlook.com>
* test(reasoning): cover persist-guard added in this PR, extract for testability
ModelInference's post-probe persistence of ReasoningConfig.DisableReasoning /
DisableReasoningTagPrefill had no test: the guard logic lived inline in a
closure only reachable through a live gRPC backend. Extract it into
persistProbedReasoning (pure refactor, no behavior change) so it can be
exercised directly against a ModelConfigLoader, then add specs covering:
- a probe-filled slot (nil beforehand) gets persisted
- a slot that already carried a request-scoped value (e.g. from
reasoning_effort: none) is left alone, i.e. the #10622 regression stays
fixed
- an operator's explicit persisted disable is preserved when the guard is
false
- the media marker still persists unconditionally
Verified red/green: reverting persistProbedReasoning to the old unconditional
copy fails exactly the two guard specs.
Assisted-by: Claude:claude-sonnet-5 go vet
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(reasoning): ignore os.Remove error in temp file cleanup (errcheck)
Signed-off-by: Tai An <antai12232931@outlook.com>
* chore: empty commit to re-trigger flaky Agent Jobs CI test
Signed-off-by: Tai An <antai12232931@outlook.com>
---------
Signed-off-by: Tai An <antai12232931@outlook.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
Operators need a scrape-friendly signal for agent-turn health (completing,
erroring, cancelled, duration) — log-derived counters proved brittle (ANSI/
timezone parsing, restart gaps). Adds localai_agent_runs_total{agent,outcome}
and localai_agent_run_seconds histogram, recorded at the Chat() response
handoff (single choke point of the local execution path). Lazy meter init,
same pattern as the PII events counter (#10641).
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
The no-models getting-started wizard (`.home-wizard`) rendered
left-aligned instead of centered. `.home-page` is a column flexbox with
the default `align-items: stretch`; a child with `max-width: 48rem`
cannot be stretched past its max-width, so it falls back to the
cross-start (left) edge. The populated home branch never exposed this
because its children are full-width.
Add `margin: 0 auto` to `.home-wizard` so the max-width block centers
horizontally, for both the admin getting-started wizard and the
non-admin no-models hero.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Additive superset of /v1/models that enriches each model entry with the
capabilities it supports plus its input/output modalities
(text / image / audio / video). Clients that only understand /v1/models
are unaffected -- they simply never call the new route.
Audio and video *input* are derived from the model's multimodal limits
(vLLM limit_mm_per_prompt), which no single usecase FLAG expresses. That
gap is exactly why a plain capability list is insufficient and this
enriched endpoint exists: an attachment router can now decide whether an
image/audio/video file can go to the active model directly, or must be
converted/transcribed first.
Capability derivation lives in core/config as the single source of truth
(ModelConfig.Capabilities / InputModalities / OutputModalities /
VisionSupported / ...); the Ollama capability surface now delegates to
it instead of keeping a parallel copy. Vision is gated on
chat/completion capability so a MediaMarker hydrated onto a non-chat
model (e.g. a pure ASR/TTS backend) no longer reports a false vision
capability.
Read-only listing: no new FLAG_* flag, reuses the existing `models`
swagger tag, and intentionally exposes no MCP admin tool (there is
nothing to manage conversationally).
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The OAuth callback discarded the error returned by user-info resolution
before sending the generic 500, so real failures were completely opaque
in the logs: ID-token verification errors (e.g. issuer/audience mismatch
behind a reverse proxy), a missing id_token, claim-parse errors, or a
rejecting GitHub userinfo endpoint all collapsed into
"failed to fetch user info" with nothing logged.
Log the wrapped cause with xlog.Error (provider + error), matching the
code-exchange step just above it. The client-facing message is unchanged,
so no internal detail leaks to the browser.
Refs #10677
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
POST /models/apply with an empty "id" fetches the attacker-supplied
"url" gallery config directly via http.Client, with no check that the
URL resolves to a public IP. In the default Docker deployment no API key
is configured, so any network-reachable client can coerce LocalAI into
issuing requests to internal services or cloud-metadata endpoints (and
exfiltrate a small slice of the response through the job error message).
Guard the config fetch chokepoints (GetGalleryConfigFromURL and
GetGalleryConfigFromURLWithContext, which back both the /models/apply
worker and gallery installs) with utils.ValidateExternalURL, matching
the protection already applied to the CORS proxy and image/video/audio
download paths. Only plain http(s) URLs are validated; non-network
schemes (huggingface://, github:, oci://, ollama://, file://) resolve to
fixed public services or local files and are left untouched.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The PII EventStore ring buffer is capacity-bound and meant for
recent-audit browsing via /api/pii/events; operators also want a
monotonic, scrape-friendly signal on /metrics — how many
detections/masks/blocks per hour, per origin, and whether the filter
stopped firing after a deploy (silent-failure class).
EventStore.Record is the single choke point every producer already goes
through (request middleware, response scrubbing, MITM proxy
connects/intercepts), so one lazily-initialised counter there covers all
paths without touching any producer:
localai_pii_events_total{kind, origin, action, direction}
Same lazy otel.Meter pattern as core/services/routing/billing, so the
counter lands on the Prometheus-backed global MeterProvider installed by
the monitoring service. No behaviour change; label cardinality is
bounded (enum-like fields only, no pattern IDs or user IDs).
Assisted-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
Realtime sessions previously lazy-loaded each pipeline sub-model (VAD,
transcription, LLM, TTS) on first use, so every cold session paid a
per-request model-load stall and load errors only surfaced mid-stream.
Warm the whole pipeline eagerly and blockingly at session start
(including the voice-gate speaker-recognition model, which an enforced
gate blocks each utterance on; compaction's summary_model stays lazy
since it only runs off the response path):
- Add backend.PreloadModel / PreloadModelByName as the single load path
for every modality (no transcription special-case; backend-omitted
configs are deprecated).
- The realtime session blocks on Model.Warmup and returns a
model_load_error to the client if any stage fails to load;
updateSession warms in the background. Opt out per pipeline with
pipeline.disable_warmup, exposed as a UI toggle via the
config-metadata registry.
Add a LocalAI-native POST /backend/load (and /v1/backend/load) that
pre-loads a model -- expanding realtime pipelines into their sub-models
-- as the inverse of /backend/shutdown. There is one preload engine
(backend.PreloadStages): the realtime Warmup methods, /backend/load and
the --load-to-memory startup flag all use it, so --load-to-memory now
also expands pipeline models and records load-failure traces. Pipeline
sub-model alias resolution is likewise shared
(ModelConfigLoader.LoadResolvedModelConfig). Surface the endpoint
everywhere an admin manages models:
- MCP admin tool load_model (httpapi + inproc clients, safety/catalog
prompts, catalog/dispatch tests).
- "Load into memory" action in the React models UI.
- Swagger regenerated; docs moved to the general backend-monitor page
since it is not realtime-specific.
Fix a Traces UI crash ("json: unsupported value: -Inf"): audio-snippet
RMS/peak now floor at a finite dBFS, and backend-trace data is sanitized
to drop non-finite floats before marshaling. The sanitizer is
copy-on-write -- it runs on every RecordBackendTrace, so containers are
only re-allocated on the paths that actually changed.
Migrate core/http/openresponses_test.go onto the prebuilt mock-backend
the rest of the http suite already uses -- it was the last spec still
pointing at a real HuggingFace model, so it 404'd wherever no vision
backend was built -- and fix its item_reference specs to send the
spec's "id" field instead of "item_id", which the handler never
accepted.
Assisted-by: Claude:claude-opus-4-8 Claude Code
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(ui): clone a chat into a new conversation (#10645)
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): retry any assistant answer, not just the last (#10645)
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): copy an entire chat to the clipboard (#10645)
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): branch a new chat from any assistant answer (#10645)
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ui): send truncated history on mid-conversation retry (#10645)
Mid-conversation retry regenerated an answer with the downstream turns
still in the model's context. handleRegenerate truncated the DOM history
via updateChatSettings (a scheduled state update), but the synchronous
sendMessage that followed read the stale, pre-truncation history from its
closure to build the outbound API payload. Thread the intended base
history explicitly through sendMessage's options.baseHistory so the
request body matches the truncated view. Backward compatible: the normal
send path (no baseHistory) is unchanged.
Also guard two minor issues in Chat.jsx: the "Branch from here" button now
renders under !isStreaming to match the retry button, and the duplicate
toast only fires when forkChat returns a chat (not on a null result).
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(backends): make backend install ops idempotent unless forced
POST /backends/apply hardcoded force=true through
LocalBackendManager.InstallBackend, so applying an already-installed
backend re-downloaded and re-extracted the whole artifact every time.
API clients that ensure a backend exists at startup paid a full OCI
image pull on every boot.
Backend install ops now default to non-forced — an installed, runnable
backend short-circuits (the orphaned-meta reinstall path in
InstallBackendFromGallery is preserved) — and reinstall stays available:
- ManagementOp gains a Force field; the local manager passes it through
instead of hardcoding true.
- /backends/apply accepts an optional "force" boolean in the body.
- The React UI install route keeps forcing, since its button doubles as
the explicit "Reinstall backend" action.
Distributed installs already behaved this way (workers skip when the
binary exists unless force is set); this aligns single-node behavior.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(backends): don't force-reinstall LOCALAI_EXTERNAL_BACKENDS on boot
The startup loop for LOCALAI_EXTERNAL_BACKENDS runs
InstallExternalBackend for each listed backend on every boot, and its
gallery-name path hardcoded force=true — so every start re-downloaded
and re-extracted each listed backend's OCI image even when it was
installed and runnable. Supervising apps that list several backends
paid several full OCI pulls per launch.
Give InstallExternalBackend an explicit force parameter (it only
affects the gallery-name fallback; URI installs always write) and pass:
- false from the boot loop and `local-ai backends install` (idempotent
ensure — `backends upgrade` is the refresh path),
- op.Force from the local manager's external-URI op,
- the request's force on the worker install path and true on its
upgrade path (behavior unchanged).
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): don't let a dead worker pin the model-load advisory lock
In distributed mode a chat request could fail with:
failed to route model with internal loader: routing model ...:
loading model ...: advisorylock: acquiring lock <id>:
ERROR: canceling statement due to lock timeout (SQLSTATE 55P03)
Root cause is two independent defects in the cross-replica model-load path:
1. SmartRouter.Route holds a per-model PostgreSQL advisory lock for the whole
cold-load sequence, which includes installBackendOnNode -> InstallBackend,
a NATS request-reply with a 15m deadline (DefaultBackendInstallTimeout) that
ignored ctx. When the chosen worker died mid-install, the holder sat on the
lock for up to 15m. The detached loadCtx (WithoutCancel) had no deadline, so
nothing capped the hold.
2. The acquiring statement, pg_advisory_lock(), is subject to any deployment
global lock_timeout. A common operator setting (e.g. 10s) aborts the wait
with SQLSTATE 55P03, so every other replica's request for that model hard
-errored instead of waiting for the in-progress load and reusing it. For the
~15m window the model was effectively unroutable.
Fixes:
- advisorylock.WithLockCtx (postgres): SET lock_timeout = 0 on its dedicated
connection (RESET before it returns to the pool) so the Go context, not a
deployment-wide GUC, governs how long we wait. Waiters now block and then
re-check, reusing the model another replica just loaded.
- SmartRouter: bound the detached loadCtx with a single ModelLoadCeiling so the
lock is always released in bounded time even if a sub-step wedges. Default is
the configured backend.install deadline + 10m (staging + LoadModel margin),
so a legitimately slow load is never cut.
- installBackendOnNode: use singleflight.DoChan + select on ctx.Done() so the
install wait honors cancellation; the ceiling can then actually free a caller
pinned behind a dead worker. The shared install still coalesces via
singleflight.
Reproduced both defects as failing tests first (a real 55P03 against a
testcontainer with a short lock_timeout; a wedged install that blocks Route)
and confirmed green.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* fix(distributed): bound advisory-lock wait instead of disabling lock_timeout
Setting lock_timeout = 0 to override a deployment's short global lock_timeout
meant "wait forever" server-side. Safe for SmartRouter.Route (its loadCtx now
carries the model-load ceiling) but unsafe for the schema-migration callers
that pass context.Background(): a holder whose session never releases would
hang them indefinitely.
Derive the server-side lock_timeout from the caller's context instead: its
remaining budget plus a margin (so the Go context's cancellation still wins
with a clean error and the server bound is only a backstop), or a finite
30m backstop when the context has no deadline. Never zero - "wait forever"
is no longer possible, while a deployment's hostile short lock_timeout is
still overridden so legitimate cross-replica waits don't fail with 55P03.
Added a spec proving a deadline-less waiter gives up at the (shrunk) backstop
rather than hanging.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
