backend.stop was the one lifecycle subject a worker never answered. The
controller published and returned nil as soon as the local publish
succeeded, so a stop that killed nothing, and a stop that failed
outright, were indistinguishable from one that worked.
The unload endpoint calls model.unload and then StopBackend. Only the
first is acknowledged, so the endpoint answered 200 while the backend
kept running and held its VRAM, and its own "backend stop failed" branch
could never run. The worker logged the failure and nobody saw it.
Give the subject a reply. The worker now enumerates the process keys it
terminated and reports any per-process error, so StopBackend fails when
the stop failed. Resolving to nothing stays a success: stopping a backend
that is not running leaves the caller in the state it asked for, and
eviction paths stop already-gone models routinely. The empty list is what
says nothing matched, and ReportsStoppedProcesses is what makes that
emptiness trustworthy, the same way BackendDeleteReply handles it.
A worker built before this reply still receives the request and still
stops the backend, it only stays silent, so a timeout degrades to the old
assumption rather than failing every stop on a fleet mid-upgrade. Only
silence degrades: a transport error is still reported, because
UnloadRemoteModel skips its registry cleanup for a node it could not
reach and needs to keep hearing about that.
Assisted-by: Claude:claude-opus-5 golangci-lint
Adds FunASR/SenseVoice as a Python backend for speech-to-text with
support for CPU, CUDA 12/13, ROCm, Intel SYCL, L4T, and Apple MPS.
Co-authored-by: xingyifeng <xingyifeng@users.noreply.github.com>
Saved profiles previously resolved to one audio path and transcript, so
cloning backends could not use several examples of one personality.
Store ordered audio and transcript pairs while preserving the legacy
first-reference fields. Fish Speech and audio.cpp receive all pairs,
including on distributed workers. Other backends retain their
single-reference behavior.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Let studio users guide speech delivery for backends that support request instructions. Blank guidance stays out of requests and media history.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Clients cannot discover the named voices that an installed TTS model accepts without consulting backend-specific documentation. Expose voice metadata through the audio API and let custom model configs declare their own catalog.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The pinned audio.cpp revision supports HIP, but LocalAI neither builds a ROCm image nor accepts its backend option. AMD hosts therefore fall back to the CPU image.
Build and publish the HIP variant, connect it to AMD capability selection, and accept both upstream HIP names.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Expose diffusers audio pipelines through the existing sound-generation RPC. AudioLDM2 can now return PCM WAV output from the model gallery without a separate backend.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a NATS event contract and exact-residency provider for backend KV cache reports. Keep guessed request observations as the default routing source while maintaining the reported index for future producers.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a Darwin-only MLX-Video backend for LTX-2 and converted Wan checkpoints, expose it through the existing video API, and wire packaging, discovery, tests, docs, and an example.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Document how to connect LocalAI to the upstream RKLLM OpenAI-compatible server through cloud-proxy, including supported SoCs and current limitations.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Prefer an explicitly configured Triton assembler, otherwise use the executable ptxas from CUDA_HOME so torch.compile can target GPU architectures newer than Triton bundled tooling.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Broadcast forced-disturb events and successful scale-up resets so every frontend shares the same rolling autoscale signal. Deduplicate NATS echoes, expose an origin-only Prometheus counter, and document cluster behavior.
Closes#10083
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Route prefix-cache replica selection through composable filters, weighted scorers, and a replaceable picker. Preserve the existing load guard and deterministic selection policy while exposing per-model scorer weights through scheduling configuration.
Assisted-by: Codex:gpt-5 [go-vet]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(auth): bypass API-key auth for CORS preflight (OPTIONS) requests
When API-key auth is enabled, a browser making a cross-origin API call
first sends an OPTIONS CORS preflight, which cannot carry credentials by
HTTP spec. The auth middleware is registered (app.go:324) before the CORS
middleware (app.go:337-347), so the preflight hit auth first and returned
401 before the CORS middleware could answer it, blocking the actual call.
Bypass auth for OPTIONS so the request reaches the CORS middleware, which
answers the preflight with 200 + headers. Real API requests (GET/POST/etc.)
still require auth. Regression test added (red on master, green on branch).
Refs #4576
Signed-off-by: supermario_leo <leo.stack@outlook.com>
* fix(auth): exempt CORS preflights via publicRouteRegistry instead of middleware bypass
Route the global OPTIONS exemption through publicRouteRegistry
(OPTIONS on every path, replacing the OPTIONS-under-/api/auth/ rule
it subsumes) instead of a hardcoded method check inside Middleware,
so "which requests skip auth" has one mechanism. Preflights now flow
through the same authenticate-then-public-rules path as other public
routes, which also lets a credentialed OPTIONS request keep its user
context.
Update the route-coverage allowlist and the near-prefix lookalike
table for the new semantics (OPTIONS is public on every path by
design; near-prefix privacy stays pinned by the non-OPTIONS entries),
and fix the authentication docs' exempt-route enumeration, which
still described OPTIONS as an /api/auth/-only exemption.
Signed-off-by: supermario_leo <leo.stack@outlook.com>
---------
Signed-off-by: supermario_leo <leo.stack@outlook.com>
Add the published UD-Q4_K_XL shards and BF16 vision projector.
Document installation and the default context size.
Assisted-by: Codex:gpt-6
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add Q4_K_M and Q5_K_M builds with the vision projector and an install
example. Pin downloads to the verified Hugging Face revision.
Assisted-by: Codex:gpt-6
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* docs(faq): note OPENAI_BASE_URL env var for pointing clients at LocalAI
The "Can I use it with a Discord bot, or XXX?" answer says a client can be
pointed at the LocalAI endpoint, but not how. Most OpenAI clients pick the
endpoint up from OPENAI_BASE_URL, and clients that insist on a non-empty API
key accept any placeholder value while api_keys is unset.
Assisted-by: Claude:claude-opus-5
Signed-off-by: seven7763 <seven7763@users.noreply.github.com>
* docs(faq): use Hugo notice shortcode per coding-style
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: seven7763 <seven7763@users.noreply.github.com>
---------
Signed-off-by: seven7763 <seven7763@users.noreply.github.com>
Co-authored-by: seven7763 <seven7763@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
When no sampled candidate fits GPU memory, ranking falls back to the
oversized pool and labels its first model Best fit. Keep GPU picks within
the existing 95% budget and hide the section when no candidate qualifies.
Remove static GPU starter picks so Home cannot reintroduce the same error.
Add browser regressions for both sections and document the empty result.
CPU fallback behavior stays unchanged.
Assisted-by: Codex:gpt-6 [Codex]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(chat): template.system_messages_after_first — merge or forward late system turns
Tokenizer chat templates such as Qwen3.8 / Qwen3.8-Flash-Next raise
'System message must be at the beginning' for system-role messages that
appear after the leading system block, while agent frameworks (cogito tool
selection and adjustment prompts) legitimately append system instructions
mid-conversation. Every such request failed with a 500 (48 errors in one
10-task agent run).
New per-model option template.system_messages_after_first:
merge fold late system turns into the leading system message
user forward them as user-role turns at their original position
Default (unset) keeps the current pass-through behaviour.
Fixes#11876
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
* docs(model-config): document template.system_messages_after_first
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
* fix(config/meta): register template.system_messages_after_first in the field registry
TestAllFieldsHaveRegistryEntries requires every model-config field to have
a registry entry. Adds the entry (templates section, select component) and
the option list for the new field so the coverage gate passes.
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
---------
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
* fix(realtime): support session voice profile switching
Keep the active resolved voice binding on the realtime session so updates can atomically replace model, voice, and profile parameters while releasing leases at the correct lifecycle boundaries.
Assisted-by: Codex:gpt-5
* docs(realtime): explain voice profile switching
Document the session.update payload for selecting a Voice Library URI and clarify precedence when changing the model in the same event.\n\nAssisted-by: Codex:gpt-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Backend processes shared the host temporary directory, so crashes could leave request images and audio behind until the filesystem filled. Give each process a locked LocalAI-owned runtime, remove scratch on exit, and sweep only marked abandoned runtimes at the next start.
Also close known request error-path leaks in the Python media backends, CrispASR, LongCat Video, and stable-diffusion.cpp.
Assisted-by: Codex:gpt-5
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(realtime): resolve saved voice profiles
Realtime pipelines now validate saved voices against the selected TTS model and retain leased audio until session teardown. Each synthesis request receives its own transcript parameter map.
Assisted-by: Codex:GPT-5
* docs(tts): document realtime voice defaults
Show how a realtime pipeline selects a saved Voice Library profile at session start. Clarify which session voice updates remain supported.
Assisted-by: Codex:GPT-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* docs: design ephemeral staging retention
High-frequency camera and audio inputs can fill a worker before the current six-hour cleanup window expires.
Define a one-hour retention policy that preserves recently modified request payloads.
Assisted-by: Codex:gpt-6
* docs: make ephemeral staging request-owned
Time-based retention can still fill a worker under bursty or high-rate input. Define request-lifecycle cleanup with capacity reservation and crash recovery.
Assisted-by: Codex:gpt-6
* feat(distributed): release exact staged keys
Request inputs need transport-neutral cleanup after backend calls. Add authenticated exact-key deletion for HTTP and coordinated cache eviction before shared-object deletion for S3/NATS.
Preserve URL metacharacters as filename data, reject unsafe keys, and remove upload sidecars while pruning empty request directories.
Assisted-by: Codex:gpt-6
* fix(distributed): release staged request inputs
Ephemeral inputs remained on workers after inference completed. Release each exact key after synchronous and streaming calls, including partial staging failures.
Use a bounded cleanup context so caller cancellation cannot suppress release. Preserve caller requests and backend results when cleanup fails.
Assisted-by: Codex:gpt-6
* feat(worker): bound ephemeral staging capacity
Concurrent staging can otherwise exceed its byte limit or consume reserved filesystem headroom. Explicit states keep bytes charged through each reservation, write, and commit transition.
Use a synchronized waiter count to prove Commit blocks until bounded writers close, and retain committed baselines across re-reservation.
Assisted-by: Codex:gpt-6
* feat(worker): enforce ephemeral staging bounds
Share capacity accounting across HTTP and S3 request inputs so workers
reject uploads before exhausting their filesystem. Reconcile exact release
and crash recovery with the same guard.
Assisted-by: Codex:gpt-6
* fix(distributed): make staged release race-safe
Pin each release path component before removing request-owned inputs and sidecars. Stop pruning when a directory identity changes.
Assisted-by: Codex:gpt-6
* fix(worker): retain staged input ownership
Keep committed request inputs protected from age recovery until exact release ends their ownership. Startup-scanned files remain reclaimable and can acquire ownership through reservation.
Assisted-by: Codex:gpt-6
* fix(worker): claim cached ephemeral inputs
Keep startup-scanned cache hits owned while inference uses them and reconcile their actual size against capacity.
Assisted-by: Codex:gpt-6
* fix(distributed): enforce staging admission
Propagate multimodal staging failures before inference and claim matching ephemeral HTTP cache entries. Fall back to PUT when an older worker does not support claims.
Assisted-by: Codex:gpt-6
* fix(distributed): close staging accounting gaps
Keep unknown-length reservations charged until bytes reach disk and bound NATS release waits by the lifecycle cleanup deadline.
Assisted-by: Codex:gpt-6
* fix(distributed): restage swept cache hits
Treat files removed between cache probing and ownership claims as misses so HTTP and S3 workers can stage them again.
Assisted-by: Codex:gpt-6
* fix(distributed): release staged inputs by request
Release every input from one inference with one fixed-size worker coordination request. Fence request ingress against cleanup, bound staging capacity and cleanup state, and retain exact-key release for rolling upgrades.
Assisted-by: Codex:gpt-6
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): default audio-cpp models to backend:best
The audio-cpp engine creates its session on the CPU backend when no
backend option is given, so every gallery model ran CPU-only even on
machines where a CUDA/Vulkan/Metal device was registered. backend:best
selects the best available backend and falls back to CPU.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
* docs(audio-cpp): explain gallery device selection
Document automatic compute backend selection and the CPU override.
Assisted-by: Codex:gpt-6
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
---------
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The device fell back to CPU unless the model config set cuda: true,
while MPS right below was auto-detected — GPU hosts silently rendered
on CPU for any gallery entry missing the flag. Use CUDA whenever torch
reports it available (ROCm builds included), keep cuda: true as an
explicit force, and allow pinning with the device: model option (e.g.
options: ["device:cpu"]). Gallery entries stay untouched.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
Virtual model names have no primary file to anchor the worker path.
Companion assets still stage successfully, but relative options retain
an incorrect model directory and fail to load.
Derive the worker root from successfully staged option assets when the
primary path is absent. Cover Buffalo packs, files, directories,
overrides, and failed transfers. Document the frontend upgrade.
Assisted-by: Codex:gpt-6 golangci-lint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
A worker can retain all model bytes with an unfinished-upload marker.
Retries then start at zero and repeatedly fail with HTTP 416.
Verify the existing bytes and finalize same-file retries at full size.
Reuse the normal integrity checks so corrupt content cannot be accepted.
Add regression coverage and document worker recovery.
Assisted-by: Codex:gpt-6 golangci-lint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Accept original embeddings and timestamps so clients can restore faces
when the in-memory store restarts. Derive stable IDs from exact vectors
to make registration retries preserve identity without duplicate entries.
Assisted-by: Codex:GPT-6 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): stage sound detection audio
Sound detection passes frontend temporary paths directly to remote
workers, unlike transcription. Stage the WAV before classification so
CED can read it without a shared temporary directory.
Preserve the original request for retries and propagate staging errors
without calling the backend. Cover staging, request preservation, and
error handling with regression tests.
Assisted-by: Codex:GPT-6 golangci-lint
* test(distributed): verify routed sound staging
Call sound detection through the client returned by SmartRouter.Route.
This checks interface dispatch through both routing wrappers, rather
than constructing FileStagingClient directly.
The test fails without the sound-staging override and passes with it.
Assisted-by: Codex:GPT-6 golangci-lint
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
