Do not seed streaming reasoning state when the latest prompt thinking marker is already followed by its matching closing marker. This keeps direct Gemma 4 output in content when its template disables thinking with a preclosed channel.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
backend.stop was the one lifecycle subject a worker never answered. The
controller published and returned nil as soon as the local publish
succeeded, so a stop that killed nothing, and a stop that failed
outright, were indistinguishable from one that worked.
The unload endpoint calls model.unload and then StopBackend. Only the
first is acknowledged, so the endpoint answered 200 while the backend
kept running and held its VRAM, and its own "backend stop failed" branch
could never run. The worker logged the failure and nobody saw it.
Give the subject a reply. The worker now enumerates the process keys it
terminated and reports any per-process error, so StopBackend fails when
the stop failed. Resolving to nothing stays a success: stopping a backend
that is not running leaves the caller in the state it asked for, and
eviction paths stop already-gone models routinely. The empty list is what
says nothing matched, and ReportsStoppedProcesses is what makes that
emptiness trustworthy, the same way BackendDeleteReply handles it.
A worker built before this reply still receives the request and still
stops the backend, it only stays silent, so a timeout degrades to the old
assumption rather than failing every stop on a fleet mid-upgrade. Only
silence degrades: a transport error is still reported, because
UnloadRemoteModel skips its registry cleanup for a node it could not
reach and needs to keep hearing about that.
Assisted-by: Claude:claude-opus-5 golangci-lint
Gallery installs merged family defaults at the YAML root and only re-marshaled them on the artifact path. Persist the defaults in the loader-visible parameters map for every install path while preserving authored overrides.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Adds FunASR/SenseVoice as a Python backend for speech-to-text with
support for CPU, CUDA 12/13, ROCm, Intel SYCL, L4T, and Apple MPS.
Co-authored-by: xingyifeng <xingyifeng@users.noreply.github.com>
Agent Status replaced the chat route and unmounted its EventSource. Any response still in flight could then disappear from the conversation.\n\nOpen status in a separate tab so the chat keeps its live connection until the response completes.\n\nAssisted-by: Codex:gpt-5 [eslint]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Saved profiles previously resolved to one audio path and transcript, so
cloning backends could not use several examples of one personality.
Store ordered audio and transcript pairs while preserving the legacy
first-reference fields. Fish Speech and audio.cpp receive all pairs,
including on distributed workers. Other backends retain their
single-reference behavior.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The API mock also matched navigation to /app/tts and returned a WAV download instead of the React page. Let non-POST requests reach the test server.
Assisted-by: Codex:gpt-5 [Playwright]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
TTS instructions add a second textarea to the page. Target the speech input by its placeholder so the history test does not depend on the page having one textarea.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Let studio users guide speech delivery for backends that support request instructions. Blank guidance stays out of requests and media history.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Clients cannot discover the named voices that an installed TTS model accepts without consulting backend-specific documentation. Expose voice metadata through the audio API and let custom model configs declare their own catalog.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The pinned audio.cpp revision supports HIP, but LocalAI neither builds a ROCm image nor accepts its backend option. AMD hosts therefore fall back to the CPU image.
Build and publish the HIP variant, connect it to AMD capability selection, and accept both upstream HIP names.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Expose diffusers audio pipelines through the existing sound-generation RPC. AudioLDM2 can now return PCM WAV output from the model gallery without a separate backend.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a NATS event contract and exact-residency provider for backend KV cache reports. Keep guessed request observations as the default routing source while maintaining the reported index for future producers.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a Darwin-only MLX-Video backend for LTX-2 and converted Wan checkpoints, expose it through the existing video API, and wire packaging, discovery, tests, docs, and an example.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Broadcast forced-disturb events and successful scale-up resets so every frontend shares the same rolling autoscale signal. Deduplicate NATS echoes, expose an origin-only Prometheus counter, and document cluster behavior.
Closes#10083
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Expose original_config_file through the model configuration metadata registry so registry completeness checks and generated configuration surfaces stay in sync.
Assisted-by: Codex:gpt-5 [systematic-debugging]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Route prefix-cache replica selection through composable filters, weighted scorers, and a replaceable picker. Preserve the existing load guard and deterministic selection policy while exposing per-model scorer weights through scheduling configuration.
Assisted-by: Codex:gpt-5 [go-vet]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(auth): bypass API-key auth for CORS preflight (OPTIONS) requests
When API-key auth is enabled, a browser making a cross-origin API call
first sends an OPTIONS CORS preflight, which cannot carry credentials by
HTTP spec. The auth middleware is registered (app.go:324) before the CORS
middleware (app.go:337-347), so the preflight hit auth first and returned
401 before the CORS middleware could answer it, blocking the actual call.
Bypass auth for OPTIONS so the request reaches the CORS middleware, which
answers the preflight with 200 + headers. Real API requests (GET/POST/etc.)
still require auth. Regression test added (red on master, green on branch).
Refs #4576
Signed-off-by: supermario_leo <leo.stack@outlook.com>
* fix(auth): exempt CORS preflights via publicRouteRegistry instead of middleware bypass
Route the global OPTIONS exemption through publicRouteRegistry
(OPTIONS on every path, replacing the OPTIONS-under-/api/auth/ rule
it subsumes) instead of a hardcoded method check inside Middleware,
so "which requests skip auth" has one mechanism. Preflights now flow
through the same authenticate-then-public-rules path as other public
routes, which also lets a credentialed OPTIONS request keep its user
context.
Update the route-coverage allowlist and the near-prefix lookalike
table for the new semantics (OPTIONS is public on every path by
design; near-prefix privacy stays pinned by the non-OPTIONS entries),
and fix the authentication docs' exempt-route enumeration, which
still described OPTIONS as an /api/auth/-only exemption.
Signed-off-by: supermario_leo <leo.stack@outlook.com>
---------
Signed-off-by: supermario_leo <leo.stack@outlook.com>
fix(ui): preserve decoded route parameters
React Router already decodes dynamic path segments before exposing them through useParams. Decoding those values again crashes pages for names containing a literal percent sign and mutates escape-like substrings.
Use route parameters as-is, encode the model editor API path at the outbound boundary, and cover all affected pages with Playwright.
Fixes#11882
Assisted-by: Codex:gpt-5 eslint playwright
Signed-off-by: QiuLG <l237455523@outlook.com>
fix(ds4): resolve repository imports to GGUF
Select a concrete DeepSeek V4 Flash GGUF from Hugging Face metadata while honoring quantization preference order. Reject unresolved repository imports instead of emitting an invalid bare URI.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
When no sampled candidate fits GPU memory, ranking falls back to the
oversized pool and labels its first model Best fit. Keep GPU picks within
the existing 95% budget and hide the section when no candidate qualifies.
Remove static GPU starter picks so Home cannot reintroduce the same error.
Add browser regressions for both sections and document the empty result.
CPU fallback behavior stays unchanged.
Assisted-by: Codex:gpt-6 [Codex]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
JPEG base64 starts with /9j and was classified as an absolute path in distributed mode. Detect valid JPEG base64 by its decoded magic bytes so vision requests reach the backend unchanged while malformed path-like inputs retain existing staging behavior.
Assisted-by: Codex:gpt-5
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(chat): template.system_messages_after_first — merge or forward late system turns
Tokenizer chat templates such as Qwen3.8 / Qwen3.8-Flash-Next raise
'System message must be at the beginning' for system-role messages that
appear after the leading system block, while agent frameworks (cogito tool
selection and adjustment prompts) legitimately append system instructions
mid-conversation. Every such request failed with a 500 (48 errors in one
10-task agent run).
New per-model option template.system_messages_after_first:
merge fold late system turns into the leading system message
user forward them as user-role turns at their original position
Default (unset) keeps the current pass-through behaviour.
Fixes#11876
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
* docs(model-config): document template.system_messages_after_first
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
* fix(config/meta): register template.system_messages_after_first in the field registry
TestAllFieldsHaveRegistryEntries requires every model-config field to have
a registry entry. Adds the entry (templates section, select component) and
the option list for the new field so the coverage gate passes.
Assisted-by: Claude:claude-fable-5-1
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
---------
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
* fix(realtime): support session voice profile switching
Keep the active resolved voice binding on the realtime session so updates can atomically replace model, voice, and profile parameters while releasing leases at the correct lifecycle boundaries.
Assisted-by: Codex:gpt-5
* docs(realtime): explain voice profile switching
Document the session.update payload for selecting a Voice Library URI and clarify precedence when changing the model in the same event.\n\nAssisted-by: Codex:gpt-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Capacity guards reject symlink components. On macOS, temporary paths
start with /var, which links to /private/var, so the new staging tests
fail before exercising cleanup or capacity accounting.
Resolve the fixture directories before building guarded paths. Keep
explicit symlinks within the fixtures for containment tests.
Assisted-by: Codex:gpt-6
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Backend processes shared the host temporary directory, so crashes could leave request images and audio behind until the filesystem filled. Give each process a locked LocalAI-owned runtime, remove scratch on exit, and sweep only marked abandoned runtimes at the next start.
Also close known request error-path leaks in the Python media backends, CrispASR, LongCat Video, and stable-diffusion.cpp.
Assisted-by: Codex:gpt-5
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(realtime): resolve saved voice profiles
Realtime pipelines now validate saved voices against the selected TTS model and retain leased audio until session teardown. Each synthesis request receives its own transcript parameter map.
Assisted-by: Codex:GPT-5
* docs(tts): document realtime voice defaults
Show how a realtime pipeline selects a saved Voice Library profile at session start. Clarify which session voice updates remain supported.
Assisted-by: Codex:GPT-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Realtime turns could invoke the LLM and TTS even when speech transcription returned only whitespace. This let ambient noise produce unsolicited assistant output and polluted conversation history with an empty user turn.
Require non-blank transcript text before automatic response generation while preserving the completed transcription event.
Assisted-by: Codex:gpt-5 golangci-lint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* docs: design ephemeral staging retention
High-frequency camera and audio inputs can fill a worker before the current six-hour cleanup window expires.
Define a one-hour retention policy that preserves recently modified request payloads.
Assisted-by: Codex:gpt-6
* docs: make ephemeral staging request-owned
Time-based retention can still fill a worker under bursty or high-rate input. Define request-lifecycle cleanup with capacity reservation and crash recovery.
Assisted-by: Codex:gpt-6
* feat(distributed): release exact staged keys
Request inputs need transport-neutral cleanup after backend calls. Add authenticated exact-key deletion for HTTP and coordinated cache eviction before shared-object deletion for S3/NATS.
Preserve URL metacharacters as filename data, reject unsafe keys, and remove upload sidecars while pruning empty request directories.
Assisted-by: Codex:gpt-6
* fix(distributed): release staged request inputs
Ephemeral inputs remained on workers after inference completed. Release each exact key after synchronous and streaming calls, including partial staging failures.
Use a bounded cleanup context so caller cancellation cannot suppress release. Preserve caller requests and backend results when cleanup fails.
Assisted-by: Codex:gpt-6
* feat(worker): bound ephemeral staging capacity
Concurrent staging can otherwise exceed its byte limit or consume reserved filesystem headroom. Explicit states keep bytes charged through each reservation, write, and commit transition.
Use a synchronized waiter count to prove Commit blocks until bounded writers close, and retain committed baselines across re-reservation.
Assisted-by: Codex:gpt-6
* feat(worker): enforce ephemeral staging bounds
Share capacity accounting across HTTP and S3 request inputs so workers
reject uploads before exhausting their filesystem. Reconcile exact release
and crash recovery with the same guard.
Assisted-by: Codex:gpt-6
* fix(distributed): make staged release race-safe
Pin each release path component before removing request-owned inputs and sidecars. Stop pruning when a directory identity changes.
Assisted-by: Codex:gpt-6
* fix(worker): retain staged input ownership
Keep committed request inputs protected from age recovery until exact release ends their ownership. Startup-scanned files remain reclaimable and can acquire ownership through reservation.
Assisted-by: Codex:gpt-6
* fix(worker): claim cached ephemeral inputs
Keep startup-scanned cache hits owned while inference uses them and reconcile their actual size against capacity.
Assisted-by: Codex:gpt-6
* fix(distributed): enforce staging admission
Propagate multimodal staging failures before inference and claim matching ephemeral HTTP cache entries. Fall back to PUT when an older worker does not support claims.
Assisted-by: Codex:gpt-6
* fix(distributed): close staging accounting gaps
Keep unknown-length reservations charged until bytes reach disk and bound NATS release waits by the lifecycle cleanup deadline.
Assisted-by: Codex:gpt-6
* fix(distributed): restage swept cache hits
Treat files removed between cache probing and ownership claims as misses so HTTP and S3 workers can stage them again.
Assisted-by: Codex:gpt-6
* fix(distributed): release staged inputs by request
Release every input from one inference with one fixed-size worker coordination request. Fence request ingress against cleanup, bound staging capacity and cleanup state, and retain exact-key release for rolling upgrades.
Assisted-by: Codex:gpt-6
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Virtual model names have no primary file to anchor the worker path.
Companion assets still stage successfully, but relative options retain
an incorrect model directory and fail to load.
Derive the worker root from successfully staged option assets when the
primary path is absent. Cover Buffalo packs, files, directories,
overrides, and failed transfers. Document the frontend upgrade.
Assisted-by: Codex:gpt-6 golangci-lint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
A worker can retain all model bytes with an unfinished-upload marker.
Retries then start at zero and repeatedly fail with HTTP 416.
Verify the existing bytes and finalize same-file retries at full size.
Reuse the normal integrity checks so corrupt content cannot be accepted.
Add regression coverage and document worker recovery.
Assisted-by: Codex:gpt-6 golangci-lint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Accept original embeddings and timestamps so clients can restore faces
when the in-memory store restarts. Derive stable IDs from exact vectors
to make registration retries preserve identity without duplicate entries.
Assisted-by: Codex:GPT-6 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): stage sound detection audio
Sound detection passes frontend temporary paths directly to remote
workers, unlike transcription. Stage the WAV before classification so
CED can read it without a shared temporary directory.
Preserve the original request for retries and propagate staging errors
without calling the backend. Cover staging, request preservation, and
error handling with regression tests.
Assisted-by: Codex:GPT-6 golangci-lint
* test(distributed): verify routed sound staging
Call sound detection through the client returned by SmartRouter.Route.
This checks interface dispatch through both routing wrappers, rather
than constructing FileStagingClient directly.
The test fails without the sound-staging override and passes with it.
Assisted-by: Codex:GPT-6 golangci-lint
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
A quantization job that runs with no client on its progress stream stays
"queued" forever, in the API and in state.json, while the finished artifact
sits on disk. state.json was written once by StartJob, and the only code that
advanced a job afterwards lived inside the stream callback of StreamProgress,
so job state depended on somebody watching it.
The backend's progress stream cannot simply gain a second reader: each job owns
one queue.Queue and QuantizationProgress pops from it, so two consumers split
the updates rather than both seeing them. The stream has to be opened exactly
once per job.
StartJob now starts watchProgress on the application context (the request
context is done as soon as the handler returns). That goroutine is the single
reader: it applies each update to the job -- in the cross-replica store and in
state.json, terminal statuses still winning over late updates -- and republishes
it in-process. StreamProgress becomes a pure reader over that fan-out and no
longer loads a backend or opens a stream. A client attaching to a job that has
already finished, including one hydrated from disk after a restart, gets a final
event built from the stored job instead of blocking.
Two paths used to end a client's stream by breaking the gRPC connection and now
release it explicitly: StopJob kills the backend, so it publishes the stopped
event itself; and a stream that ends without a terminal update means the backend
is gone, so the job is recorded as failed rather than left running forever.
Signed-off-by: Tai An <antai12232931@outlook.com>
The recommended sampling parameters for a model family were applied at
install and then never took effect. Two things went wrong on the way to
disk.
They were written as top level keys. ModelConfig embeds PredictionOptions
under the "parameters" yaml key, so temperature, top_p, top_k, min_p,
repeat_penalty and presence_penalty are only read from there. At the top
level they parse without error and are then ignored for the life of the
model.
They were also merged in after the YAML had already been marshalled. The
only re-marshal sat behind the artifact binding, which an entry carrying
files: never reaches, so for those entries the defaults were computed and
then dropped before anything was written.
Neither failure was visible in normal use. ApplyInferenceDefaults runs
again at load time and fills the same values from the same table, so the
model ends up tuned correctly while the file on disk pins nothing. It
surfaces when someone edits one of those values expecting it to win, or
when a family is absent from inference_defaults.json and there is nothing
to refill from.
Both install paths are covered: an entry carrying files:, and one that
binds a primary artifact instead.
The empty base spec asserted that the authored parameters block landed
verbatim. It now checks the authored keys individually, because the family
defaults are merged into that same block.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me>
Web Chat Settings left the System Prompt field empty but still treated a
blank/whitespace value as an explicit system turn. That satisfied
tokenizer chat templates' messages[0].role == system check and suppressed
the model YAML system_prompt on fresh chats.
Omit empty/whitespace system messages in the React and Alpine UIs, strip
them server-side, and inject config.SystemPrompt for tokenizer-template
models when the request has no real system turn.
Fixes#11834
Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
* fix(distributed): evict only when a node is known to be full
scheduleNewModel asked the registry for a free replica slot and treated
every error as "this node is full", so a control-plane database slow
enough to time out the lookup evicted a healthy loaded model. The
evicted process died, a peer frontend still holding its address dialled
the dead port and retried, and the model thrashed between nodes. The
comment on the branch already said it meant a full node; the code never
tested for it.
Evict only on ErrNoFreeSlot. Any other error now returns and names the
lookup that failed, so a slow database degrades into a diagnosable
load failure instead of into lost work.
An audit of the rest of the router found one branch of the same shape:
node selection discarded the error from its last-resort finder, so a
database timeout there also produced a nil node and evicted for it.
That path now returns unless the finder said gorm.ErrRecordNotFound,
which is the only answer that means the cluster had no node to give.
No other destructive branch in router.go fires on a generic error.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): checkpoint heartbeat writes instead of writing every beat
Every heartbeat UPDATEd backend_nodes. Six nodes at a ten second beat is
roughly 52,000 writes a day against a six-row table, and that churn is
what turned a blocked autovacuum into a 460 MB table whose six-row scan
cost 867 ms and timed out the queries that place models.
A beat carrying only a fresher timestamp now waits for the checkpoint
interval. Each reported field is compared against the value last
persisted rather than tested for presence, because a worker sends its
disk figures on every beat and presence alone would suppress nothing.
A node's first beat, a changed total VRAM, total disk or GPU vendor,
and a free VRAM, RAM or disk reading that has moved more than 256 MiB
from the persisted value all still write at once. A node that is not
active is never suppressed, because it recovers only when the health
monitor sees a fresh timestamp.
The persisted column is up to one interval stale by design, so the
stale-node threshold moves from 60s to 5m to cover it.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): fail worker readiness when a held backend is unreachable
The readiness gate tracked only the NATS link, so a worker whose backend
processes had died still answered /readyz with 200 and kept receiving
loads. One node did exactly that during an incident: it reported healthy
while its backend port refused connections, and every load routed to it
failed.
Readiness is now the NATS link and, for each backend process the worker
believes it is running, a short dial of its recorded address. A worker
holding no backends stays ready, because idle is a healthy state.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): keep a starting backend out of the readiness dial set
A backend process is inserted into the supervisor map with its gRPC
address already recorded, but the address refuses connections until the
gRPC server binds, which the startup poll allows up to 30 seconds for and
which takes 10 to 15 seconds on a slow node. The new data-path readiness
probe dialled that address straight away, so a worker answered /readyz
with 503 for the whole of every cold backend start. The container
HEALTHCHECK absorbs that, but a Kubernetes readinessProbe at 10s does
not, and the worker would leave rotation each time it loaded a model.
The skip for a stopping process had no counterpart at the other end of
the lifecycle. Backend processes now carry a serving flag, set where the
startup health-check gate succeeds, and the probe dials only processes
that are serving and not yet stopping. backendStartStillValid becomes
markBackendServing: the check and the mark must share one lock hold, so
the flag can only ever land on the entry the key currently owns.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(distributed): export control-plane database health gauges
Four transactions wedged on a corrupt index held the vacuum horizon open
for 42 days. Nothing measured it, so the first symptom anyone saw was
models failing to load six weeks later, by which time a six-row table
had grown to 460 MB.
Export the oldest xmin age, the longest open transaction, and the dead
tuple ratio on the registry tables. The first is the number that would
have caught it: it sits near zero in health and was 21,002,291.
Sampling is scrape-driven behind a cache, and a failed sample reports
the last good values rather than failing the scrape, because these
gauges matter most when the database is already struggling.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): rate-limit failed control-plane database samples
The cache advanced its clock only on a successful sample, so once the
database started failing every scrape retried the query immediately.
That turned the cache off in the one regime it exists for: a retry
storm at scrape cadence aimed at a database already in trouble. A
catalog read that consistently exceeds the 5 second timeout also paid
that cost on every scrape, with all scrapes serialised behind the
sampler mutex.
Time every attempt rather than every success, so failures and timeouts
cost the same interval as good samples. Whether a good sample exists
moves to its own field, keeping the gauges absent until the first
success and holding the last good values through later failures.
Also note in the runbook that pg_stat_activity cannot see prepared
transactions or replication slot xmins, so a healthy-looking xmin age
does not by itself rule out a blocked horizon.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(distributed): pin that a failing database evicts nothing
Exercises the real distributed stack against a control-plane database that
refuses the router's slot lookup, and asserts the scheduler reports the
lookup it could not answer instead of falling through to eviction.
The failure is injected with privileges rather than a statement timeout. A
timeout set with ALTER DATABASE also breaks AutoMigrate, and it leaks into
every later spec in the suite unless it is reset, so the spec would end up
testing the migration rather than the scheduler. Instead the spec creates a
dedicated login role, points a second gorm handle at it, and revokes that
role's SELECT on node_models.replica_index. This has to be a separate role:
the test container's owner is a PostgreSQL superuser, and superusers bypass
every privilege check, so revoking from CURRENT_USER is recorded and then
ignored.
The revoke is scoped to one column on purpose. Revoking the whole table
would also blind node selection, which runs first and has a guard of its
own, so the scheduler would never reach the slot lookup this spec is about.
Leaving every other column readable lets selection succeed and lands the
refusal exactly on NextFreeReplicaIndex, which plucks replica_index. The
grant is restored from BeforeEach via DeferCleanup, so a failing assertion
or a panic cannot hand the next spec a role that cannot read.
Reverting the eviction guard fails this spec, which is the point of it: the
router then reports "no replica slot on keeper and eviction failed" for an
error that was never evidence the node was full. The surviving-row
assertions are secondary under this injection, because the eviction path
reads whole node_models rows and the same revoke blinds it too; a comment
in the spec says so, so nobody mistakes them for the load-bearing ones.
Also documents why the vector store and the control plane must not share a
database: the removable-tuple cutoff is per database, not per table, so one
transaction left open anywhere stops autovacuum reclaiming the node
registry, and a six-row table bloats into hundreds of megabytes. The note
names LOCALAI_AUTH_DATABASE_URL and LOCALAI_AGENT_POOL_DATABASE_URL as the
two knobs that must differ, and the localai_control_plane_oldest_xmin_age
gauge as the way to see it coming.
grep for StaleNodeThreshold and HealthCheckInterval in
core/config/runtime_settings_registry.go returns no matches: the
distributed duration knobs are not exposed as runtime settings, so the new
heartbeat checkpoint interval follows them and needs no registry entry.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): close the review gaps in the heartbeat and health path
The stale-node threshold moved from 60 seconds to 5 minutes in this branch
because checkpointing makes last_heartbeat up to one checkpoint interval
behind by design. Two things were left inconsistent with that. NewHealthMonitor
still fell back to a hardcoded 60 seconds when handed a zero threshold, so any
future caller that stopped passing the configured value would mark every
healthy, beating node offline on every cycle. And the threshold itself had a
flag-name constant but no AppOption, no CLI field and no env binding, so an
operator who widened --node-heartbeat-checkpoint had no way to widen the
threshold to match. The fallback now tracks config.DefaultStaleNodeThreshold,
and --stale-node-threshold / LOCALAI_STALE_NODE_THRESHOLD is wired the same
way its sibling is.
Heartbeat suppression compared the RAW reported free VRAM against the
snapshot, but the column persists capAvailable(raw, ceiling). On any node with
a VRAM budget set, whose actual free VRAM oscillates above that ceiling, every
beat looked material while the persisted value never moved: suppression was
defeated on exactly the nodes an operator had configured, and the write
amplification this branch exists to remove came straight back there. The
comparison and the snapshot now both hold the capped figure, so they measure
the same quantity as the column.
Fixing that needs the ceiling, and reading it cost a SELECT on every beat,
including suppressed ones. The skip decision therefore moved ahead of the
updates map and now reuses the ceiling cached on the last durable write, while
the write path still re-reads it before capping anything. A ceiling that
changed inside the checkpoint window can cost one extra or one late write; it
cannot persist a wrong figure. A suppressed beat now costs no query at all.
Also: the operations section now says to grant pg_read_all_stats to the
LocalAI role, because PostgreSQL blanks backend_xmin and xact_start for
sessions owned by other roles, and the transaction that wedged the horizon in
the incident was a co-located vector store connecting as a different role, so
without the grant the new gauge sees only our own sessions. The compose
healthcheck comment now describes readiness covering the backend data path,
and the control-plane gauge registration records the otel.SetMeterProvider
ordering it depends on.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): resolve the gauge's table names through gorm
The dead-tuple gauge queried pg_stat_user_tables against a hardcoded list
of three table names. Those three do not agree on where their name comes
from: BackendNode and NodeModel take gorm's default pluralisation, while
GalleryOperationRecord overrides TableName, and gallery_operations
already had a constant of its own that the list duplicated.
A literal list keeps compiling after any of that moves, and the query
then matches nothing. The failure is silent and it points the wrong way:
a dead-tuple ratio that matched no rows reports the same numbers as a
cluster with no bloat, so the gauge would look healthiest exactly when it
had stopped working.
Ask gorm what each model is stored as instead, which follows a TableName
override and the default pluralisation alike. A spec pins that the
override really is consulted: naive pluralisation of the type would give
gallery_operation_records, so the resolution cannot quietly stop asking
the model.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
stageDirectory and countStageableFiles already skip them, but
stageOptionDir did not - and it is the path sherpa-onnx voices take for
espeak-ng-data. The receiver writes "<file>.sha256" for every file it
accepts, so staging the sidecars made it write sidecars for those in
turn, one level deeper on every load.
Observed on a live node: "<file>.sha256" repeated eleven times, 5077
junk files out of 7832 in the models dir, and still growing. Staging
never finished, so vits-piper-it_IT-paola-sherpa stayed permanently
"staging on node" and every realtime warmup needing that voice failed
with the session then going silent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0142UfUh8HWxdim5JZqf8Tr6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Install 404s because the gallery still points at mmproj-...-f16.gguf.
HF only ships ...-F16.gguf now, with a different sha256.
Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
