Both GGUF entries select llama-cpp, whose pinned revision does not
recognize instella-moe. Remove the broken install options and document
why they are unavailable until LocalAI ships a compatible backend.
Refs #11681
Assisted-by: Codex:GPT-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
grpc::ServerWriter::Write() returns false once the peer is gone, and
PredictStream ignored that result at every call site. The handler kept
pulling decoded tokens and writing them into a dead stream, so the
llama.cpp slot stayed busy until the generation ended on its own terms.
A model configured with max_tokens 0 and a large context ends on its own
terms only at the context limit. On a 35B model at ~41 t/s a 120k context
is about fifty minutes, and a slot held that long is a slot every other
request for that model queues behind. Two abandoned requests were enough
to make a node with free VRAM and a healthy control plane serve nothing:
new requests timed out waiting for a slot, each timeout abandoned another
generation, and the node fell further behind the longer it ran.
Track the peer instead. The first failed write retires it for good, since
a stream never recovers, and the RPC's own cancellation flag folds into
the same predicate so the loop has one condition to test. Returning early
is what frees the slot: ~server_response_reader() posts
SERVER_TASK_TYPE_CANCEL for whatever is still decoding.
TTSStream already checked Write(); this brings PredictStream in line.
Cancellation stays cooperative and is checked between decoded results, so
a batch already in flight may finish before the request stops.
Assisted-by: Claude:claude-opus-5
backend.stop was the one lifecycle subject a worker never answered. The
controller published and returned nil as soon as the local publish
succeeded, so a stop that killed nothing, and a stop that failed
outright, were indistinguishable from one that worked.
The unload endpoint calls model.unload and then StopBackend. Only the
first is acknowledged, so the endpoint answered 200 while the backend
kept running and held its VRAM, and its own "backend stop failed" branch
could never run. The worker logged the failure and nobody saw it.
Give the subject a reply. The worker now enumerates the process keys it
terminated and reports any per-process error, so StopBackend fails when
the stop failed. Resolving to nothing stays a success: stopping a backend
that is not running leaves the caller in the state it asked for, and
eviction paths stop already-gone models routinely. The empty list is what
says nothing matched, and ReportsStoppedProcesses is what makes that
emptiness trustworthy, the same way BackendDeleteReply handles it.
A worker built before this reply still receives the request and still
stops the backend, it only stays silent, so a timeout degrades to the old
assumption rather than failing every stop on a fleet mid-upgrade. Only
silence degrades: a transport error is still reported, because
UnloadRemoteModel skips its registry cleanup for a node it could not
reach and needs to keep hearing about that.
Assisted-by: Claude:claude-opus-5 golangci-lint
Adds FunASR/SenseVoice as a Python backend for speech-to-text with
support for CPU, CUDA 12/13, ROCm, Intel SYCL, L4T, and Apple MPS.
Co-authored-by: xingyifeng <xingyifeng@users.noreply.github.com>
Saved profiles previously resolved to one audio path and transcript, so
cloning backends could not use several examples of one personality.
Store ordered audio and transcript pairs while preserving the legacy
first-reference fields. Fish Speech and audio.cpp receive all pairs,
including on distributed workers. Other backends retain their
single-reference behavior.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Let studio users guide speech delivery for backends that support request instructions. Blank guidance stays out of requests and media history.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Clients cannot discover the named voices that an installed TTS model accepts without consulting backend-specific documentation. Expose voice metadata through the audio API and let custom model configs declare their own catalog.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The pinned audio.cpp revision supports HIP, but LocalAI neither builds a ROCm image nor accepts its backend option. AMD hosts therefore fall back to the CPU image.
Build and publish the HIP variant, connect it to AMD capability selection, and accept both upstream HIP names.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Expose diffusers audio pipelines through the existing sound-generation RPC. AudioLDM2 can now return PCM WAV output from the model gallery without a separate backend.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a NATS event contract and exact-residency provider for backend KV cache reports. Keep guessed request observations as the default routing source while maintaining the reported index for future producers.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a Darwin-only MLX-Video backend for LTX-2 and converted Wan checkpoints, expose it through the existing video API, and wire packaging, discovery, tests, docs, and an example.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Document how to connect LocalAI to the upstream RKLLM OpenAI-compatible server through cloud-proxy, including supported SoCs and current limitations.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Prefer an explicitly configured Triton assembler, otherwise use the executable ptxas from CUDA_HOME so torch.compile can target GPU architectures newer than Triton bundled tooling.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Broadcast forced-disturb events and successful scale-up resets so every frontend shares the same rolling autoscale signal. Deduplicate NATS echoes, expose an origin-only Prometheus counter, and document cluster behavior.
Closes#10083
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Route prefix-cache replica selection through composable filters, weighted scorers, and a replaceable picker. Preserve the existing load guard and deterministic selection policy while exposing per-model scorer weights through scheduling configuration.
Assisted-by: Codex:gpt-5 [go-vet]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(auth): bypass API-key auth for CORS preflight (OPTIONS) requests
When API-key auth is enabled, a browser making a cross-origin API call
first sends an OPTIONS CORS preflight, which cannot carry credentials by
HTTP spec. The auth middleware is registered (app.go:324) before the CORS
middleware (app.go:337-347), so the preflight hit auth first and returned
401 before the CORS middleware could answer it, blocking the actual call.
Bypass auth for OPTIONS so the request reaches the CORS middleware, which
answers the preflight with 200 + headers. Real API requests (GET/POST/etc.)
still require auth. Regression test added (red on master, green on branch).
Refs #4576
Signed-off-by: supermario_leo <leo.stack@outlook.com>
* fix(auth): exempt CORS preflights via publicRouteRegistry instead of middleware bypass
Route the global OPTIONS exemption through publicRouteRegistry
(OPTIONS on every path, replacing the OPTIONS-under-/api/auth/ rule
it subsumes) instead of a hardcoded method check inside Middleware,
so "which requests skip auth" has one mechanism. Preflights now flow
through the same authenticate-then-public-rules path as other public
routes, which also lets a credentialed OPTIONS request keep its user
context.
Update the route-coverage allowlist and the near-prefix lookalike
table for the new semantics (OPTIONS is public on every path by
design; near-prefix privacy stays pinned by the non-OPTIONS entries),
and fix the authentication docs' exempt-route enumeration, which
still described OPTIONS as an /api/auth/-only exemption.
Signed-off-by: supermario_leo <leo.stack@outlook.com>
---------
Signed-off-by: supermario_leo <leo.stack@outlook.com>
Add the published UD-Q4_K_XL shards and BF16 vision projector.
Document installation and the default context size.
Assisted-by: Codex:gpt-6
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Add Q4_K_M and Q5_K_M builds with the vision projector and an install
example. Pin downloads to the verified Hugging Face revision.
Assisted-by: Codex:gpt-6
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(realtime): support session voice profile switching
Keep the active resolved voice binding on the realtime session so updates can atomically replace model, voice, and profile parameters while releasing leases at the correct lifecycle boundaries.
Assisted-by: Codex:gpt-5
* docs(realtime): explain voice profile switching
Document the session.update payload for selecting a Voice Library URI and clarify precedence when changing the model in the same event.\n\nAssisted-by: Codex:gpt-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(realtime): resolve saved voice profiles
Realtime pipelines now validate saved voices against the selected TTS model and retain leased audio until session teardown. Each synthesis request receives its own transcript parameter map.
Assisted-by: Codex:GPT-5
* docs(tts): document realtime voice defaults
Show how a realtime pipeline selects a saved Voice Library profile at session start. Clarify which session voice updates remain supported.
Assisted-by: Codex:GPT-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* docs: design ephemeral staging retention
High-frequency camera and audio inputs can fill a worker before the current six-hour cleanup window expires.
Define a one-hour retention policy that preserves recently modified request payloads.
Assisted-by: Codex:gpt-6
* docs: make ephemeral staging request-owned
Time-based retention can still fill a worker under bursty or high-rate input. Define request-lifecycle cleanup with capacity reservation and crash recovery.
Assisted-by: Codex:gpt-6
* feat(distributed): release exact staged keys
Request inputs need transport-neutral cleanup after backend calls. Add authenticated exact-key deletion for HTTP and coordinated cache eviction before shared-object deletion for S3/NATS.
Preserve URL metacharacters as filename data, reject unsafe keys, and remove upload sidecars while pruning empty request directories.
Assisted-by: Codex:gpt-6
* fix(distributed): release staged request inputs
Ephemeral inputs remained on workers after inference completed. Release each exact key after synchronous and streaming calls, including partial staging failures.
Use a bounded cleanup context so caller cancellation cannot suppress release. Preserve caller requests and backend results when cleanup fails.
Assisted-by: Codex:gpt-6
* feat(worker): bound ephemeral staging capacity
Concurrent staging can otherwise exceed its byte limit or consume reserved filesystem headroom. Explicit states keep bytes charged through each reservation, write, and commit transition.
Use a synchronized waiter count to prove Commit blocks until bounded writers close, and retain committed baselines across re-reservation.
Assisted-by: Codex:gpt-6
* feat(worker): enforce ephemeral staging bounds
Share capacity accounting across HTTP and S3 request inputs so workers
reject uploads before exhausting their filesystem. Reconcile exact release
and crash recovery with the same guard.
Assisted-by: Codex:gpt-6
* fix(distributed): make staged release race-safe
Pin each release path component before removing request-owned inputs and sidecars. Stop pruning when a directory identity changes.
Assisted-by: Codex:gpt-6
* fix(worker): retain staged input ownership
Keep committed request inputs protected from age recovery until exact release ends their ownership. Startup-scanned files remain reclaimable and can acquire ownership through reservation.
Assisted-by: Codex:gpt-6
* fix(worker): claim cached ephemeral inputs
Keep startup-scanned cache hits owned while inference uses them and reconcile their actual size against capacity.
Assisted-by: Codex:gpt-6
* fix(distributed): enforce staging admission
Propagate multimodal staging failures before inference and claim matching ephemeral HTTP cache entries. Fall back to PUT when an older worker does not support claims.
Assisted-by: Codex:gpt-6
* fix(distributed): close staging accounting gaps
Keep unknown-length reservations charged until bytes reach disk and bound NATS release waits by the lifecycle cleanup deadline.
Assisted-by: Codex:gpt-6
* fix(distributed): restage swept cache hits
Treat files removed between cache probing and ownership claims as misses so HTTP and S3 workers can stage them again.
Assisted-by: Codex:gpt-6
* fix(distributed): release staged inputs by request
Release every input from one inference with one fixed-size worker coordination request. Fence request ingress against cleanup, bound staging capacity and cleanup state, and retain exact-key release for rolling upgrades.
Assisted-by: Codex:gpt-6
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): default audio-cpp models to backend:best
The audio-cpp engine creates its session on the CPU backend when no
backend option is given, so every gallery model ran CPU-only even on
machines where a CUDA/Vulkan/Metal device was registered. backend:best
selects the best available backend and falls back to CPU.
Assisted-by: Claude:claude-fable-5
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
* docs(audio-cpp): explain gallery device selection
Document automatic compute backend selection and the CPU override.
Assisted-by: Codex:gpt-6
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
---------
Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Virtual model names have no primary file to anchor the worker path.
Companion assets still stage successfully, but relative options retain
an incorrect model directory and fail to load.
Derive the worker root from successfully staged option assets when the
primary path is absent. Cover Buffalo packs, files, directories,
overrides, and failed transfers. Document the frontend upgrade.
Assisted-by: Codex:gpt-6 golangci-lint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
A worker can retain all model bytes with an unfinished-upload marker.
Retries then start at zero and repeatedly fail with HTTP 416.
Verify the existing bytes and finalize same-file retries at full size.
Reuse the normal integrity checks so corrupt content cannot be accepted.
Add regression coverage and document worker recovery.
Assisted-by: Codex:gpt-6 golangci-lint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Accept original embeddings and timestamps so clients can restore faces
when the in-memory store restarts. Derive stable IDs from exact vectors
to make registration retries preserve identity without duplicate entries.
Assisted-by: Codex:GPT-6 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): stage sound detection audio
Sound detection passes frontend temporary paths directly to remote
workers, unlike transcription. Stage the WAV before classification so
CED can read it without a shared temporary directory.
Preserve the original request for retries and propagate staging errors
without calling the backend. Cover staging, request preservation, and
error handling with regression tests.
Assisted-by: Codex:GPT-6 golangci-lint
* test(distributed): verify routed sound staging
Call sound detection through the client returned by SmartRouter.Route.
This checks interface dispatch through both routing wrappers, rather
than constructing FileStagingClient directly.
The test fails without the sound-staging override and passes with it.
Assisted-by: Codex:GPT-6 golangci-lint
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update leejet/stable-diffusion.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(stablediffusion): adapt streaming options
Upstream now selects segmented weight streaming automatically and removes the stream_layers field. Keep the old LocalAI option as a no-op for existing model configurations.
Assisted-by: Codex:gpt-5
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
WhisperX silently returned a plain transcript when diarization lacked
the Hugging Face token required to load pyannote. Reject that request
clearly so callers do not mistake missing speaker labels for a
successful diarization.
Convert WhisperX seconds to the nanosecond duration unit used by the
transcription API.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Propagate gRPC cancellation into DS4 prompt synchronization and poll it at decode boundaries.
Stop on failed stream writes and skip parser finalization and KV persistence for abandoned partial requests.
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Claudio Maradonna <git@codeshifter.xyz>
* fix(distributed): evict only when a node is known to be full
scheduleNewModel asked the registry for a free replica slot and treated
every error as "this node is full", so a control-plane database slow
enough to time out the lookup evicted a healthy loaded model. The
evicted process died, a peer frontend still holding its address dialled
the dead port and retried, and the model thrashed between nodes. The
comment on the branch already said it meant a full node; the code never
tested for it.
Evict only on ErrNoFreeSlot. Any other error now returns and names the
lookup that failed, so a slow database degrades into a diagnosable
load failure instead of into lost work.
An audit of the rest of the router found one branch of the same shape:
node selection discarded the error from its last-resort finder, so a
database timeout there also produced a nil node and evicted for it.
That path now returns unless the finder said gorm.ErrRecordNotFound,
which is the only answer that means the cluster had no node to give.
No other destructive branch in router.go fires on a generic error.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): checkpoint heartbeat writes instead of writing every beat
Every heartbeat UPDATEd backend_nodes. Six nodes at a ten second beat is
roughly 52,000 writes a day against a six-row table, and that churn is
what turned a blocked autovacuum into a 460 MB table whose six-row scan
cost 867 ms and timed out the queries that place models.
A beat carrying only a fresher timestamp now waits for the checkpoint
interval. Each reported field is compared against the value last
persisted rather than tested for presence, because a worker sends its
disk figures on every beat and presence alone would suppress nothing.
A node's first beat, a changed total VRAM, total disk or GPU vendor,
and a free VRAM, RAM or disk reading that has moved more than 256 MiB
from the persisted value all still write at once. A node that is not
active is never suppressed, because it recovers only when the health
monitor sees a fresh timestamp.
The persisted column is up to one interval stale by design, so the
stale-node threshold moves from 60s to 5m to cover it.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): fail worker readiness when a held backend is unreachable
The readiness gate tracked only the NATS link, so a worker whose backend
processes had died still answered /readyz with 200 and kept receiving
loads. One node did exactly that during an incident: it reported healthy
while its backend port refused connections, and every load routed to it
failed.
Readiness is now the NATS link and, for each backend process the worker
believes it is running, a short dial of its recorded address. A worker
holding no backends stays ready, because idle is a healthy state.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): keep a starting backend out of the readiness dial set
A backend process is inserted into the supervisor map with its gRPC
address already recorded, but the address refuses connections until the
gRPC server binds, which the startup poll allows up to 30 seconds for and
which takes 10 to 15 seconds on a slow node. The new data-path readiness
probe dialled that address straight away, so a worker answered /readyz
with 503 for the whole of every cold backend start. The container
HEALTHCHECK absorbs that, but a Kubernetes readinessProbe at 10s does
not, and the worker would leave rotation each time it loaded a model.
The skip for a stopping process had no counterpart at the other end of
the lifecycle. Backend processes now carry a serving flag, set where the
startup health-check gate succeeds, and the probe dials only processes
that are serving and not yet stopping. backendStartStillValid becomes
markBackendServing: the check and the mark must share one lock hold, so
the flag can only ever land on the entry the key currently owns.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(distributed): export control-plane database health gauges
Four transactions wedged on a corrupt index held the vacuum horizon open
for 42 days. Nothing measured it, so the first symptom anyone saw was
models failing to load six weeks later, by which time a six-row table
had grown to 460 MB.
Export the oldest xmin age, the longest open transaction, and the dead
tuple ratio on the registry tables. The first is the number that would
have caught it: it sits near zero in health and was 21,002,291.
Sampling is scrape-driven behind a cache, and a failed sample reports
the last good values rather than failing the scrape, because these
gauges matter most when the database is already struggling.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): rate-limit failed control-plane database samples
The cache advanced its clock only on a successful sample, so once the
database started failing every scrape retried the query immediately.
That turned the cache off in the one regime it exists for: a retry
storm at scrape cadence aimed at a database already in trouble. A
catalog read that consistently exceeds the 5 second timeout also paid
that cost on every scrape, with all scrapes serialised behind the
sampler mutex.
Time every attempt rather than every success, so failures and timeouts
cost the same interval as good samples. Whether a good sample exists
moves to its own field, keeping the gauges absent until the first
success and holding the last good values through later failures.
Also note in the runbook that pg_stat_activity cannot see prepared
transactions or replication slot xmins, so a healthy-looking xmin age
does not by itself rule out a blocked horizon.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(distributed): pin that a failing database evicts nothing
Exercises the real distributed stack against a control-plane database that
refuses the router's slot lookup, and asserts the scheduler reports the
lookup it could not answer instead of falling through to eviction.
The failure is injected with privileges rather than a statement timeout. A
timeout set with ALTER DATABASE also breaks AutoMigrate, and it leaks into
every later spec in the suite unless it is reset, so the spec would end up
testing the migration rather than the scheduler. Instead the spec creates a
dedicated login role, points a second gorm handle at it, and revokes that
role's SELECT on node_models.replica_index. This has to be a separate role:
the test container's owner is a PostgreSQL superuser, and superusers bypass
every privilege check, so revoking from CURRENT_USER is recorded and then
ignored.
The revoke is scoped to one column on purpose. Revoking the whole table
would also blind node selection, which runs first and has a guard of its
own, so the scheduler would never reach the slot lookup this spec is about.
Leaving every other column readable lets selection succeed and lands the
refusal exactly on NextFreeReplicaIndex, which plucks replica_index. The
grant is restored from BeforeEach via DeferCleanup, so a failing assertion
or a panic cannot hand the next spec a role that cannot read.
Reverting the eviction guard fails this spec, which is the point of it: the
router then reports "no replica slot on keeper and eviction failed" for an
error that was never evidence the node was full. The surviving-row
assertions are secondary under this injection, because the eviction path
reads whole node_models rows and the same revoke blinds it too; a comment
in the spec says so, so nobody mistakes them for the load-bearing ones.
Also documents why the vector store and the control plane must not share a
database: the removable-tuple cutoff is per database, not per table, so one
transaction left open anywhere stops autovacuum reclaiming the node
registry, and a six-row table bloats into hundreds of megabytes. The note
names LOCALAI_AUTH_DATABASE_URL and LOCALAI_AGENT_POOL_DATABASE_URL as the
two knobs that must differ, and the localai_control_plane_oldest_xmin_age
gauge as the way to see it coming.
grep for StaleNodeThreshold and HealthCheckInterval in
core/config/runtime_settings_registry.go returns no matches: the
distributed duration knobs are not exposed as runtime settings, so the new
heartbeat checkpoint interval follows them and needs no registry entry.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): close the review gaps in the heartbeat and health path
The stale-node threshold moved from 60 seconds to 5 minutes in this branch
because checkpointing makes last_heartbeat up to one checkpoint interval
behind by design. Two things were left inconsistent with that. NewHealthMonitor
still fell back to a hardcoded 60 seconds when handed a zero threshold, so any
future caller that stopped passing the configured value would mark every
healthy, beating node offline on every cycle. And the threshold itself had a
flag-name constant but no AppOption, no CLI field and no env binding, so an
operator who widened --node-heartbeat-checkpoint had no way to widen the
threshold to match. The fallback now tracks config.DefaultStaleNodeThreshold,
and --stale-node-threshold / LOCALAI_STALE_NODE_THRESHOLD is wired the same
way its sibling is.
Heartbeat suppression compared the RAW reported free VRAM against the
snapshot, but the column persists capAvailable(raw, ceiling). On any node with
a VRAM budget set, whose actual free VRAM oscillates above that ceiling, every
beat looked material while the persisted value never moved: suppression was
defeated on exactly the nodes an operator had configured, and the write
amplification this branch exists to remove came straight back there. The
comparison and the snapshot now both hold the capped figure, so they measure
the same quantity as the column.
Fixing that needs the ceiling, and reading it cost a SELECT on every beat,
including suppressed ones. The skip decision therefore moved ahead of the
updates map and now reuses the ceiling cached on the last durable write, while
the write path still re-reads it before capping anything. A ceiling that
changed inside the checkpoint window can cost one extra or one late write; it
cannot persist a wrong figure. A suppressed beat now costs no query at all.
Also: the operations section now says to grant pg_read_all_stats to the
LocalAI role, because PostgreSQL blanks backend_xmin and xact_start for
sessions owned by other roles, and the transaction that wedged the horizon in
the incident was a co-located vector store connecting as a different role, so
without the grant the new gauge sees only our own sessions. The compose
healthcheck comment now describes readiness covering the backend data path,
and the control-plane gauge registration records the otel.SetMeterProvider
ordering it depends on.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): resolve the gauge's table names through gorm
The dead-tuple gauge queried pg_stat_user_tables against a hardcoded list
of three table names. Those three do not agree on where their name comes
from: BackendNode and NodeModel take gorm's default pluralisation, while
GalleryOperationRecord overrides TableName, and gallery_operations
already had a constant of its own that the list duplicated.
A literal list keeps compiling after any of that moves, and the query
then matches nothing. The failure is silent and it points the wrong way:
a dead-tuple ratio that matched no rows reports the same numbers as a
cluster with no bloat, so the gauge would look healthiest exactly when it
had stopped working.
Ask gorm what each model is stored as instead, which follows a TableName
override and the default pluralisation alike. A spec pins that the
override really is consulted: naive pluralisation of the type would give
gallery_operation_records, so the resolution cannot quietly stop asking
the model.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
OpenAI GA clients send multipart or raw SDP requests. They expect a bare
SDP answer. LocalAI only accepted its legacy JSON envelope, so signaling
failed before media setup.
Keep the JSON contract for existing clients. Accept both GA request
shapes and choose the matching response format.
Assisted-by: Codex:gpt-5
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Node placement and replica rules could only name a model, so an operator
who pinned "llama3" to the GPU tier had to rewrite the rule whenever a
different model took over that job. An alias already gives a stable name
for whichever model serves it, and a rule on that name makes it a
deployment slot: repoint the alias and the placement follows.
A rule keeps the name the operator chose. Reads resolve that name through
the config loader to the model the rule governs, so the reconciler counts,
schedules and trims replicas of the target, and the router finds an
alias-keyed rule from the target it is already routing. An alias that
resolves to nothing governs nothing loadable, so the reconciler skips it
and the write paths refuse it.
A replica is shared by every name that resolves to it, so only one rule
can decide where it runs. The REST and MCP write paths reject a rule whose
target another rule already governs. A pair that arrives some other way,
such as a seed file or an alias repointed onto a model that already has a
rule, resolves in favour of the rule named after the model itself and then
the oldest, and the rest are listed as shadowed.
The eviction guard is the exception: it matches rules to replicas in raw
SQL inside a locking transaction and cannot resolve an alias. It reads a
stored target that the reconciler refreshes each tick, and falls back to
the rule's own name when that target is empty.
Assisted-by: Claude:claude-opus-5 golangci-lint eslint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
