A worker needs no inbound port: it dials GET /api/cluster/connect, the
connection becomes one multiplexed yamux session, and the frontend opens a
stream on it per request. This adds the endpoint that accepts that dial and
attaches it to the tunnel registry.
The dial is authenticated against the NODE's own stored token hash rather than
the deployment's registration token. That is the mechanism, not yet the
isolation, since a worker still registers by presenting the shared token; what
it rules out is the shortcut of comparing against the configured value, which
would have to be unpicked the day workers get their own secrets.
Every refusal happens BEFORE the WebSocket upgrade, so a dialer reads an HTTP
status rather than a handshake error. The route is registered in every
deployment, single-binary ones included, which is what puts it in front of the
route-coverage test that holds that rule in place; with no node registry it
refuses every dial, and tells a credentialed one the frontend has no cluster
rather than that its token is wrong.
A lookup that FAILED is answered as a failure. Reporting a database that could
not be read as "unauthorized" would send a worker re-registering, throwing away
the identity its tunnel and loaded models are keyed by.
Wires the tunnel registry in core/application/distributed.go and hands it to
the membership loop. Without that call the re-claim after a replica is reaped
had no production caller and could never run.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Opus 5 [claude-code]
Tasks 1 to 5 built an instances table, a splice, both halves of a peer link
and an epoch fence, and nothing in the tree called any of it: no replica
registered, no route was mounted, no sweeper ran. Proving phase 1 end to
end therefore had to start by wiring it.
A frontend in distributed mode now publishes the address its peers dial,
heartbeats it, and sweeps replicas that stopped answering along with the
connection rows they owned, in one pass so the two can never disagree about
who is alive. It serves the peer link and owns the sessions peers dial in,
refusing streams on them until phase 2 installs a relay: a session nobody
accepts on does not fail a peer's Open, it hangs it.
The address is the one peers use, not the one the process binds, and it is
derived from the route to PostgreSQL. That derivation only holds while the
database is remote, so LOCALAI_DISTRIBUTED_ADVERTISE_ADDR sets it
explicitly and a replica that can determine neither warns and keeps
serving rather than failing to start.
Three e2e scenarios run against real local-ai processes, real PostgreSQL
and real dials: replicas publish addresses that can actually be connected
to; a sibling opens a stream over the peer link and is refused without the
cluster token; and a killed replica is reported unreachable, never absent,
loses the claim it held, and takes no worker with it. Each was verified by
mutation: eight injected defects, each failing the scenario that claims to
catch it.
Also moves RegisterClusterRoutes to core/http/routes beside every other
registrar, folds AutoMigrate and the epoch sequence into one
cluster.Migrate, and turns the peer route's auth-coverage spec into a real
assertion: it drives the request through the actual auth middleware
instead of comparing two string constants, which the old spec would have
passed even with the exemption deleted.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Use a regular (non-editable) pip/uv install so the package lands in the
venv site-packages. An editable finder records the builder source path,
which breaks after the backend is copied out of the image (#9162).
Adds a regression test (scripts/build/vllm-omni-install_test.sh) that
verifies imports survive relocation with a regular install and fail with
an editable install.
Supersedes #12040 (DCO not signed by contributor).
Assisted-by: MAKI:regolo/glm5.2
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Show partial layer offload and CPU expert placement for llama-cpp.
Correct the documented gpu_layers default to match the backend config.
Refs #10557
Assisted-by: Codex:gpt-6
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Expose a negative_prompt string parameter on the image endpoints,
matching Stable Diffusion WebUI / vLLM-Omni conventions. When both the
negative_prompt parameter and a '|'-suffixed negative prompt in the
main prompt are present, they are joined with a comma so callers can
keep a global negative prompt in negative_prompt and add per-image
negative tags after '|'.
Assisted-by: Pi: DeepSeek V4 Pro
Signed-off-by: Fedor Zuev <Fedor.Zuev@gmail.com>
Consolidates all 25 pending gallery bot PRs into a single merge to resolve
the conflict cascade — every PR branched from a different point in master
and they all touch gallery/index.yaml, so merging them individually was
blocked by constant conflicts.
Changes:
- gallery/index.yaml: +739 lines (new model entries and fixes)
- docs/content/features/model-gallery.md: +116 lines (new model docs)
- docs/content/features/audio-cpp.md: +12 lines (Sortformer checksum fix)
Entry count: 1597 -> 1890 (293 new entries, no duplicates, YAML validated).
Supersedes: #11986#11992#11994#11996#11999#12002#12017#12019#12021#12025#12027#12029#12032#12036#12037#12038#12041#12042#12043#12047#12050#12064#12065#12066#12118
Assisted-by: MAKI:regolo/glm5.2
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vulkan): preserve host ICD discovery for packaged backends
Add bundled Mesa manifests through VK_ADD_DRIVER_FILES instead of replacing the system driver list. Merge inherited and model-specific additive paths while preserving explicit operator overrides, with regression coverage.
Assisted-by: Codex:gpt-5 golangci-lint
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(3d): add Kimodo CPU and Vulkan animation backend
Introduce a distinct animation capability and model-described 3D operations, with a typed /3d/animate API, RPC transport, distributed media staging, permissions, and tracing.
Add a persistent kimodo.cpp adapter, skeleton GLB export, CPU/Vulkan packages, model and backend galleries, importer support, CI builds, and documentation. Adapt Studio inputs to each model and provide real-time skeleton playback, seeking, and history.
Cover backend validation, packaging, API behavior, importer inventories, distributed staging, and Studio workflows. Validate real-model CPU/Vulkan generation and deploy the integration to the local QA instance.
Assisted-by: Codex:gpt-5 golangci-lint
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(kimodocpp): adopt monolithic encoders and resident inference
Update upstream for resident weights, packed execution paths, and cached motion graphs. Default to all 32 text layers while retaining configurable streaming and legacy bundle support.
Use monolithic Q8_0 encoders by default and offer all six published quantizations through the gallery and importer. Refresh pinned hashes, tests, and documentation; remove the obsolete thread patch and ensure cached source checkouts follow the upstream pin.
Validated CPU and Vulkan generation, lower-bit streaming, gallery/importer suites, packaging, lint, and cold/warm Studio generation on localai-dev.
Assisted-by: Codex:gpt-5 golangci-lint
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
The node restructure hid backend logs and split related controls
across inconsistent layouts.
Restore contextual log actions and align the detail page with the
fleet dashboard. Make multi-node selection clear and accessible.
Assisted-by: Codex:gpt-5 Playwright ESLint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(credentials): parse and match download credential rules
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): keep secrets out of parse errors and tighten URL matching
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(credentials): resolve secrets lazily and authenticate HTTP per hop
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): redact secrets in nested and store formatting
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(credentials): add registry keychain and oras credential adapters
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): match repository rules for Docker Hub in the oras adapter
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(downloads): authenticate HTTP downloads and gallery reads from the credentials store
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(oci): authenticate registry pulls, resumes, blobs and cosign from the credentials store
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(cli): load download credentials from --credentials-file
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(credentials): correct the local-network registry rules
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): keep secrets out of match and YAML parse errors
A match that fails to parse is no longer quoted in the Parse error, since
it may be a URL with a token in it. Userinfo is detected before the scheme
check, so ftp://user:token@host is refused as userinfo, and a match with a
query string or fragment is refused because it can never apply and a query
string is where signed URLs carry their token. Every YAML decode error is
now redacted, not only type errors: quoted scalars such as a secret under a
mismatched !!int tag are replaced and unquoted map keys are cut off.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(downloads): make auth errors name the real cause and never retry unresolved secrets
AuthError now appends its cause, so a registry's DENIED or UNAUTHORIZED
detail reaches the operator. HTTP auth errors print only the status text in
place of the cause, because the downloader builds that cause from the
requested URL, which can carry a signed query string. Registry pulls say
that docker config credentials were tried too, and a download that carried
a caller-provided credential (WithBearerToken, or an explicit authorization
on gallery reads) reports that credential as rejected instead of blaming
the store.
The Range probe for a leftover partial file now returns an unresolved
secret as a permanent error, like the download request already did.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): keep oras pulls anonymous on a broken docker helper and close bodies
When docker config names a credsStore helper that cannot run, the oras
credential func now logs at debug and returns no credential, so public
pulls keep working as they did before the adapter existed. The transport
closes the request body when a rule's secret cannot be resolved, as the
RoundTripper contract requires. The redirect spec now uses a custom header
rule on the origin, which net/http would not strip on its own, to prove the
transport does not carry credentials to the next hop.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(oci): cover FetchImageBlob authentication against a private registry
FetchImageBlob now has a spec that pulls a layer blob by digest from a
basic-auth registry through the oras credential adapter, and one that shows
the same fetch fails when no rule matches. oras only speaks HTTPS here, so
the registry serves TLS and the spec points http.DefaultTransport, which
retry.DefaultClient falls back to, at the test server's client for its
duration instead of adding a transport seam to production code.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(credentials): document auth error wording, ollama manifests and registry tokens
The errors section now lists the registry and provided-credential messages
and says the server's reason is appended. ollama:// manifests are fetched
without credentials, so only blob downloads use the file. GHCR, Docker Hub
and Quay need basic auth with the token as password, and match rules must
not carry a query string or fragment. The backend gallery docs and the
container troubleshooting section now point to the private sources page.
Assisted-by: Claude:claude-opus-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(credentials): document trusted file path
The credentials path comes from operator configuration. Mark the file
read with a scoped G304 explanation to resolve the gosec false positive.
Assisted-by: Codex:gpt-6 gosec
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(cli): benchmark configured text models
Compare model aliases through a running LocalAI server. Report full
request latency and server-reported throughput with raw JSON samples.
Keep warmups separate and fail before writing results on request errors.
Assisted-by: Codex:GPT-6
* fix(cli): satisfy benchmark error checks
Explicitly discard errors from buffered report writes, HTTP response cleanup, and test server writes to pass errcheck without changing behavior.
Assisted-by: Codex:gpt-6 golangci-lint
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The Ministral 3 14B Reasoning entry inherits a Mistral 0.3 prompt and
JSON parser. Its name-first tool calls can therefore reach clients as
plain text.
Use the embedded template and llama.cpp's native tool parser. Document
migration for installed configurations, which gallery updates do not
rewrite.
Assisted-by: Codex:gpt-6
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* [gallery] feat: add Orukeet to the existing NeMo speech backend
Assisted-by: Codex:gpt-6
Signed-off-by: Nathan Roll <nathan@oruk.ai>
* docs(nemo): remove model-specific instructions
Keep the backend guide focused on model families, as requested by mudler.
Remove the gallery limitation instead of restoring an outdated claim.
Assisted-by: Codex:GPT-6
Signed-off-by: Nathan Roll <nathan@oruk.ai>
---------
Signed-off-by: Nathan Roll <nathan@oruk.ai>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
grpc::ServerWriter::Write() returns false once the peer is gone, and
PredictStream ignored that result at every call site. The handler kept
pulling decoded tokens and writing them into a dead stream, so the
llama.cpp slot stayed busy until the generation ended on its own terms.
A model configured with max_tokens 0 and a large context ends on its own
terms only at the context limit. On a 35B model at ~41 t/s a 120k context
is about fifty minutes, and a slot held that long is a slot every other
request for that model queues behind. Two abandoned requests were enough
to make a node with free VRAM and a healthy control plane serve nothing:
new requests timed out waiting for a slot, each timeout abandoned another
generation, and the node fell further behind the longer it ran.
Track the peer instead. The first failed write retires it for good, since
a stream never recovers, and the RPC's own cancellation flag folds into
the same predicate so the loop has one condition to test. Returning early
is what frees the slot: ~server_response_reader() posts
SERVER_TASK_TYPE_CANCEL for whatever is still decoding.
TTSStream already checked Write(); this brings PredictStream in line.
Cancellation stays cooperative and is checked between decoded results, so
a batch already in flight may finish before the request stops.
Assisted-by: Claude:claude-opus-5
pinned: true was only honoured by the per-node watchdog. Every distributed
eviction path was pinned-blind: the router's LRU eviction (EvictLRU,
evictLRUAndFreeNode) and the replica reconciler's idle scale-down would
happily unload a pinned model — and since eviction is gated on
in_flight = 0, a pinned model became eviction-eligible the instant each
response completed. Under capacity pressure that surfaces as the backend
being freed immediately after every request (#11101).
Wire the model config loader into the router and reconciler through a new
PinnedModelResolver seam (mirroring ConcurrencyConflictResolver):
- EvictLRU passes the pinned set into FindLRUModel's query so the
next-oldest unpinned model is selected instead of the attempt failing
- evictLRUAndFreeNode filters pinned models inside its locked selection
- scaleDownIdle skips pinned models entirely: trimming to the floor still
means requests beyond the survivor's capacity pay a cold reload
Deliberate teardown (admin unload, model delete, node drain) intentionally
still applies to pinned models, as does dead-row reaping (state correction,
not eviction).
Regression specs verified to fail with the exclusion disabled.
Addresses the cluster-side eviction gap in #11101
Assisted-by: Claude Code:claude-fable-5 [Claude Code]
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
backend.stop was the one lifecycle subject a worker never answered. The
controller published and returned nil as soon as the local publish
succeeded, so a stop that killed nothing, and a stop that failed
outright, were indistinguishable from one that worked.
The unload endpoint calls model.unload and then StopBackend. Only the
first is acknowledged, so the endpoint answered 200 while the backend
kept running and held its VRAM, and its own "backend stop failed" branch
could never run. The worker logged the failure and nobody saw it.
Give the subject a reply. The worker now enumerates the process keys it
terminated and reports any per-process error, so StopBackend fails when
the stop failed. Resolving to nothing stays a success: stopping a backend
that is not running leaves the caller in the state it asked for, and
eviction paths stop already-gone models routinely. The empty list is what
says nothing matched, and ReportsStoppedProcesses is what makes that
emptiness trustworthy, the same way BackendDeleteReply handles it.
A worker built before this reply still receives the request and still
stops the backend, it only stays silent, so a timeout degrades to the old
assumption rather than failing every stop on a fleet mid-upgrade. Only
silence degrades: a transport error is still reported, because
UnloadRemoteModel skips its registry cleanup for a node it could not
reach and needs to keep hearing about that.
Assisted-by: Claude:claude-opus-5 golangci-lint
Adds FunASR/SenseVoice as a Python backend for speech-to-text with
support for CPU, CUDA 12/13, ROCm, Intel SYCL, L4T, and Apple MPS.
Co-authored-by: xingyifeng <xingyifeng@users.noreply.github.com>
Saved profiles previously resolved to one audio path and transcript, so
cloning backends could not use several examples of one personality.
Store ordered audio and transcript pairs while preserving the legacy
first-reference fields. Fish Speech and audio.cpp receive all pairs,
including on distributed workers. Other backends retain their
single-reference behavior.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Let studio users guide speech delivery for backends that support request instructions. Blank guidance stays out of requests and media history.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Clients cannot discover the named voices that an installed TTS model accepts without consulting backend-specific documentation. Expose voice metadata through the audio API and let custom model configs declare their own catalog.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The pinned audio.cpp revision supports HIP, but LocalAI neither builds a ROCm image nor accepts its backend option. AMD hosts therefore fall back to the CPU image.
Build and publish the HIP variant, connect it to AMD capability selection, and accept both upstream HIP names.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Expose diffusers audio pipelines through the existing sound-generation RPC. AudioLDM2 can now return PCM WAV output from the model gallery without a separate backend.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a NATS event contract and exact-residency provider for backend KV cache reports. Keep guessed request observations as the default routing source while maintaining the reported index for future producers.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a Darwin-only MLX-Video backend for LTX-2 and converted Wan checkpoints, expose it through the existing video API, and wire packaging, discovery, tests, docs, and an example.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Document how to connect LocalAI to the upstream RKLLM OpenAI-compatible server through cloud-proxy, including supported SoCs and current limitations.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Prefer an explicitly configured Triton assembler, otherwise use the executable ptxas from CUDA_HOME so torch.compile can target GPU architectures newer than Triton bundled tooling.
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Broadcast forced-disturb events and successful scale-up resets so every frontend shares the same rolling autoscale signal. Deduplicate NATS echoes, expose an origin-only Prometheus counter, and document cluster behavior.
Closes#10083
Assisted-by: Codex:gpt-5
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Route prefix-cache replica selection through composable filters, weighted scorers, and a replaceable picker. Preserve the existing load guard and deterministic selection policy while exposing per-model scorer weights through scheduling configuration.
Assisted-by: Codex:gpt-5 [go-vet]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(auth): bypass API-key auth for CORS preflight (OPTIONS) requests
When API-key auth is enabled, a browser making a cross-origin API call
first sends an OPTIONS CORS preflight, which cannot carry credentials by
HTTP spec. The auth middleware is registered (app.go:324) before the CORS
middleware (app.go:337-347), so the preflight hit auth first and returned
401 before the CORS middleware could answer it, blocking the actual call.
Bypass auth for OPTIONS so the request reaches the CORS middleware, which
answers the preflight with 200 + headers. Real API requests (GET/POST/etc.)
still require auth. Regression test added (red on master, green on branch).
Refs #4576
Signed-off-by: supermario_leo <leo.stack@outlook.com>
* fix(auth): exempt CORS preflights via publicRouteRegistry instead of middleware bypass
Route the global OPTIONS exemption through publicRouteRegistry
(OPTIONS on every path, replacing the OPTIONS-under-/api/auth/ rule
it subsumes) instead of a hardcoded method check inside Middleware,
so "which requests skip auth" has one mechanism. Preflights now flow
through the same authenticate-then-public-rules path as other public
routes, which also lets a credentialed OPTIONS request keep its user
context.
Update the route-coverage allowlist and the near-prefix lookalike
table for the new semantics (OPTIONS is public on every path by
design; near-prefix privacy stays pinned by the non-OPTIONS entries),
and fix the authentication docs' exempt-route enumeration, which
still described OPTIONS as an /api/auth/-only exemption.
Signed-off-by: supermario_leo <leo.stack@outlook.com>
---------
Signed-off-by: supermario_leo <leo.stack@outlook.com>
