Testing the progress-based cold-load deadline on the live cluster surfaced a
false positive. The stall window observed UPLOAD bytes only, but the staging
path has a phase that does real work while moving zero upload bytes: the
resumable-upload verify phase.
When a shard is already present on the worker from an earlier attempt, the
frontend HEADs it, hashes the local copy to confirm it matches, and skips the
transfer. Staging a 70 GB model with 56 GB already staged:
17:27:34 INFO Upload skipped (file already exists with matching hash) ...
17:28:20 INFO Upload skipped (file already exists with matching hash) ...
17:29:07 INFO Upload skipped (file already exists with matching hash) ...
... six-plus consecutive minutes, no bytes uploaded at all
~45s per skipped ~4 GB shard. That is correct and desirable - it is what makes
resume work - but it was indistinguishable from a stall. At 45s per shard it
sits inside the 5m window, so the run in flight was fine; the problem is the
600 GB scale this machinery exists to enable, where one shard can plausibly hash
for longer than the window. The guard would then fire during verification of a
transfer that is working perfectly.
Verified mechanism: probeExisting() HEADs the worker and then calls
downloader.CalculateSHA(). The staging progress callback is only consulted
inside doUpload(), which the skip path never reaches, so observeLoadProgress was
called zero times for the whole verify phase.
Verification exposed a second, worse bug in the same path: CalculateSHA consults
no context at all. An expired cold load kept hashing to completion, compared the
hashes, and returned success - reporting a file as staged on a dead load. The
failure only surfaced on the NEXT file, whose HEAD died immediately. That is
exactly the shape of the red test here, which fails on shard 3.
Fix: hash in 1 MiB chunks via hashFileWithActivity(), ticking the cold-load
deadline per chunk and checking ctx per chunk. A successful HEAD also counts,
since a 200 with a content hash proves the worker is serving right now.
Counting hash progress does not make a dead transfer look alive: hashing is
bounded, terminating work proportional to file size, in probeExisting it runs
only after a HEAD proved the worker was up, and the 24h absolute cap still
bounds the whole hold. The alternative of simply widening the window was
rejected - it would reintroduce the size cliff this work removes.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
A 70 GB video checkpoint (longcat-video-avatar-1.5) could not be loaded on a
distributed cluster. The request failed with HTTP 500 after 1499.98s - exactly
the 25m00s cold-load ceiling - while staging was demonstrably healthy: 26 of 57
files and 39 GB transferred at a sustained ~26 MB/s, zero errors, no stalls. It
was not wedged, it was killed by a timer.
ModelLoadCeilingFor covers node selection, backend install, file staging and the
remote LoadModel. Install and load carry their own budgets; staging was covered
only by a FIXED 5-minute margin. But staging time is bytes over bandwidth, not a
constant: 70 GB at 26 MB/s needs ~45m against a 25m ceiling, so the failure is
deterministic for any sufficiently large model rather than a flake. Simply
raising the constant moves the cliff to the next model size - the deployment
target here is checkpoints of 600 GB and beyond.
The ceiling's real purpose is that "a wedged worker can never pin the lock
indefinitely". Progress, not elapsed time, is what distinguishes a wedged worker
from a large one. The hold is now a deadline that extends whenever the transfer
reports bytes and expires a 5-minute stall window after they stop:
- A large model transferring fine continues, for hours if needed.
- A worker that died mid-transfer still fails within the stall window.
Progress is observed at byte level on the transfer itself, via the existing
staging progress callback. Per-file completion would be too coarse - a single
600 GB shard would be indistinguishable from a stall for hours. The observation
point is back-pressured by the socket, so it reflects the network rather than
local disk reads. Observation is coarsened to one timer touch per stall/20 so
the per-read callback stays cheap.
The base budget (unchanged, and still derived from the install and load
timeouts) continues to cover the steps that report no progress, so
LOCALAI_NATS_MODEL_LOAD_TIMEOUT keeps working exactly as before. An absolute
cap of 24h bounds the hold even while progress keeps arriving, so a peer
trickling bytes forever cannot pin the advisory lock; 600 GB at the measured
26 MB/s is ~6.5h, so the cap sits far above any legitimate transfer.
Also fixes the incoherent layering the same error exposed: the resumable upload
carried a 1h retry budget nested inside the 25m ceiling, so the inner budget was
unreachable and the message still blamed it ("failed after 1 attempts within
1h0m0s budget") while the 25m parent was the actual killer. The upload now
adopts the caller's deadline when there is one, and applies its fixed budget
only when nothing above bounded it - which also stops a fixed 1h from
reintroducing the size cliff under the now-extendable parent.
This is the successor to #10968, where a hardcoded 5-minute LoadModel gRPC
timeout was replaced by this derived ceiling. Fixing the inner timeout exposed
the outer ceiling as the new binding constraint.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
The "Bump Backend dependencies" workflow has failed every night for over
ten days. Four upstreams — ced.cpp, moss-transcribe.cpp, voice-detect.cpp
and rf-detr.cpp — moved from the mudler org to localai-org, so the GitHub
API answers 301 for the old slugs. ced.cpp additionally renamed its
default branch to main.
bump_deps.sh fetched without -L or -f and never checked the response, so
the redirect's JSON body was passed straight to sed, which died with
"unterminated `s' command". The loud failure was luck: an error body
without slashes would have been substituted into the Makefile as the new
pin, silently corrupting the version and shipping it in a bump PR.
Point the matrix at the new slugs and branch, and harden the script so a
bad response can never reach sed: follow redirects, fail on HTTP errors,
and require a bare 40-hex SHA before rewriting anything. Also refresh the
now-stale repository URLs in the backend Makefiles, test scripts,
backend/index.yaml and the docs.
Verified all 25 matrix entries resolve to a commit SHA and that the four
previously-failing jobs run end to end against the real API.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Picks up mudler/go-processmanager#7, which closes the check-then-act
window in Run's "already started" guard. Concurrent Run calls on one
handle could each observe a nil p.proc, each start a process and each
launch a monitor goroutine; every monitor does `defer close(p.done)`
against a channel created once in New, so the second monitor to see its
process exit panicked on close of a closed channel. The same window
raced on p.proc itself.
LocalAI does not hit this today: the only New/Run pair
(pkg/model/process.go:178-191) runs a freshly created handle, and
core/services/worker/supervisor.go only ever calls Stop on handles it
holds. The bump is defensive, and keeps the dependency from drifting
further behind a fix in the process lifecycle we rely on.
No exported signature changes upstream; ErrProcessAlreadyRun is
additive and keeps the historical "command already started" prefix, so
any caller matching on that text is unaffected. Nothing in this repo
matches it.
Verified: go build ./core/... ./pkg/... clean; go vet clean;
go test -race ./pkg/model/ ./core/services/worker/ both ok.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Bash] [Edit]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
ci(gallery): propose variant groupings for review on a schedule
A gallery entry may declare `variants:`, references to other entries that
are alternative builds of the same weights, and auto-selection then installs
the best build for the host. Those families exist only because humans curated
them in two manual sweeps.
The gallery agent dedupes on the HuggingFace repo URL and picks one
quantization per model, so it never adds a second build of a repo it already
has, and consequently never creates a family and never joins one. A model
published across two repos lands as two unrelated standalone entries. The
grouping decays as the gallery grows and nothing notices.
Add a scheduled job, in the same shape as the checksum checker: compute
offline, edit the index textually, open a pull request against ci-forks. It
proposes and never decides. Grouping is a judgement call that has gone wrong
in both directions, so the value is catching drift and surfacing candidates
with their evidence.
Three grouping signals, taken from the manual sweeps: same name once
quantization markers are stripped, the `:` config-suffix convention, and the
same primary weight filename once quantization markers are stripped. The
third requires the same upstream repository. Excluding auxiliary files is not
enough on its own: bert-embeddings, an ultravox audio model and a roleplay
finetune all declare a primary file called llama-3.2-1b-instruct-q4_k_m.gguf,
and grouping on that is the same error that linked four wan-2.1 entries and
Z-Image-Turbo to qwen3-4b.
Add gallery/variant-exclusions.yaml, a checked-in rejection ledger. A job
that re-proposes declined candidates every night becomes noise and gets
ignored. Declining a proposal is one flow-mapping line a reviewer adds inside
the proposal pull request itself. Seeded with the six -abliterated pairs whose
base is also in the gallery, the mistral-small multimodal pair, the whisper-1
alias, the kokoros language set, and the recurring finetune tokens. qat and
apex are deliberately not on it: they are quantization techniques.
Proposals refuse to nest, to let two parents claim one target, to target an
entry that installs nothing, and to touch a merge anchor, since a variants key
added to an anchor is inherited by every merging child. The anchor refusal
names every entry that would inherit, which is the worklist a human needs.
Run against the pre-sweep gallery, the job rediscovers 12 of the 19 groupings
the second manual sweep made, with no false positives. The rest it reports as
refusals or ledger declines rather than missing silently.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Every writer used to stage into the same `.artifacts/.partial/<cacheKey>`.
That was safe only because the artifact lock held: two writers that both
believed they had it opened the same blob with O_APPEND and interleaved
their bytes into one file, while the resume probe read the other writer's
in-flight size. SHA verification caught the damage only after both had
burned the entire download.
#10986 restored the lock's precondition on CIFS but left the dependency in
place. Suffix the staging tree with a writer identity drawn once per
process run, so concurrent writers cannot corrupt each other whatever the
lock does. The lock stops being a correctness dependency and becomes a
pure efficiency optimisation: a lock failure now costs a duplicated
download, not a corrupted one.
Commit stays an atomic rename. The loser of a commit race reconciles onto
the winner's tree instead of surfacing a bare ENOTEMPTY for work that
actually succeeded, since the artifact is content-addressed and both trees
hold the same verified bytes.
Writer-unique staging means a crashed writer's tree is no longer
overwritten by its successor, so two things are added to keep it from
becoming a disk leak and a resume regression:
- A sweep reclaims trees whose contents have been untouched for 24h,
matching the window the startup reaper already uses for stray *.partial
files. It reads the newest mtime anywhere inside the tree, because
writing a blob never touches an ancestor, and refuses any name this
package did not write. A live download writes continuously, and the
downloader's stall watchdog aborts a silent one long before it could
look abandoned.
- Adoption lets a restarted process claim a dead predecessor's tree for
the same artifact and resume from its bytes, which a tens-of-gigabytes
repo depends on. The claim is an atomic rename, so racing adopters
cannot both win. It runs only under the artifact lock - which is
released exactly when the owning process dies - and only on a tree idle
for 5 minutes as a second line of defence for when the lock does not
exclude.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(worker): give the worker a real health endpoint (#10987)
The image bakes in a single HEALTHCHECK that curls
http://localhost:8080/readyz, but the same image also runs `local-ai
worker`, which serves HTTP on the gRPC base port minus one and never
binds 8080. Every worker container was therefore permanently
`unhealthy` (43 consecutive failures observed on a production node),
which is worse than having no healthcheck: a genuinely broken worker and
a perfectly good one both report `unhealthy`, so the signal carries no
information and orchestration that keys on it misbehaves.
The worker already served /readyz on that port via the file-transfer
server, but as a constant 200 — it only proved the listener was bound,
which is precisely the failure mode at issue. Readiness now tracks the
live NATS connection: all of a worker's actual work (backend lifecycle
events, inference dispatch, file staging) arrives over NATS, so a worker
whose link is dead is up and useless. Registration is already implied,
since the server only starts after registration succeeds.
This reports something the controller cannot already see. The node
registry's status/last_heartbeat is fed by an HTTP heartbeat to the
frontend, a different network path from NATS — a worker can keep
heartbeating while its NATS connection is dead and still look healthy in
the registry. /healthz stays a constant 200: liveness must not follow
readiness, or a NATS blip becomes a cluster-wide restart storm.
The HEALTHCHECK is now a script that derives its endpoint from the mode
the container is actually running plus the env vars that configure the
bind address, so a frontend moved off 8080 with LOCALAI_ADDRESS (broken
the same way) and a worker on a non-default base port are both probed
correctly. Modes with no HTTP surface (agent-worker, one-shot commands)
report healthy rather than false-unhealthy. HEALTHCHECK_ENDPOINT remains
as an explicit override, so the workaround shipped in
docker-compose.distributed.yaml keeps working; both overrides in that
file are now unnecessary and have been removed.
Also fixes the latent --start-period gap. Since #10949 a frontend's
startup preload materializes HuggingFace artifacts before the HTTP
server binds (31 GB observed on a live cluster), so a healthy replica
can legitimately fail probes for a long time. --start-period is Docker's
knob for exactly this: failures inside it leave the container `starting`
instead of burning retries, and it ends early on the first success, so a
generous 60m costs a fast-starting container nothing. --timeout drops
from 10m to 10s — it is a per-probe deadline, and a localhost curl that
has not answered in 10s is itself the fault being detected.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
In distributed mode the Open Responses store is process-local: a
sync.OnceValue over a map behind an RWMutex. With several frontend
replicas behind a round-robin load balancer, every request that lands on
a replica other than the creator misses.
Measured on a live 2-replica cluster (#10993): the same response id
returns 200 on the creating replica and 404 on its peer, and a cancel on
the peer returns 404 without ever invoking CancelFunc, so generation runs
to completion on the other replica while the caller is told the response
does not exist. previous_response_id chaining fails through the same
lookup.
Split the state by what can actually cross a process boundary:
- Replicated: response metadata (request, response resource, owner,
expiry, stream/background flags) via syncstate.SyncedMap, the same
component finetune, quantization and agent tasks already use. A local
miss in Get/FindItem now falls back to it and returns a read-only
remote view, so polling and chaining resolve on any replica.
- Delegated: cancellation. context.CancelFunc is a function pointer and
exists only in the creating process, so a cancel that lands elsewhere
is broadcast on responses.<id>.cancel and applied by whichever replica
holds the function. The broadcast is fire-and-forget rather than
request/reply: if the owner crashed or was scaled down nobody answers,
and the handler must not block on a reply that will never come. The
replicated status moves to cancelled either way, which is truthful,
since a dead owner's generation died with its process.
- Refused: streaming resume. The resume buffer is a byte log plus a live
notification channel and cannot be replicated without shipping every
token over the bus. A resume that reaches the wrong replica now returns
HTTP 409 naming the owning replica via the new ErrResponseNotLocal,
instead of an empty event list that looks like a finished stream. It is
deliberately distinct from ErrOffsetLost, which means the owner's
buffer evicted the requested events.
Standalone deployments never call EnableDistributed and keep exactly the
previous process-local behaviour.
Fixes#10993
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
gallery: remove duplicated entries and lint against them coming back
gallery/index.yaml declared eight names twice: deepseek-r1-distill-llama-8b,
llama3.2-3b-enigma, qwen3-asr-0.6b, qwen3-asr-1.7b, qwopus-glm-18b-merged,
voice-en-us-kathleen-low, whisper-large-q5_0 and whisper-small-q5_1.
FindGalleryElement resolves a reference by returning the first match, so in
every pair the second copy was unreachable: it could not be installed, could
not be selected as a variant target, and could not be corrected, because any
edit to it went to a copy nobody reads. A reference to such a name is also
ambiguous to anything reasoning over the catalog, which is why the variant
proposal job refuses to propose against them.
Each pair was compared both as parsed entries and as raw text, and all eight
were byte-identical apart from position. None of the sixteen blocks defines a
YAML anchor or pulls one in with a merge key, so nothing was reachable only
through a deleted block, and no entry named a removed copy as a variant target.
Removing the second copy of each therefore changes no behaviour: the parsed set
loses exactly eight entries and every surviving entry is field-for-field
unchanged.
The removal is textual, by line range, so the diff is pure deletions rather than
a reflow of forty thousand lines.
checkNoDuplicateEntryNames is the rule that keeps them out, added beside the
existing gallery invariants and reporting in the same style.
checkSingleVariantClaim closes the adjacent gap in the same place. VariantParents
resolves a build claimed by two parents by taking the first in gallery order and
calls that deterministic "for a gallery the linter would reject", but nothing
rejected it: the invariant was held by curation alone. Now it is a rule, and the
comment describes something real. No target is doubly claimed today, so the rule
is green on arrival.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
flock(2) on CIFS/SMB returns EACCES when another client holds the lock:
the kernel maps STATUS_LOCK_NOT_GRANTED and STATUS_FILE_LOCK_CONFLICT to
-EACCES and never produces EWOULDBLOCK on that path. gofrs/flock only
recognises EWOULDBLOCK as contention, so TryLockContext returned a bare
"permission denied" and Ensure aborted. Both replicas then fell back to
legacy loading, which makes the worker download the whole repo in-band
inside LoadModel and blow the remote-load deadline.
Replace TryLockContext with an explicit wait loop over a new Locker
interface, classifying EWOULDBLOCK/EAGAIN/EACCES/EBUSY as contention.
EACCES is ambiguous at the syscall boundary but not here: the lock file
is already open O_CREATE|O_RDWR, so a real permission problem would have
failed the open with an *fs.PathError, and flock(2) documents no EACCES
on Linux at all. The wait is bounded (DefaultLockWait, overridable via
WithLockWait), so even a misclassification degrades to a delay. On
timeout the committed result is re-checked before reporting the new
ErrLockContended, so a peer that finished the work still wins.
Locker also exists so the contention path is testable without a network
filesystem: nothing in CI can make flock(2) return EACCES on demand.
Raise the fallback to error for a managedArtifactBackends backend, via a
shared config.LogArtifactFallback used by both call sites. For those
backends the legacy path is not graceful degradation, and the operator
otherwise sees only a timeout with no causal link. The fallback stays
non-fatal.
Drop the os.Chmod(layout.Lock, 0o600) after acquisition: flock.New
already creates the file 0600, and the chmod was gratuitous risk on a
nounix mount that ignores modes.
Fixes#10981
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
#10970 gave the four PredictOptions RPCs a model-identity check so a
backend reached through a stale distributed route rejects the request
instead of answering from whatever model it holds (#10952). Every other
modality shares that exposure: the route is cached by host:port, a worker
can recycle a stopped backend's port for another model's backend, and a
liveness-only probe cannot tell a stale row from a valid one.
Extends the same mechanism to the 21 remaining request messages that reach
a backend through the router, using the pattern #10970 established rather
than a parallel one:
- proto: ModelIdentity on each modality request message.
- controller: populated from ModelConfig.Model at the call site that also
builds ModelOptions, so load-time and request-time values are equal by
construction.
- backends: one generic guard in pkg/grpc/server.go (27 Go backends), the
method set in backend/python/common (36 Python backends), llama-cpp
(AudioTranscription/Stream, Rerank, Score) and privacy-filter
(TokenClassify).
- reconcile already drops the stale row on IsModelMismatch; no change.
TTSRequest and SoundGenerationRequest get a SEPARATE ModelIdentity field
rather than reusing their existing `model`: FileStagingClient rewrites
`model` to a worker-local path, so comparing it would reject valid
requests in exactly the configuration this guards.
AudioEncode/AudioDecode are deliberately left unguarded: the opus codec
backend is loaded from a literal rather than a ModelConfig, so no value
carries the equality guarantee the comparison depends on. The four
bidirectional stream RPCs are out of scope; they bypass reconcile.
Empty means skip on both sides, so an old controller, an old backend, and
the bare request structs in tests/e2e-backends all keep working.
Assisted-by: Claude Code:claude-opus-4-8 [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
PR #10975 taught the backend matrix filter about shared build inputs, but
left two paths that still rebuild nothing.
Go backends link code from the main tree. `go list -deps ./backend/go/...`
resolves to exactly six pkg subtrees (audio, grpc incl. base/grpcerrors/proto,
httpclient, sound, store, utils), identical for GOOS/GOARCH in
{linux,darwin} x {amd64,arm64}. Editing any of them changes the shipped
binary, but they sit outside every backend directory so the prefix match
never saw them. Enumerating those six rather than taking all of pkg/ is the
point: all of pkg/ changes in ~8.6% of commits, these six in 2.0% — the same
order as the already accepted scripts/build/ rule (1.9%). Blast radius
199/417 Linux, 26/56 Darwin; the ~21 core-server-only pkg subtrees still
rebuild nothing, and neither do _test.go files.
.github/backend-matrix.yml was excluded wholesale because matching its path
would rebuild all 417 entries on every new-backend PR. That hid a real hole:
editing an existing entry's base-image, build-type or cuda version changes
the image it produces while touching no file the filter can see. Since the
change is within a structured file, compare it against the base revision and
rebuild only the entries whose fields actually differ — 1 entry for a
base-image edit, 0 for a comment or whitespace edit, and all 417 only when
the previous revision cannot be resolved. This also closes a third hole: a
new matrix entry for an existing backend (a new CUDA variant, say) touches
nothing under that backend's directory and previously rebuilt nothing.
changed-backends.js fetches the base revision via the contents API, and only
when the changed-file list actually names the matrix file, so the common path
costs no extra request.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(http): make /readyz reflect startup readiness instead of always 200
/readyz was registered as a static handler returning 200 unconditionally,
so it carried no information: it was green whenever it could be reached at
all. Readiness could not distinguish "serving" from "still starting", and
any future change that started the HTTP listener earlier would silently
turn the probe into a lie.
Track startup completion on the Application (atomic flag, flipped at the
very end of New() on the success path only) and have the readiness handler
consult it per request, returning 503 with a small JSON body while startup
is in progress. A nil readiness source fails open so embedders keep the
historical behaviour.
/healthz is deliberately left readiness-independent. Liveness and readiness
answer different questions, and failing liveness during a long preload makes
an orchestrator restart the pod mid-download so the preload never finishes.
This matters because since #10949 the startup preload materializes
HuggingFace artifacts for managed backends: tens of GB for a large model
(31 GB observed on a live cluster). Both probes stay in quietPaths and stay
exempt from auth.
Note the listener is still started only after New() returns, so today the
not-ready state is not observable over HTTP. Moving the listener earlier is
a separate, deliberate decision and is not made here.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(gitignore): anchor the mock-backend pattern so its source dir is traversable
The bare `mock-backend` pattern matched the *directory*
tests/e2e/mock-backend/, not just the binary built into it. Git will not
descend into an ignored directory even for tracked files, so
`git add tests/e2e/mock-backend/main.go` required -f. This was hit while
working on #10970.
Anchor it to the artifact's full path. The built binary stays ignored (it is
also covered by tests/e2e/mock-backend/.gitignore) while the source directory
becomes traversable again.
Verified with `git check-ignore -v`: a new source file under
tests/e2e/mock-backend/ is no longer ignored, and the binary produced by
`make build-mock-backend` still is.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(coverage): raise the coverage ratchet from 48.5% to 54.2%
The committed baseline had drifted well below reality: it still read 48.5%
while a full instrumented run measures 54.2%. A stale-low baseline makes the
gate meaningless — coverage could regress by more than 5 percentage points
and still pass.
Raising a ratchet is a deliberate act, not something to fold into an
unrelated fix, so it gets its own commit. The headroom was earned by tests
landed in #10946, #10947, #10948, #10949, #10956, #10967, #10968, #10970 and
#10975.
Measured with `make test-coverage` on this branch (the same instrumented run
`make test-coverage-baseline` uses: ginkgo over ./pkg and ./core plus the
in-process tests/e2e suite, --covermode=atomic, --coverpkg over core/... and
pkg/..., generated protobuf excluded). The run completed with exit 0 and zero
spec failures; the total was then written with the exact command the
test-coverage-baseline target uses:
go tool cover -func=coverage/coverage.out \
| awk '/^total:/{gsub(/%/,"",$NF); print $NF}' > coverage-baseline.txt
Verified afterwards with scripts/coverage-check.sh, which reports OK.
Note the measured figure includes the readiness specs added earlier on this
branch, so it is a demonstrated floor rather than an estimate.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Two independent defects in the download path, both surfaced by the same
incident (#10982).
A failed `io.Copy` was always reported as "failed to write file", because
`io.Copy` folds read and write errors into a single return value. A
peer-cancelled HTTP/2 stream therefore presented as a filesystem failure and
sent an investigation after mount permissions while the disk was healthy. The
source is now wrapped in a recorder so the error names the side that actually
broke, and a write failure names the `.partial` it was writing rather than the
final blob path.
The plan runner returned on the first task error with no retry, so one
transient stream cancel discarded every file already downloaded in a
multi-file materialization. The `.partial` resume machinery already existed
but was unreachable because nothing made a second attempt. Transient failures
(dropped transport, mid-stream read failure, stall, 5xx, 429) are now retried
with bounded exponential backoff and resume from the partial; permanent ones
(4xx, checksum mismatch, local write failure, caller cancellation) fail
immediately.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
feat(gallery): group 21 more model families under variants
Second variant-grouping sweep. QAT and APEX builds are now treated as
quantization techniques rather than distinct weights, per maintainer
ruling, so they group with their base entry instead of standing alone.
Adds 15 new families: quantization and serving-config pairs for
llama-3.2-1b/3b-instruct, dolphin-2.9-llama3-8b, phi-2-chat, ideogram-4,
meta-llama-3.1-8b-instruct, omnivoice-cpp and qwen3-tts-cpp; the gemma-3
4b/12b/27b QAT families; and three cross-backend pairs (silero-vad plus
its sherpa-onnx build, and the vibevoice TTS and ASR builds shared
between the vibevoice-cpp and crispasr backends). The cross-backend
pairs are the first entries that meaningfully exercise engine-preference
ranking during auto-selection.
Restructures four gemma-4 families (31b-it, 26b-a4b-it, e2b-it, e4b-it).
Those bare entries were skipped by the first sweep, which left a QAT
build as parent by default. The bare entry is what installs when nothing
else fits, so it reclaims the parent slot and the former parent becomes
a plain target. Every pre-existing relationship is preserved; nothing is
dropped and nothing nests. gemma-4-12b-it has no bare entry, so it is
left as is.
qwen3-tts-cpp is a YAML anchor with nine merging children, so the five
children that did not already override variants get an explicit empty
list to stop them inheriting the parent's.
Abliterated builds stay excluded: abliteration edits the weights to
remove refusal behaviour, which makes them a different model rather than
another build of the same one.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(system): expose raw detected capability for model meta resolution
Model meta gallery entries express hardware fallback through candidate
ordering rather than a capability map, so they need the undecorated
detected capability string without Capability's default/cpu fallback
chain.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactor(system): drop duplicate capability accessor, cover DetectedCapability
ReportedCapability was added with a body identical to the existing
DetectedCapability. Keep one accessor and move the specs onto it, since
DetectedCapability had no direct coverage of its no-fallback behavior.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): parse IEC binary size suffixes (KiB..PiB)
ParseSizeString accepted only SI suffixes, so a "20GiB" floor was rejected
outright. Model and VRAM sizes are conventionally quoted in IEC units, and
silently reading GiB as GB would understate a floor by about 7%.
Purely additive: these inputs previously returned an unknown-suffix error.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): add Candidate type for meta model entries
Candidate is one option in a meta entry's ordered variant list. It names a
concrete gallery entry and declares when that entry suits the host.
EffectiveMinVRAM resolves the VRAM floor, letting an authored min_vram win
over a nightly-inferred one. An unparseable floor errors instead of being
treated as absent: swallowing a typo would turn a constrained candidate into
an unconstrained one and select a too-large variant rather than fail loudly.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): add hardware-aware model variant resolver
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): allow gallery model entries to declare variant candidates
A gallery entry with a non-empty candidates list is a meta entry: it names
an ordered list of concrete entries and resolves to the first one the host
can satisfy, instead of describing model files directly.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): resolve meta model entries to hardware-appropriate variants at install
Meta gallery entries carry an ordered candidate list; at install time the
first candidate the host satisfies is resolved and its payload installed
under the meta's name, so the model keeps a stable name regardless of which
variant backs it. The resolution is recorded in the installed gallery
config so a reinstall honors a prior pin and operators can see the backing
variant.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): key meta pin recall on the installed name and detach resolved entries
Six review findings on the meta-entry install path.
Pin recall was keyed on the gallery entry name while applyModel writes the
record under the install name (req.Name when supplied), so a meta installed
under a custom name with a pin lost that pin on reinstall and was silently
re-resolved onto a different variant, possibly swapping its backend. Compute
the install name with applyModel's own precedence before the recall.
ResolveMetaModel returned a shallow struct copy, so the resolved entry's
Overrides aliased the gallery entry's map and the install path's in-place
mergo merge wrote the caller's request into the shared catalog. Detach
Overrides, ConfigFile, AdditionalFiles, URLs and Tags. Not exploitable today
only because this path re-unmarshals the gallery per call, which is a
property nobody should have to rely on.
Also: overlay the meta's name onto the persisted config for meta installs so
the gallery file no longer records the variant's name; move the pinned-VRAM
warning below the variant validation so a pin naming a nonexistent entry does
not warn about VRAM before failing for an unrelated reason; and stop seeding
config.URLs in the config_file branch, which duplicated every declared URL.
Add seven network-free specs driving InstallModelFromGallery with a meta
entry: variant payload wins over the meta's legacy url fallback, the
resolution record round-trips to disk, a pin is recorded and honored on
reinstall including under a custom install name, and the resolved entry does
not alias the gallery's maps.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): deep-copy meta overrides and make two specs functional
ResolveMetaModel detached the resolved entry's Overrides and ConfigFile with
maps.Clone, which only copies the top level. Gallery overrides are nested in
practice (parameters.model is near-universal) and the install path merges the
caller's request with mergo.WithOverride, which recurses into nested maps and
overwrites them in place, so the gallery entry's own inner maps were still
reachable and still got rewritten by the last caller to install.
Copy both maps all the way down instead, recursing through the container shapes
a YAML decoder produces. ConfigFile is not mutated on the install path today,
but it carries the same kind of nested payload and leaving it shallowly cloned
would invite the bug back.
Also fix two specs that passed whether or not their target fix was present:
- "does not write the caller's overrides back into the gallery entry" re-read
the catalog from disk, which re-unmarshals fresh structs and so cannot
observe in-memory aliasing. It now asserts against the in-memory gallery
entry and drives the real mergo merge.
- "round-trips the resolution record to disk under the meta's name" asserted a
name that is already correct in the config_file branch. It now drives the url
branch via a file:// fixture, where the meta-name overlay actually applies.
Both were verified red by reverting their fix.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(gallery): lint meta model entry invariants in index.yaml
Adds Ginkgo specs that parse the shipped gallery/index.yaml and enforce
the invariants that keep meta entries safe: a legacy url fallback equal
to the final candidate's url, references only to existing non-meta
entries, a min_vram floor on every candidate but the last-resort one,
a capability drawn only from the vocabulary the system can report, and
descending VRAM floors within a capability group.
The capability check is the only compensating control for a typo there.
Candidate matching is a case-sensitive exact comparison against
SystemState.DetectedCapability(), so an unknown value never matches and
falls through silently instead of erroring. The vocabulary therefore
mirrors the raw return set of getSystemCapabilities(), which notably
excludes "cpu": that is a fallback key inside Capability(capMap) on the
meta backend path, never a reported capability. A CPU-only host reports
"default".
These pass vacuously until the pilot meta entry lands; the guard is
intentionally in place before the thing it guards.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(gallery): close coverage gaps in the meta entry lint
The ordering invariant grouped candidates by capability and asserted floors
descend within a group. A candidate with an EMPTY capability matches every
host, so it does not belong in its own group: it dominates every later
candidate whose floor is at or above its own, across capability groups.
Track a running minimum floor over the unconditional candidates instead,
which subsumes the old same-group check for the empty capability.
Every spec skipped non-meta entries, so with zero meta entries in the index
all five bodies were no-ops. Aligning GalleryModel.IsMeta() with
GalleryBackend.IsMeta(), whose semantics are deliberately opposite, would
have made all of them pass while checking nothing. Extract each invariant
into a helper over a slice of entries returning the violations it finds, and
cover those helpers with synthetic fixtures so the logic stays tested at zero
meta entries. The index-driven specs are now a thin application of already
proven logic.
Also assert the index parses non-empty, report every violation in one run
rather than aborting on the first, and parse the index once for the suite.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* ci(gallery): add nightly denormalization of meta model candidates
Fills the read-only backend, quantization and inferred_min_vram fields on
meta gallery candidates and opens a PR, modeled on the existing
checksum_checker job. Computing these needs network access, so it happens
nightly rather than at install time.
An authored min_vram is never modified: a human who measured a real load
knows more than a pre-download estimate does.
The index is rewritten via yaml.Node rather than a document round-trip. A
full round-trip reflows all ~26k lines of gallery/index.yaml, which would
bury the computed values and make the nightly PR unreviewable. The rewrite
touches only the three derived keys, so authored styling survives and a run
that computes nothing leaves the file untouched.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ci): keep the gallery denormalize diff reviewable and self-healing
The nightly denormalization job edits YAML nodes instead of round-tripping
structs so its PR stays small enough for a human to review, but the write
path undid that: yaml.Marshal re-encoded the node tree at yaml.v3's default
4-space indent and dropped the leading document marker, reflowing roughly
6000 lines around the handful of real changes. Encode through
yaml.NewEncoder at the index's authored 2-space indent and restore the
header. A write that changes three fields now changes three lines.
Stale inferred_min_vram values were also never cleared. Both skip paths
(an authored min_vram is present, or the candidate is the last resort)
returned before touching the field, so a candidate that gained a floor or
became the last resort after a reorder kept an inferred value that
EffectiveMinVRAM reported as a real constraint, failing the meta lint with
no way for the job to self-heal. Clear the field before both skips.
The workflow discarded a whole night's work on any single failure: the
program exits 1 when a candidate cannot be estimated, which aborted the job
before the PR step, so one unreachable candidate blocked every other
refresh indefinitely. Capture the status, open the PR with what was
computed, mark the PR body as partial, and fail the run afterwards so the
problem still surfaces.
Also preserve the index's existing file mode instead of forcing 0644, and
drop the redundant //go:build ignore tag, since Go already skips dot
directories and the sibling modelslist.go carries no tag.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): add nanbeige4.1-3b meta entry with hardware-resolved variants
Adds the first real meta entry to the gallery index. It resolves to the
Q8_0 build on hosts with at least 6GiB of VRAM and to the Q4_K_M build
everywhere else, installing either payload under the stable name
nanbeige4.1-3b.
The entry carries a url equal to its final candidate's url. LocalAI
releases that predate candidates support parse the index non-strictly
and drop the key silently, so without that url they would list the entry
and install nothing. A regression spec parses the index the way those
releases do and asserts every meta entry stays installable for them.
Also teaches core/schema/gallery-model.schema.json about candidates. The
schema sets additionalProperties: false at the top level, so an author
following CONTRIBUTING.md and adding the yaml-language-server comment
would otherwise get a validation error on this entry.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): make candidate entries complete, installable entries
Reworks hardware-resolved gallery variants after a design pivot. There is no
longer a separate "meta" entry kind. A gallery entry is a normal, complete
entry that may additionally carry candidates:, a list of hardware-gated
upgrades over itself, and the entry is itself the last-resort candidate.
The previous design relied on a bare url: as the fallback for LocalAI releases
that predate candidates support. That fallback is empty in practice: none of
the 80 gallery/*.yaml files carry a top-level files:, and 1216 of 1281 index
entries carry their payload in the index entry itself, so a url alone yields a
config template with nothing to download. Since every released LocalAI reads
gallery/index.yaml live from master, merging a payload-less entry would have
shown every existing user a model that installs to a broken state. Making the
entry its own base candidate removes the problem at the root: old clients drop
the candidates key and install the entry exactly as they do today.
Resolution order is now explicit pin, then capability plus VRAM over the
declared upgrades, then the entry itself. The entry ALWAYS installs: when its
own min_vram or capability is unmet the installer warns and installs it
anyway, because there is nothing below it and refusing would make the gallery
behave worse the newer the client is. A pin naming the entry's own name is
valid and is how an operator declines an upgrade.
IsMeta() becomes HasCandidates(), ResolveMetaModel becomes ResolveVariant, and
the persisted meta_name record key becomes entry_name. GalleryBackend.IsMeta()
is a separate concept and is untouched.
The lint drops the three rules the pivot makes wrong (url equality with the
final candidate, no inline payload, unconstrained final candidate) and gains
one: the entry's own floor must sit strictly below every candidate's, since a
base that outranks a candidate makes that candidate unreachable.
The pilot entry is now the existing nanbeige4.1-3b-q4, which gains a 2GiB
floor of its own and a single 6GiB upgrade to nanbeige4.1-3b-q8, replacing the
separate nanbeige4.1-3b entry added in d0d441bb4.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): select model variants by hardware fit, not authored order
Gallery entries could already carry a list of alternatives, but selection was
an authored, ordered, first-match policy: every candidate declared a
`capability` string and the VRAM floors had to descend in a hand-tuned order.
That pushed hardware knowledge onto whoever edits the gallery and made ordering
load-bearing, so a reordered list silently changed what users installed.
None of it was necessary. SystemState.IsBackendCompatible already derives
hardware support from a backend name alone: it knows MLX and metal are
Darwin-only, CUDA is NVIDIA-only, ROCm AMD-only, SYCL Intel-only. Selection can
read that instead of asking authors to restate it.
Authoring is now just a list of names:
- name: qwen3.6-27b
min_memory: 4GiB
variants:
- model: qwen3.6-27b-mlx-8bit
- model: qwen3.6-27b-gguf-q8
min_memory: 28GiB
and all the intelligence moved into the selector. Given a host it drops the
variants whose backend cannot run here, drops those whose known memory
requirement exceeds what the host has, and takes the LARGEST of what is left,
because a bigger footprint is a higher quality quantization of the same model.
A variant of unknown size is kept, since nothing proves it does not fit, but it
ranks last so a proven fit always beats a guess. An explicit pin still wins
outright, and if nothing survives the entry installs its own payload: the base
always installs, this never refuses.
Available memory is VRAM when a GPU was detected and system RAM otherwise, read
through xsysinfo so a cgroup limit is honored and a container gets its own
limit rather than the node's RAM.
Capability disappears entirely, from the types, the schema and the lint. VRAM
and RAM collapse into one `min_memory`, because a model's footprint is roughly
the same wherever it lives and one figure is compared against whichever applies.
The lint rules about ordering, the capability vocabulary and floor
relationships are deleted with the hazards they described; what remains is that
every variant names an entry that exists and does not itself declare variants,
plus that any memory figure actually parses.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* gallery: size model variants with a live probe, drop the nightly denormalizer
Selection needs each variant's size to decide whether it fits and to rank
largest-first. That figure was written into the index by a nightly job, which
made the gallery carry a derived value that could drift from the entry it was
derived from. Derive it at install time instead.
pkg/vram already sizes a model without downloading it, and the gallery UI
already uses it: a remote GGUF header range-fetch, then an HTTP HEAD for the
content length, then any declared size:. It caches its results, so reuse it
rather than writing a second probing path.
A probe failure must never fail an install, so an unprobeable variant is
treated as unknown: it survives the memory filter, because nothing proves it
does not fit, and it ranks last, so a known-good fit always beats a guess. If
every probe fails, selection still terminates on the base entry.
The probe is injected through ResolveEnv rather than called directly, for the
same reason the backend compatibility check is: specs pin an exact size, or an
exact failure, without reaching the network.
With that in place three things are dead weight and go:
- The nightly job and the fields it populated. Variant.Backend was redundant
because the backend is resolved live from the referenced entry during
selection, and Quantization was display-only that nothing read.
- min_memory on the base entry. The base always installs and its floor could
only warn, so it could not change any outcome.
- The lint rules and schema entries for both.
min_memory on individual variants stays, as the override for when the probed
size is wrong. An authored figure now suppresses the probe entirely rather
than merely outranking it, so it costs no round trip.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): expose model variants for selection over API, CLI and MCP
A gallery entry may carry `variants:`, alternative builds of the same model.
Selection already worked at install time, but nothing could see what an entry
offered or ask for a specific build, so the feature was undrivable.
Listing: `GET /api/models` now reports `variants` and `auto_variant` for the
entries that declare variants. Each variant carries its resolved backend, its
measured size and whether it fits this host. `auto_variant` is what installing
without a choice would pick right now.
The new gallery.DescribeVariants runs the same variantOptions + SelectVariant
pass the installer runs, so the reported default cannot drift from what
installing actually does, and HostResolveEnv is extracted so both derive the
host and share pkg/vram's probe cache from one place.
Performance: an entry that declares no variants returns early without touching
the probe, so the ~1280 ordinary entries cost exactly what they cost before.
Selection: `variant` is accepted on POST /models/apply, as a query param on
POST /api/models/install/:id, on the gallery apply file/string request, as
`local-ai models install --variant`, and as a parameter on the install_model
MCP tool (both the httpapi and inproc clients). Empty means auto-select.
An unknown variant name now fails the install naming what was requested. This
closes a real hole: an entry declaring no variants short-circuits before
selection runs, so a requested variant was previously dropped silently and the
install reported success.
startup.InstallModels ends in a variadic model list, so install options could
not be appended to it; InstallModelsWithOptions is added alongside and
InstallModels delegates to it. No caller signature changed.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): drop the redundant variant min_memory field
Variant.MinMemory was an authored override for when the live probe misreads
a variant's footprint. It duplicated an existing field: probeEntryMemory
already passes the entry's declared size: into EstimateModelMultiContext,
whose cascade prefers that declared size over its own guesswork. Correcting
size: on the referenced entry fixes the figure for every consumer rather
than only for variant selection, so min_memory shadowed the right answer.
A variant is now nothing but a name. Its effective size is exactly the probe
result, and an unknown stays unknown: it survives the filter and ranks last.
EffectiveMemory loses its error return along with the field. The authored
string was the only thing that could fail to parse, so the error had no
remaining source and was propagating dead nil-checks through SelectVariant,
DescribeVariants and the pin warning.
Selection behaviour is unchanged. The specs covering probe-derived sizing,
ranking, filtering, the unknown-size path, pin recall, entry/variant
metadata split and deep-copy isolation all survive; the three install specs
that needed a definite size now declare it through the referenced entry's
own size:, which exercises the documented escape hatch directly.
gallery/index.yaml is untouched: no entry ever carried the key.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): rank the entry's own build against its variants
Variant selection pulled the declaring entry's own payload, the base, out
of the candidate set and consulted it only once every declared variant had
been rejected. Two real failures followed.
A variant whose size the probe cannot determine deliberately survives the
memory filter, because nothing proves it does not fit. As the only survivor
it then won outright on any host, however small: a 2GiB machine installed an
unmeasured variant in preference to the 4GiB build the entry itself ships,
with no warning. 241 of the 1280 current index entries carry no files and no
size, which is exactly that shape.
"Largest wins" also broke whenever the base was the largest. An author
writing a Q8 entry that offers a Q4 downgrade for small hosts, a natural
shape that nothing in the lint, schema or docs discourages, had the Q4
installed on every large host instead.
Make the base an ordinary participant. It is still exempt from both filters,
so selection always terminates on something installable, but it is now
ranked against the variants: a proven fit first and largest, then the base,
then any variant whose size nothing could measure. Both failures disappear
together. The base is probed for its size accordingly, which it was not
before, because an unsized base would lose every contest to an unmeasurable
variant.
FellBackToBase is kept but narrowed to "no declared variant survived",
rather than "the base was chosen", since the base now also wins on merit and
that is not worth warning about.
A recalled variant pin also became a permanent install failure. A pin the
caller supplies on this request must stay fatal, but one recalled from
._gallery_<name>.yaml can be invalidated by any later gallery edit, and
failing on it turned one rename into a model that could never be reinstalled
or upgraded again short of deleting a dotfile the user has never heard of.
A stale recalled pin is now dropped with a warning naming it, and selection
runs as if it had never been recorded.
Also drop the last textual reference to two abandoned designs from the
DetectedCapability comment, correct the documented variants JSON example,
which showed a memory_bytes of 0 that omitempty makes impossible, and remove
an em dash from the install skill.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): budget variant memory from RAM when a GPU reports no VRAM
Variant selection read its memory budget from VRAM whenever a GPU
capability was detected, and from system RAM only when none was. Apple
Silicon satisfies the first branch and fails the premise: arm64 macs report
the metal capability unconditionally, without probing anything, while
TotalAvailableVRAM has no discrete VRAM pool to find and returns zero. The
budget therefore came out as zero on every Mac.
Zero drops every variant carrying a known size, so the base build was
installed on all of them however much memory the machine had. The feature
was inert on the platform, and silently: falling back to the base is a
legitimate outcome, so nothing looked wrong.
Take VRAM only when it is actually a number, and fall back to RAM
otherwise. On a unified-memory host RAM is not an approximation of the
budget, it is the budget, since the GPU shares it. A discrete GPU whose
VRAM could not be read also lands on RAM, which overstates what the card
holds but understates nothing the host has; the previous zero understated
both.
An unreadable RAM figure still yields zero and still installs the base, so
a genuinely unknown host is not talked into a larger download.
This is what turned tests-apple red: "installs a fitting variant's payload
under the entry's own name" asserts on selection, and the runner resolved
to the base because its budget was zero. The added specs pin the branch
directly rather than relying on a macOS runner to notice again.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): add a model variant picker to the models gallery
PR #10943 shipped the server side: a gallery entry may declare `variants:`,
`GET /api/models` attaches `variants` and `auto_variant` to declaring
entries, and `POST /api/models/install/:id` accepts a `variant` query
parameter. Nothing in the UI consumed any of it, so the feature was not
reachable from the browser. This wires it up.
modelsApi.install takes an optional second argument and appends an encoded
`?variant=` only when one is given, so every existing call site keeps
sending exactly the request it sent before.
On the models table, an entry that declares variants gets a split button.
The primary Install still installs the auto-selected build, because auto is
the default and the point of the feature; the chevron opens a menu for a
deliberate override. It follows the Backends.jsx precedent: one shared
Popover re-anchored per row, rendering .action-menu items, which brings
Escape, outside-click and focus return along with it. An entry that
declares no variants renders exactly as it did before.
A variant that does not fit is dimmed but stays selectable, since the server
honors an explicit choice with a warning rather than refusing it.
memory_bytes is omitempty on the wire, so an absent key means the size is
unknown and never zero. A single helper guards both the menu and the detail
row, because formatBytes would otherwise render a falsy value as "0 B",
which reads as "needs nothing".
The expanded detail row gains a Variants section listing each build's
backend, size, whether it fits, which is the entry's own build, and which
one auto-selection would pick, built from the existing DetailRow helper and
.badge classes.
Eight Playwright specs cover the picker, including that plain Install sends
no variant parameter and that choosing one sends it. One pre-existing
assertion was scoped with .first(): the Variants section legitimately adds
more llama-cpp badges to the detail row, which tripped strict mode.
UI line coverage 49.42% -> 49.36% against a 40.0 baseline and 0.8pp
tolerance; branch coverage rose 72.04% -> 72.66%.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): describe model variants from a companion endpoint
Variant description probes each referenced entry's weight files over the
network: an HTTP HEAD plus a ranged GET, serial, five seconds per probe
with no aggregate deadline. Running it inline in GET /api/models made one
listing cost (entries x variants) round trips. The Manage page fetches
with items=9999, so at 200 declaring entries that is ~1000 serial probes,
minutes of a blocked handler and gigabytes of range traffic for a single
page load. Only one entry declares variants today, but the feature exists
so that many will.
Follow the precedent already set for VRAM estimates. The listing now
reports only has_variants, a length check on loaded metadata that touches
nothing, and GET /api/models/variants/:id returns the description for one
entry, mirroring estimate/:id in route shape, auth and error handling.
DescribeVariants itself is unchanged; only its caller moved.
The picker fetches lazily at the two points where a user asks to see
variants, opening the split-button menu and expanding the detail row, and
caches per entry for the page session. An entry declaring no variants
issues no request at all.
A spec counts real HTTP hits on the weight files, so it goes red if
description becomes reachable from the listing path again through any
caller.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): filter the model gallery to entries that declare variants
The gallery is heading towards showing parent entries and hiding the
individual builds they reference, so a user sees one row per model
rather than six quantizations of it.
Adoption is a single entry today, so defaulting to that would leave a
one-row gallery. This ships the migration-phase inverse instead: the
default is untouched, and a toggle narrows the list to only the entries
that declare variants. It previews the end state and changes nothing
until someone asks for it.
The filter is server-side, next to term/tag/backend/capability and above
the pagination arithmetic. The listing paginates at 9 items, so
narrowing on the client would leave totalPages and availableModels
describing the unfiltered set and hand the user empty pages. It selects
on HasVariants(), which reads already-loaded metadata, so it issues no
variant probes.
The parameter is named has_variants after the listing field it selects
on, and is compared against "true" like the other boolean query params
(all_users, save_checkpoint), so has_variants=false reads as absent.
With it omitted the response is byte-for-byte what it was before.
The control is the shared Toggle component, matching the fitsFilter
toggle already on this page: same wrapper class, same icon and label
shape, same localStorage persistence. Unlike fitsFilter it resets to
page 1 on change, which a server-side filter has to do.
Stacking the toggle with a tag or backend filter easily yields nothing
while one entry declares variants, so the empty state now names the
variants filter as the cause rather than leaving a user to conclude the
gallery is broken.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ui): render gallery model descriptions as Markdown
Gallery descriptions are Markdown, but the React UI dumped them raw, so a
model whose description opens with an ATX heading showed a literal
"# Qwen3.6-27B [](https://chat.qwen.ai)" in the list.
Full-description areas now render through renderMarkdown (marked +
DOMPurify), matching how Backends.jsx and the Manage detail panels already
handle the same content:
- Models.jsx expanded detail row
- VoiceLibrary.jsx voice detail header
The truncated one-line previews must not render block Markdown: a leading
"#" would become an <h1> and wreck the row height and rhythm. They get a new
stripMarkdown() helper instead, which reduces Markdown to a single line of
readable plain text. It is used for the cell text and for the title tooltip,
since a tooltip full of "[](url)" is no better than a cell full of it:
- Models.jsx gallery table description cell
- Manage.jsx model and backend resource-row descriptions
stripMarkdown walks marked's lexer output rather than running regexes over
the source, so what it strips is by construction what renderMarkdown would
have rendered, and it needs no new dependency. Output lands in JSX text
nodes, so React escapes it; no new dangerouslySetInnerHTML beyond the two
full-description sites, both of which run DOMPurify.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ui): strip Markdown from the backends table description cell
Commit b35d630cf fixed this for gallery models but left the Backends admin
page with the same asymmetry: its detail panel renders the description
through renderMarkdown, while the collapsed table row dumped the raw gallery
string into both the cell body and the title tooltip.
That is user-visible. 40 of the 949 entries in backend/index.yaml carry
Markdown - insightface uses inline code backticks, others use lists and
links - and backend descriptions also contain embedded newlines, so the
one-line cell showed literal syntax.
The cell now runs stripMarkdown over the description once and uses the
result for the text and the title, matching Models.jsx and the
ResourceRowDesc component in Manage.jsx. The '-' placeholder is preserved,
and now also fires when a description reduces to nothing after stripping.
The detail panel is untouched and no new dangerouslySetInnerHTML is
introduced: stripMarkdown output lands in a JSX text node, so React escapes
it.
Three Playwright specs cover it: a description with a heading, inline code
and a link renders as clean text with no literal syntax and no block
element in the cell, the title tooltip carries the same stripped text, and
a backend without a description still shows the placeholder.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* ui(models): polish the variant detail view and scope rendered Markdown
The gallery detail pane rendered every field through the same two-column
label/value row, including the description. Multi-paragraph prose in a value
cell ran eight rows tall at the top of the pane on a ~1200px measure, breaking
the grid's rhythm exactly where the eye enters. Move it into its own full-width
block above the table, capped at a 68ch measure, keeping the label.
Rendered Markdown had no scoped typography anywhere in the app, so a
description opening with `#` inherited the browser default 2em inside a 13px
surface while a `##` further down was indistinguishable from body text. Add a
reusable .markdown-body block mapping h1-h6, paragraphs, lists, links, code,
blockquotes, images and tables onto the existing type scale, and apply it to
every renderMarkdown() consumer: the models detail, the backends detail, both
Manage details and the voice library detail.
Rebalance the variants list so the name leads. Backend and size drop from
badge/secondary weight to muted metadata; the FITS badge goes entirely, since
it was true of nearly every row and so said nothing, while the variant that
does not fit keeps a warning badge and a dimmed name. AUTO-SELECTED stays
marked because it answers what a plain Install produces. Rows share the
parent's grid tracks via subgrid so name, backend, size and status line up
down the list instead of raggedly following name length.
Finally, make each variant row actionable. It looked like a list of choices
but was inert text, with per-variant install hidden behind the split-button
chevron elsewhere; each row is now a button onto the existing
handleInstall(modelId, variant) path, with hover, keyboard focus and a
disabled state while an install is in flight.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): collapse the listing to one row per model
The listing supported has_variants=true, which narrowed to entries that
DECLARE variants. With adoption at three entries that showed three rows,
which is useless; it was always a placeholder.
Replace it with the view that is actually useful: the deduplicated
gallery. Show every entry installable in its own right and nothing twice,
which means the parents plus every entry nobody references, and hide only
the builds another entry already offers as a variant, since those are
reachable through their parent.
The parameter is renamed to collapse_variants accordingly: the filter is
no longer a predicate on a row's own metadata but a view over the whole
gallery. Default stays off, so the response with the parameter absent is
unchanged.
VariantReferencedIDs never reports an entry that declares variants of its
own, so parents are always visible. That guarantees every hidden entry
has a visible entry offering it, and no chain can strand a row. Variant
resolution already refuses to install such a reference, but the listing
has to stay coherent in the presence of a gallery that has one rather
than silently swallowing entries. Self-references and dangling references
hide nothing.
The referenced set is computed over the whole gallery rather than over
what the other filters left, so an entry is hidden because a parent
offers it and never because of what the user searched for. The pass is
over metadata already in memory: it resolves nothing over the network and
triggers no variant description or size probe, so the listing's zero-probe
contract still holds.
The UI toggle keeps its behaviour (persistence, page reset, clear
filters) and becomes "One row per model", which says what the user gets.
Its localStorage key moves too, since the stored value meant a different
filter.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): show the collapsed model listing by default
The gallery listing is what a user reaches for to answer "what can I
install". Answering that with several rows for the same model, one per
build, makes the reader do the deduplication the collapsed view already
does, so the collapsed view is the one to land on.
The UI now asks for collapse_variants=true unless the toggle says
otherwise. The server default is deliberately untouched: a request with
the parameter absent still returns the full listing, because other API
clients depend on that response and collapsing it under them would be a
breaking change. Opting out omits the parameter rather than sending
false, so it asks for exactly the listing everyone else gets.
The stored preference changes vocabulary from '1'/'0' to 'on'/'off'. The
previous build wrote it from an effect that runs on mount, so a stored
'0' recorded that the page had been opened rather than that anyone chose
the expanded view, and honouring it would pin every earlier visitor to a
default they never picked. Only the new vocabulary counts as a choice;
a legacy '1' meant the collapsed view and is what the new default gives
anyway, so no earlier deliberate choice is lost.
Collapsing being the default also changes what the empty state may say
about it. An opted-into filter can be named as the cause of an empty
result; a default cannot, so the filters keep the top line and the
collapsed view drops to a hint below it, shown only once filters are
narrowing the set. For the same reason "Clear filters" now restores the
collapsed default instead of switching it off, and the toggle alone no
longer counts as a filter worth offering to clear.
The label stays "One row per model": it describes the view the user is
looking at rather than an action, so it reads the same whether it is
opted into or out of.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* gallery: group alternative builds of the same weights under variants
Sweep the gallery for entries that are alternative builds of the same
weights (different quantization, precision, or runtime format) and declare
them as variants of a single parent row, so the listing offers one row per
model instead of one row per quantization and the installer picks the
largest build that this host can actually run.
41 families over 95 entries, turning 54 entries into variants.
The parent is the bare-named entry wherever one exists, so nothing changes
about what any existing entry installs. Ranking already selects the largest
fitting build regardless of which entry is nominally the parent, so the
parent only decides the pathological case where nothing fits. For the ten
families that have no bare-named entry, the smallest build is the parent,
since that is the one that has to install when nothing fits.
Grouping was verified against the actual model filenames rather than the
entry names alone. Different parameter sizes, languages, finetunes, and
products that merely share a name prefix are left as separate rows: the
qwen3.6 APEX and pi-tune finetunes, the DFlash and MTP speculative-decoding
pairings, English-only versus multilingual Whisper, the QAT versus non-QAT
Gemma 4 weights, and the abliterated FLUX build are all distinct models.
Six parents define YAML anchors that other entries pull in with a merge key,
which would have handed their variants to every merging child. For the two
depth-anything anchors that would have made fourteen unrelated entries
advertise the base model's builds as their own. All 26 merging children
therefore carry an explicit empty variants list, which overrides the merged
key and is equivalent to the key being absent.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): rank model variants by host backend preference
Variant auto-selection filtered candidates by whether their backend can
run on the host, then ranked the survivors by size alone. The backend
never influenced the choice beyond that gate, so a Mac offered both an
MLX build and a llama.cpp build kept neither filtered and installed
whichever was larger, leaving the native accelerated runtime unused. The
same held for CUDA against CPU on NVIDIA and ROCm against Vulkan on AMD.
Rank by the host's backend preference between the fit tier and size: fit
stays a filter, preference decides among the builds the host can equally
hold, and size still separates builds on equally preferred runtimes.
The preference data stays in one declarative table in pkg/system, now
read by a prefix lookup instead of a switch, so adding a capability or
reordering one host's runtimes is a one-line edit and the gallery's
ranking code carries no per-backend branching. MLX joins the metal rule
ahead of metal itself, which is inert for the existing alias-resolution
consumer because no alias group holds a candidate named for mlx.
An unrecognised backend, an unrecognised capability and an absent
preference list all collapse to the previous size-only ordering rather
than erroring or dropping candidates.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): rank variants by engine name, not backend build tag
Variant auto-selection ranked candidates with
SystemState.BackendPreferenceTokens, but that function and the variant
ranker speak different vocabularies.
BackendPreferenceTokens returns BUILD TAGS ("cuda", "rocm", "sycl",
"vulkan", "metal", "cpu"). It exists to match installed backend build
directory names like "llama-cpp-cuda-12" during alias resolution in
ListSystemBackends. Variant ranking instead matches a gallery entry's
`backend:` value, which is an ENGINE NAME: "llama-cpp", "vllm",
"vllm-omni", "sglang", "mlx" and the rest. No engine name in
gallery/index.yaml contains "cuda", "rocm", "sycl" or "vulkan".
preferenceRank matches by substring, so on an NVIDIA host the tokens
[cuda, vulkan, cpu] matched neither "llama-cpp" nor "vllm", every
candidate scored identically and size alone decided. The NVIDIA, AMD,
Intel, darwin-x86 and vulkan rules were all inert. Only metal appeared
to work, and only because the token "mlx" happens to equal an engine
name. The mismatch does not error, it silently deletes the feature.
Separate the two vocabularies. backendBuildTagPreferenceRules keeps the
build tags and its original output for every capability, including
metal, whose "mlx" token is removed again; its alias-resolution consumer
is byte-identical to before. engineNamePreferenceRules is new, holds
engine names, and is read by the new EnginePreferenceTokens, which
HostResolveEnv wires into the renamed ResolveEnv.EnginePreference. Both
tables sit adjacent under one block comment naming each vocabulary and
each consumer, and share one lookup helper so their semantics cannot
drift.
On NVIDIA the order is vLLM, then SGLang, then llama-cpp: vLLM is the
throughput engine and a model published with a vLLM build is published
that way because that build is the one worth running. AMD and Intel get
the same order, since rocm and intel builds of both serving engines
ship. Metal prefers mlx over llama-cpp. Vulkan prefers llama-cpp, the
only LLM engine with a Vulkan build. darwin-x86 and unknown
capabilities are deliberately absent rather than guessed at, degrading
to the size-only ordering that predates preference.
preferenceRank stays generic and names no engine and no capability, so
adding a runtime remains a one-line table edit.
Specs pin the NVIDIA and metal rules through the live table and the real
HostResolveEnv wiring, so emptying the engine table or wiring the build
tag source back in both go red. A regression table asserts
BackendPreferenceTokens' original output per capability, and mirrored
locks assert neither table carries the other's vocabulary.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: record that variant selection ranks by engine before size
A gallery entry can now declare variants, and selection ranks the builds a
host can run by engine preference before size. Nothing told a contributor
adding a backend that engineNamePreferenceRules exists, so a new engine would
silently rank below every known one and lose to whatever build happened to be
larger on hosts where it should have won.
Document the step where a backend is added, warn against the sibling
backendBuildTagPreferenceRules table (build tags, not engine names: the wrong
table matches nothing, scores every candidate equally and disables the
preference without erroring), and index it from AGENTS.md.
Fix the authoring and user docs, which still claimed the largest surviving
build wins. An author grouping builds under one entry has to be able to
predict what a user gets, and size alone no longer decides it.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(cli,mcp): describe variant auto-selection as preference before size
The CLI flag help and the install_model tool schema both still said
auto-selection takes the largest build that runs. Ranking now puts engine
preference ahead of size, so on NVIDIA a vLLM build wins over a larger
llama.cpp one. An assistant reading the old schema would tell users the
wrong thing.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): prefer llama.cpp over GPU serving engines on hosts with no GPU
engineNamePreferenceRules had no row for the "default" capability, which
getSystemCapabilities() returns both when no GPU is detected and when a GPU
is present but under the 4 GiB VRAM floor. A missing row yields an empty
preference list, which preferenceRank reads as "score everything equally",
collapsing variant selection to size alone.
That would be harmless if the hardware filter dropped GPU serving engines on
such a host, but it does not. IsBackendCompatible derives support from the
engine NAME, and "vllm" and "sglang" contain none of the darwin, cuda, rocm
or sycl tokens it keys on, so they fall through to its closing "return true".
A vLLM variant therefore survives on a CPU-only box and wins whenever its
build is the larger of the two on offer: the machine installs vLLM in
preference to llama.cpp.
darwin-x86 had the identical hole. It was documented as a deliberate omission
because nothing accelerates on an Intel Mac, which is true about acceleration
and wrong about consequence: with every engine tied, download size decides.
Add rows for both putting llama-cpp first. The GPU engines are enumerated
behind it rather than left unmatched: an unmatched engine already ranks below
every listed one, so llama.cpp would win either way, but unmatched engines
also tie with each other and let size decide among them. Naming them fixes
that order. MLX is left off the darwin-x86 row on purpose so it ranks last,
since IsBackendCompatible admits darwin-tokened engines on that capability
even though MLX needs Apple silicon.
Preference orders survivors and never filters, so a model published only as a
vLLM build is still installed on a host with no GPU; there is a spec for it.
Surveyed every other value getSystemCapabilities() can return. nvidia, amd,
intel and vulkan have rows; the l4t and cuda-refined values reach the nvidia
row by prefix; "apple" and "" cannot reach the vendor fallthrough because the
darwin and no-GPU branches return earlier. default and darwin-x86 were the
only live holes.
BackendPreferenceTokens and its build-tag table are untouched, and
preferenceRank stays generic, naming no engine and no capability.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* gallery: prefer speculative-decoding builds when they fit
Rank serving features between engine preference and size, so a host that can
hold a DFlash or MTP build of a model's weights installs it instead of the
plain build. Both answer faster for the same output, so whenever one survives
the filters there is no reason to take the plain build.
Precedence is now fit, then engine, then serving feature, then size. Engine
outranks the feature deliberately: a serving feature makes the right engine
faster, it does not make a wrong engine right, so a plain vLLM build still
beats a DFlash llama.cpp build on NVIDIA. Fit outranks both, and a drafter
pairing is strictly larger than the plain build, so the existing size filter
drops it on a host too small for it before this axis is consulted.
The order lives in a third preference table in pkg/system, alongside the build
tag and engine name tables. It is the odd one of the three: not keyed by
capability, because no hardware prefers a plain build over an equivalent
faster one, and matched against whole segments of a gallery ENTRY NAME rather
than as a substring of a backend value. Nothing on a gallery entry declares a
serving feature, and tags are not a usable substitute: gemma-4-e2b-it:sglang-mtp
carries an mtp tag while ornith-1.0-9b-mtp and qwen3.6-27b-nvfp4-mtp carry
none. Entry names are author-supplied free text, unlike the closed engine
vocabulary, so a short marker can turn up inside an unrelated word and whole
segment matching is what keeps smtp-assistant from ranking as an MTP build.
The block comment over the tables now documents all three together and states
what each is matched against; the ranking code names no feature, so adding one
stays a one-line edit to the table.
29c49203b rejected these entries as serving configurations rather than
alternative builds of the same weights. The definition is now "alternative ways
to serve the same model", which includes them, so regroup 14 entries under 12
parents. Judged by the files each entry points at: the qwen3.6, qwen3.5, qwen3
and deepseek pairings are the base GGUF plus a drafter, the gemma-4 QAT MTP
entries are the same QAT weights at a different quantization plus an MTP
drafter, and the two sglang MTP entries describe themselves as the same model
served with speculative decoding. Left separate: qwen3.6-27b-mtp-pi-tune, a
finetune with its own weights, and every entry whose base model LocalAI does
not ship as its own row, which is the whole Qwopus line plus gemmable-4-12b-mtp,
mimo-7b-mtp:sglang and qwen3.5-4b-dflash.
None of the twelve parents defines a YAML anchor, so no variants key can leak
through a merge key and no empty override was needed this time. The index was
edited by line insertion only.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test: check env restore errors in capability and variant specs
errcheck flagged ten unchecked os.Setenv and os.Unsetenv returns in the
specs added while the pre-commit hook was being skipped. Restoring an env
var is exactly the place a silent failure leaks state into the next spec,
so assert on it rather than suppressing the linter.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): make the mtp tag authoritative for serving-feature ranking
Variant auto-selection ranks survivors by fit, then engine, then serving
feature, then size. The serving-feature lookup read only whole alphanumeric
segments of a variant's entry name, because tags were inconsistent: every
dflash entry carried a dflash tag, but only 7 of 20 MTP entries carried an
mtp tag.
Tag the 13 untagged MTP entries, then teach the lookup to read tags as well
as names. A tag is now the authoritative signal and is compared whole and
case-insensitively, which is safe precisely because a tag is a deliberate
declaration rather than free text: there is no word-inside-a-word failure
mode, so the segment splitting the name half needs is unnecessary there.
The name check stays as a fallback rather than being replaced. Switching to
tags only would have regressed the six already-grouped entries on the day it
shipped, and would depend on tagging discipline that does not exist yet.
The lookup still names no feature, so adding one remains a one-line edit to
servingFeaturePreferenceTokens.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): make a declared tag the sole serving-feature signal
Variant auto-selection ranks survivors by fit, then engine, then serving
feature, then size. The serving-feature lookup recognised a speculative build
by either a declared tag or a whole segment of its entry name. Drop the name
half: a tag is now the only signal.
A name is author-supplied free text and a naming convention is not a contract,
so reading a marker out of one infers a capability nobody declared. The gallery
already had the failure in it: the four NVFP4 entries name MTP-bearing weights
while setting no option that enables speculative decoding, and being live
variants they were winning the feature axis without answering any faster.
overrides.options was considered as the replacement and rejected. It carries
spec_type:draft-mtp / spec_type:draft-dflash, which is what actually turns the
feature on, but that spelling is llama.cpp's config vocabulary: ds4 spells the
same feature mtp_path and sglang spells it speculative_algorithm in a
referenced config. Keying a cross-backend ranking decision on one backend's
option syntax would rank the other backends' builds as plain. Options are the
curation-time check instead, and never reach the selection logic.
With no fallback left, tag correctness is load bearing, so audit every entry
against the rule "tagged when the entry configures that feature, in whatever
vocabulary its backend uses". Three entries configure MTP untagged and gain the
tag (hy3, glm-5.2, qwythos-9b-claude-mythos-5-1m, all spec_type:draft-mtp with
no marker in their names). Four carry the tag while configuring nothing and
lose it: qwen3.6-27b-nvfp4-mtp, qwen3.6-35b-a3b-nvfp4-mtp,
qwopus3.6-27b-coder-mtp-nvfp4 and qwopus3.6-27b-v2-mtp-nvfp4, whose only option
is use_jinja:true. The dflash side was checked independently rather than assumed
consistent: all five dflash entries declare spec_type:draft-dflash and all five
are tagged, so it needed no edits.
Four entries keep a tag that a literal spec_type-only reading would strip,
because they configure MTP through a different backend: deepseek-v4-flash-q2-mtp
via ds4's mtp_path/mtp_draft, and the three sglang entries via
speculative_algorithm in their referenced configs. Stripping those would
contradict the reason spec_type was rejected as the signal and would demote four
genuinely faster builds to plain.
The index was edited by line insertion and deletion only, never round-tripped
through a serializer. A resolved-tag diff across all 1272 named entries, taken
after merge keys are applied, shows exactly these 7 changing and no entry
gaining or losing a tag through an anchor.
The two specs that pinned the name fallback are inverted rather than deleted,
since a name silently promoting a build is the regression worth guarding. The
whole-token guard survives on the tag path, where smtp must still not match mtp.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): make deepseek-v4-flash variant targets installable
Clicking install on deepseek-v4-flash failed with "invalid gallery model".
The parent entry is fine, but all four entries it was grouped with declared
neither url: nor config_file:, and applyModel needs one of the two to have
anything to build a config from. They carry urls: (plural), the informational
HuggingFace link list, which is a different field. None of the four was ever
independently installable, so grouping them routed a previously-working
install into a broken entry.
Give each the url: the parent already resolves through. virtual.yaml is a
no-op base, and applyModel passes overrides to InstallModel separately from
the fetched config, so backend: ds4, the parameters and the ssd/mtp options
all still land exactly as authored. This is the same pattern the parent and
many other GGUF entries in the index already use.
Add the lint rule that should have caught this. checkVariantReferences only
proved a target exists and is not itself a parent, which is structural
validity: an entry can exist, declare no variants, and still be
uninstallable. checkVariantTargetsInstallable mirrors applyModel's
precondition instead, and names the parent, the target and the missing
fields, because whoever hits it is reading a gallery entry and has no reason
to know applyModel exists.
The two index-driven resolution specs live in their own Ordered container:
an Ordered container stops at its first failure, so sharing one with the lint
rules let a lint breach skip them silently.
Nine further entries gallery-wide have the same defect and are unrelated to
variants, so they are broken installs that predate this branch. They are left
alone here rather than buried in a regression fix, and widening the rule to
cover every entry is deferred with them so the gate can ratchet up in one
step instead of needing a skip list.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gallery): install entries with no url or config_file on an empty base
applyModel had three branches: fetch a base config from url:, build one from
an inline config_file:, or fail with "invalid gallery model". An entry
declaring neither is now installed on an empty base config, with overrides:
and files: supplying everything.
This is what the ~345 entries pointing at gallery/virtual.yaml were already
getting. That stub is five lines carrying name, description and license.
description and license are overwritten from the gallery entry immediately
after the fetch, and the name never reaches disk because InstallModel prefers
the install name. Crucially applyModel passes model.Overrides to InstallModel
as a separate argument rather than merging it into the fetched config, so
nothing an author writes depends on that base existing. The fetch bought a
round trip to GitHub and nothing else.
That makes f4ef80173 the wrong fix, so it is unwound. The four url: lines it
added to the deepseek-v4-flash variants are reverted: they are a pointless
network fetch now, and the family installs without them.
Relaxing the branch would hide a real authoring mistake, so a payload rule
replaces the base-config rule. An entry with no url, no config_file, no
overrides and no files installs nothing and would leave an empty model
directory while reporting success, so it is refused by name. The caller's
request counts toward the payload, because its overrides and files are merged
into the install exactly as the entry's own are. urls: (plural) is the
informational link list and does not count, which is what the four entries
that shipped broken had and why they were still uninstallable.
checkVariantTargetsInstallable asserted every variant target declares a url:
or a config_file:, which is no longer true and would now reject correct
authoring. checkEntriesInstallSomething pins what survives instead, and covers
every entry rather than only variant targets: the hazard is a half-written
stanza and a parent can be one as easily as a target. The old rule was scoped
to targets precisely because nine unrelated entries would have failed a
gallery-wide version; those nine are valid now, so the deferred ratchet
happens here in one step. 1280 entries, zero violations.
Those nine (aurore-reveil_koto-small-7b-it, lfm2-1.2b, the six liquidai_lfm2
entries and deepseek-v4-pro-q2-ssd) become installable for free. Each carries
overrides: and files:, and one of them is driven through the real install path
in a spec.
The no-fetch spec is paired rather than bare: an assertion that nothing was
fetched proves nothing unless something could have been, so a control runs the
same fixture with a url: pointing at a base config that is not there and
asserts the install fails. Only then does the identical fixture without the
url passing mean the read was skipped.
Follow-up, deliberately not here: the ~345 entries still naming virtual.yaml
can drop their url:. That is 345 index edits with their own risk, and mixing
them in would bury this change.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* ui(models): let search bypass the variant collapse, drop the toggle
The models page collapsed the gallery to one row per model by default and
offered a toggle to see every individual build. Because the collapse composed
with the search term, a build another entry offers as a variant could not be
found by typing its name, so the toggle was the only way to reach those builds
in the UI. A user who typed a name they knew existed got "no models found",
which reads as "that model does not exist".
Collapse is for browsing; search is for finding. An explicit search term now
bypasses the collapse in the listing handler, so a name lookup returns matching
entries whether or not a parent offers them. The term is trimmed once at the
top of the handler, so whitespace is neither a search nor a bypass; previously
an untrimmed blank term also narrowed the listing to whatever contained a
space. Tag and backend deliberately do not bypass: they refine a listing the
user is still reading rather than name an entry already known to exist.
That makes the toggle redundant, so it goes, along with its i18n strings in all
six locales, its localStorage persistence, its participation in "Clear filters"
and the empty-state hint telling users to turn it off. The hint was doubly
stale: it pointed at a control that no longer exists, and it was untrue exactly
when a user has a search term, since searching now sees every build. The page
always requests the collapsed listing.
The stored preference key is left inert rather than cleaned up: nothing reads
it, so a user who had the toggle off simply gets the collapsed view.
collapse_variants stays on the API, off by default, because other clients want
either view and the UI dropping its control is no reason to remove a working
parameter.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): give the models gallery filter form a deliberate structure
The filter area had accreted controls into one undifferentiated flow. The
"Fits in GPU" toggle and the backend select were direct children of
.filter-bar, the same wrapping container as the 18 taxonomy chips, so their
position was decided by how many chips happened to wrap at the current width
rather than by any layout intent. At narrow widths they were pushed past the
right edge of that container's horizontal scroll and became unreachable
entirely.
Restructure into three bands inside the house .filter-bar-group wrapper that
components/FilterBar.jsx already uses on Backends and the System tabs:
1. query scope: search plus the backend select
2. taxonomy: the chip row, alone, free to wrap
3. refinements: fits-in-GPU and context size, under a hairline rule
The backend select leads the chips rather than trailing them because picking a
backend disables the use cases that backend cannot serve, so it gates the row
below it. Fits-in-GPU and context size share a band because they are one
control group: the context size is the length the VRAM estimate is computed at,
and that estimate is what the fits filter tests against.
Chips had no visible keyboard focus indicator. The global focus ring is wrapped
in :where(), so it carries the specificity of a bare :focus-visible, ties with
.filter-btn and loses on source order, leaving focused chips showing their
resting drop shadow. Restate the ring where it outranks both resting and hover.
Also: aria-pressed on the chips, a real label association and aria-valuetext on
the context slider (it steps over an index, so it announced "2"), disabled chip
styling moved off inline styles, a prefers-reduced-motion block for the chip
transition, and the hard-coded English "Context:" moved into all seven locales.
No behaviour change: same filters, same state, same requests. Page reset on
change, localStorage persistence and "Clear filters" verified unchanged.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): let the models recommendations panel fade into the background
The "Recommended for your hardware" strip rendered at full height on every
visit regardless of how many models were already installed, costing 186px at
1600px wide (287px at 1100px, where its cards wrapped to two rows) and pushing
the first gallery row to y=554 / y=703.
Make its prominence track how much the user still needs it. The panel now
defaults to a one-line summary once anything is installed, and both the
collapse choice and the existing dismissal persist:
collapsed = explicit user choice, if one exists
: installedCount > 0
The preference is three-valued on purpose. A boolean cannot tell "the user
expanded it" apart from "the user has never chosen", and those need opposite
handling when the installed count later crosses zero: someone who deliberately
opened the panel on an empty instance should not have it collapse out from
under them when their first model finishes installing.
Collapsed keeps the card, icon, title and a suggestion count, so the panel is
recovered by clicking what you are already looking at rather than by hunting.
Expanded is unchanged, because for a user with nothing installed it was never
the problem. Collapsed reclaims 145px at 1600 and 420, and 246px at 1100.
Models.jsx gains a statsLoaded flag: stats initializes to installed:0, so
reading it before the fetch resolves would render expanded and collapse a frame
later, which is exactly the layout shove this removes.
The dismissal key moves to the page's localai-models-* convention; the old
localai_rec_models_dismissed is still read, never written, so an existing
dismissal is honoured rather than resurrected by the rename.
Accessibility: the disclosure is a real button whose accessible name is the
visible title alone, with state on aria-expanded and aria-controls resolving in
both states, because the grid is hidden via the hidden attribute rather than
unmounted. That also keeps the four install buttons out of the tab order while
collapsed. The app's global focus ring applies; no per-component outline is
added, per the warning in App.css. Reveal animates opacity and transform only,
never height, and both it and the chevron rotation are disabled under
prefers-reduced-motion.
Only en had a recommended block, so the other six locales were falling back to
English for the whole panel. Translated the complete block rather than adding
one orphaned key to files that would still render the title in English.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(downloader): recover from a leftover .partial on non-HTTP URIs
An interrupted download leaves a `<file>.partial` behind. The partial
handling in DownloadFileWithContext gated resume on `err == nil &&
uri.LooksLikeHTTPURL()`, so for any URI that is not literally http(s)
the branch fell through to `else if !errors.Is(err, os.ErrNotExist)`,
which with a nil err is true. The download then failed with an error
wrapping nil:
failed to check file ".../Ternary-Bonsai-27B-Q2_g64.gguf" existence: <nil>
Every gallery file URI uses `huggingface://`, so a single interrupted
download made that model permanently uninstallable until someone
deleted the partial by hand. The `<nil>` in the message compounded it
by pointing debugging at a filesystem failure that never happened.
Restructure the handling as an explicit switch over the four real
states: partial exists and is resumable, partial exists and is not
resumable (discard and restart, as already done for an HTTP server
without range support), no partial, and a genuine stat failure. The
error branch is now only reachable with a non-nil error, names the
path that was actually stat'd, and wraps with %w.
Discarding is required for correctness and not merely convenience: the
writer opens the partial with O_APPEND, so an un-resumed download would
concatenate a fresh body onto stale bytes.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): tell the models gallery's variant rows apart, and let browsing see every build
Both variant surfaces rendered name, backend and size. For two builds of one
model that is close to no information: a variant exists precisely because the
same weights are offered another way, so the backend usually matches and the
sizes usually land within a few hundred megabytes. Comparing
ternary-bonsai-27b-pq2 against ternary-bonsai-27b-q2-g64 meant reading two names
that differ by a suffix nobody has defined anywhere in the UI.
Report the quantization and the serving features on VariantView, and derive both
server-side from the referenced entry rather than parsing names in the browser,
so every client reads the same format out of the same file the installer will
hand the backend.
Quantization comes from overrides.parameters.model first, falling back to the
file list. That order is load bearing: entries routinely ship a vision tower
alongside the language model at a different quantization, so reading the file
list first reports the mmproj's format. Matching walks `-` and `.` delimited
segments right to left; `_` deliberately does not split, because it separates the
parts INSIDE a quant token and splitting on it reports Q4 for a Q4_K_M build. A
second, looser pass takes a segment's `_`-delimited tail, which catches the
gemma-4-E2B_q4_0-it.gguf style; it runs second so a precise match can never lose
to a fuzzy one further right in the name. An entry naming no format reports
nothing, which is the honest answer for a backend served from a directory of
weights.
Features are the same tag-against-vocabulary match servingFeatureRank already
ranks on, over the same host preference list. A build can therefore never be
shown as faster than one selection did not actually reward, nor rewarded without
being shown; a spec pins that agreement rather than trusting it.
The compact dropdown gets the quantization on its meta line and the bare feature
token. The detail row, which has the room, gets the quantization as its own
monospaced column so precision lines up down the list, and the feature spelled
out, because DFLASH names nothing to a user who has not met it. The referenced
entry's description stays out of both: the detail row already renders the
parent's prose above the table, and a second block per variant would push a
three-variant list past a screen to restate what the columns now say precisely.
The collapse toggle comes back. 462583f38 dropped it once search bypassed the
collapse, on the reasoning that nothing was unreachable any more. That holds for
finding a build whose name you know and does not hold for browsing: no sequence
of actions enumerated the 68 builds the default view hides. Collapse is for
browsing and search is for finding, and the toggle was the browsing half.
It goes in the refinements band 0d4823362 established, not back among the
taxonomy chips where its position depended on how many chips happened to wrap. It
leads that band because it decides how many rows the other two refine over, and
because unlike fits-in-GPU it is unconditional: a host with no GPU still browses.
The search bypass is untouched and re-checked by a spec in the toggle's default
state, since restoring the control must not restore the dead end it replaced. The
empty-state hint returns but only without a search term, because a term bypasses
the collapse and the hint would otherwise point at a control that cannot change
the result. The stored preference reads 'on'/'off' only: an older build wrote
'1'/'0' from an effect that ran on mount, so those record that the page was
opened, not that anyone chose a view.
Also fixes a latent flake it exposed. The collapse_variants spec compared whole
response bodies byte for byte, and the listing envelope carries live host
telemetry that drifts between two calls milliseconds apart, so it was asserting
on the machine's memory pressure. It now compares everything the parameter
governs -- the entries, their serialization and the paging -- and is green 25/25
where it was failing about one run in three.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): let the models gallery show a variant's full details
The variant list in an entry's expanded detail row says how the builds
differ: name, backend, quantization, size, and the auto-selected, base
and serving-feature markers. It cannot say what any one of them is. A
variant's own description, tags, license, source links and file list are
unreachable anywhere in the UI, because while the collapse is on a
variant has no gallery row of its own.
Give each variant row an info control that reveals its entry, rendered by
the same ModelDetail a top-level row gets, so a field added to the detail
view appears here too. variantData is withheld from the nested render: a
variant may declare variants of its own, and recursing would nest a
picker inside a picker two levels deep already.
An inline disclosure rather than a modal. The control sits inside a table
row that is already expanded, inside a variant list within that; a dialog
opened from there stacks a dismissal on a dismissal for a handful of
extra fields about the entry the user is already reading, and breaks the
page's own expand idiom. The third level is carried by an inset and a
left rule instead of another card.
The entry is fetched by exact name from the listing, once, on first use.
The listing already returns every field the detail view renders, and a
search term bypasses the variant collapse server-side, so no new endpoint
is needed and neither the listing nor DescribeVariants gains any work.
Expanding a row costs nothing; a variant nobody opens costs nothing. A
name the listing no longer returns is stated, not blanked: an empty panel
reads as a rendering fault rather than as a lookup that came back empty.
The control is a sibling of the install button, not a descendant, so
asking about a build can never install it.
The variant list keeps its content-sized columns via a trailing filler
track instead of max-content sizing, so the rows are unchanged while the
panel spanning them gets the pane width its file table needs.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(gallery): let search respect the collapse instead of switching it off
The models listing collapsed to one row per model, and an explicit search term
turned that off wholesale. Searching while collapsed therefore answered with the
individual builds a parent already offers, which are exactly the rows the view
the user asked for has no place for: typing "mtp" returned
qwen3.6-27b-nvfp4-mtp, a row that is invisible the moment the box is cleared.
The bypass was the right shape of fix for the wrong half of the problem. What a
search must not do is answer "no models found" for a build the gallery does
hold; that does not require abandoning the grouping the user asked for.
So the term is now matched against every entry either way, hidden builds
included, and the collapse decides how a match is reported rather than which
matches exist. Collapsing stops being a filter that drops rows and becomes a
substitution: a match on a build another entry offers is reported as that entry,
the one installable in its own right. Nothing becomes unfindable and nothing
comes back that the requested view cannot show.
Substitution happens after search, tag and backend, so every filter is judged
against the build that really carries the name, tag or backend rather than
against a parent that merely offers it; the other order would let backend=vllm
match a parent whose own backend is something else. The price is that the
surfaced row shows the parent's own metadata while the match was on a variant,
which is what grouping means, and the alternative is claiming the gallery holds
no such build. It happens before the count and the page math, so both describe
the rows actually handed out rather than the matches that produced them.
A parent already in the result keeps its own position and absorbs its matching
variants there, which is what leaves the browsing listing ordered exactly as it
was; a parent surfaced only by a variant takes the position of the first variant
that surfaced it. Either way it appears once, however many of its builds matched
and whether or not it matched itself. Search preserves gallery order rather than
scoring, so a surfaced parent has a real position rather than an invented one.
VariantParents never reports an entry that declares variants of its own, so a
parent is never itself hidden and one hop always lands on a visible row. The
handler follows exactly one anyway: refusing the second is what makes a gallery
the linter would have rejected terminate rather than loop.
The empty-state hint pointing at the toggle goes with it for every server-side
filter. Substitution means a match is always reported as some row, so the
collapse can no longer be why a term, a chip or a backend came back empty, and
naming it there sends the user to a control that cannot change the result. It
survives for the fits filter alone, which runs in the browser after the
substitution and judges the surfaced entry's own size: there the build that fits
really can be filtered out along with a parent that does not.
Searching a build's exact name while collapsed now answers with its parent, so
the result no longer contains the string the user typed. That is intended, and
the row is the one they can act on, but it is a real rough edge: nothing on the
row explains the connection. Closing it properly means reporting which variant
matched so the UI can say so, which the listing does not do today.
ResetGalleryModelCache is added for tests. The model cache is a package global
keyed by nothing, so a background refresh one spec triggers can land in the
middle of the next and answer it with the previous spec's gallery; the extra
specs here made that fail about one run in five. It waits for the in-flight
refresh to publish before clearing, since clearing alone only narrows the
window.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The backend matrix path filter only matched files under a backend's own
directory, so a change to shared build infrastructure rebuilt nothing at
all: an empty matrix, every job green, and the change reaching no image.
PR #10946 fixed scripts/build/package-gpu-libs.sh shipping a partial
4-of-8 cuDNN library set, which mixed versions with the venv's pip cuDNN
and produced CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH at inference time.
It merged 1h48m after the weekly full-matrix cron had already run, so no
backend image ever received the fix and nothing signalled that it had
been un-shipped.
Add a SHARED_BUILD_INPUTS table mapping each shared path to the narrowest
set of matrix entries it can honestly invalidate, plus a generic rule for
backend/Dockerfile.<x> (which each entry already names). A full matrix is
417 Linux + 56 Darwin builds, so package-gpu-libs.sh now rebuilds the 176
Python entries rather than everything. Unclassified files under
scripts/build/ fall back to a full rebuild deliberately: over-building is
recoverable, silently shipping nothing is not.
Extract the filtering logic to scripts/lib/backend-filter.mjs so it can be
unit-tested without bun, js-yaml or a GitHub API round-trip, and run those
tests from the existing lint workflow via `make test-ci-scripts`.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(distributed): reject wrong-model requests at the backend (#10952)
In distributed mode the controller caches a NodeModel row naming a backend's
host:port. A worker can recycle a stopped backend's gRPC port for a different
model's backend, and probeHealth verifies liveness rather than identity, so the
probe succeeds against whatever now occupies the port and the request is
dispatched to the wrong backend. The caller gets a silent wrong-model answer.
Nothing in the request could catch this: PredictOptions had no model field, so
model identity crossed the wire only in ModelOptions.Model at LoadModel time,
and the cached-hit path issues no LoadModel. Every backend's "model not loaded"
guard checks a nil handle, which a process holding a different model passes, so
the stale row was never dropped either.
Add PredictOptions.ModelIdentity and enforce it at the point of use:
- The controller populates it in gRPCPredictOpts from ModelConfig.Model, the
same expression ModelOptions feeds to model.WithModel and therefore the
same value the backend received as ModelOptions.Model. Both are read from
one config value in one function, so they are equal by construction and the
comparison cannot false-reject.
- Backends compare it against what they loaded and return NOT_FOUND with a
fixed sentinel. Enforced in pkg/grpc/server.go (27 Go backends), an
interceptor in backend/python/common (all 36 Python backends, no
per-backend change), and the llama-cpp / ik-llama-cpp / ds4 C++ servers.
That is every backend with real exposure: kokoros answers all four RPCs
with unimplemented and privacy-filter implements none of them.
- The router's reconcile drops the stale replica row on a mismatch, so the
next request reloads somewhere correct.
Empty means "skip the check" on both sides: a controller that predates the
field sends nothing, a backend loaded by such a controller has nothing to
compare, and the C++ server synthesizes PredictOptions internally for ASR. That
keeps upgrades working in both directions.
Scoped to the four PredictOptions RPCs. TTSRequest.model and
SoundGenerationRequest.model are deliberately NOT validated: FileStagingClient
already rewrites them to worker-local absolute paths, so in distributed mode
they already differ from the load-time value and comparing them would reject
valid requests.
IsModelMismatch requires both the NOT_FOUND code and the sentinel, unlike the
neighbouring helpers which accept either. insightface's Embedding returns
NOT_FOUND "no face detected" on a PredictOptions RPC, and a code-only check
would drop a healthy replica row on every faceless image.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* chore(deps): bump the pip group across 6 directories with 1 update
Bumps the pip group with 1 update in the /backend/python/ace-step directory: torch.
Bumps the pip group with 1 update in the /backend/python/llama-cpp-quantization directory: torch.
Bumps the pip group with 1 update in the /backend/python/longcat-video directory: torch.
Bumps the pip group with 1 update in the /backend/python/sglang directory: torch.
Bumps the pip group with 1 update in the /backend/python/trl directory: torch.
Bumps the pip group with 1 update in the /backend/python/vllm-omni directory: torch.
Updates `torch` from 2.10.0+rocm7.0 to 2.12.1+cpu
Updates `torch` from 2.10.0 to 2.12.1+cpu
Updates `torch` from 2.12.1 to 2.12.1+cu130
Updates `torch` from 2.9.0 to 2.12.1+cpu
Updates `torch` from 2.10.0 to 2.12.1+cpu
Updates `torch` from 2.7.0 to 2.12.1+cu130
---
updated-dependencies:
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cu130
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cpu
dependency-type: direct:production
dependency-group: pip
- dependency-name: torch
dependency-version: 2.12.1+cu130
dependency-type: direct:production
dependency-group: pip
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix(deps): preserve platform-specific torch requirements
Keep the 2.12.1 CPU bump only where uv resolves it cleanly, and restore ROCm, CUDA, MPS, and unrelated transformers constraints that Dependabot rewrote to incompatible wheel variants.
Assisted-by: Codex:gpt-5 [uv]
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Every vibevoice requirements file pulled diffusers straight from
git+https://github.com/huggingface/diffusers. That branch now reports
itself as 0.40.0.dev0 and requires huggingface-hub>=1.23.0,<2.0, while
transformers>=4.51.3,<5.0.0 (which upstream VibeVoice mandates) still
caps huggingface-hub at <1.0. Because a git URL offers the resolver
exactly one candidate version, uv has nothing to backtrack to and the
install fails outright:
Because only diffusers==0.40.0.dev0 is available and diffusers==0.40.0.dev0
depends on huggingface-hub>=1.23.0,<2.0 [...] we can conclude that your
requirements are unsatisfiable.
This broke the vibevoice build on every variant - cpu (amd64/arm64),
cuda 12/13, l4t 12/13, intel and rocm, plus the darwin metal job - and
has been failing the weekly full-matrix rebuild for three weeks. It is
not caught by master pushes because backend builds are path-filtered
there, so it only surfaces on the Sunday cron and on release tags.
Use the PyPI package instead. That is what upstream VibeVoice declares
in its own pyproject.toml, and what every other LocalAI backend already
does - vibevoice was the only one tracking the git branch. With a real
release series available the resolver settles on diffusers 0.39.0 with
huggingface-hub 0.36.2 and transformers 4.57.6, and it can keep
backtracking on its own if upstream shifts again.
Verified with uv pip compile against cpu, cublas12, cublas13, hipblas,
intel, mps and l4t13: all resolve to that same coherent set. l4t12 only
resolves on aarch64, since its Jetson index ships no x86_64 torch wheel.
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Bash] [uv]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Adds inference parameter defaults for the minimax-m3 model family and
includes a vendored patch of upstream llama.cpp PR #24523 to recognize
the minimax-m3 architecture. Once the upstream PR merges, the patch can
be removed and LLAMA_VERSION bumped normally.
Changes:
- backend/cpp/llama-cpp/patches/0001-add-minimax-m3-support.patch:
vendored patch from ggml-org/llama.cpp#24523 (Preliminary MiniMax-M3
support). Applied by prepare.sh during the build; keeps the pinned
LLAMA_VERSION pointing at the latest upstream tag.
- core/config/inference_defaults.json: add minimax-m3 family entry
(temperature=1.0, top_p=0.95, top_k=40, min_p=0.01,
repeat_penalty=1.0, matching the existing minimax defaults) and
register it in the patterns list before the shorter minimax-m2.7
entry for correct longest-match-first ordering.
Upstream: depends on ggml-org/llama.cpp#24523
Closes: https://github.com/mudler/LocalAI/issues/10820
Signed-off-by: Nandana Dileep <110280757+nandanadileep@users.noreply.github.com>
The worker's gRPC port allocator grew monotonically with no upper bound:
nextPort started at the base port and incremented whenever freePorts was
empty, and nothing checked 65535. Past that it handed out integers that
cannot be bound, surfacing as an opaque "backend won't start".
#10961 estimated this needed ~15,000 concurrent-peak allocations, i.e.
effectively unreachable. It is not, because of a second defect: the
"process died unexpectedly" branch in startBackend deleted the process
map entry without releasing its port at all. That port was leaked, never
quarantined and never reused. A crash-looping backend leaks one port per
restart, so a backend dying every 30s walks 50051 to 65535 in about five
days. The leak, not concurrent peak, is the realistic route to exhaustion.
Fixing the leak alone would have been wrong. Releasing that port makes it
re-bindable, and the death path is the one teardown path with no
request/reply to carry StoppedProcessKeys back to the controller (#10952's
eager row removal), so a stale NodeModel row could then resolve to a live
listener belonging to a different backend. probeHealth verifies liveness,
not identity, so the request is silently misrouted. The 15s port
quarantine does not cover this: the only reaper is the per-model health
check at ~45s, and it can be disabled outright. The residual was masked
only because the port was never rebound.
So both are fixed together:
- The allocator takes an explicit [basePort, LOCALAI_GRPC_MAX_PORT] range
and returns ErrNoFreePort naming the range, the live backend count, the
quarantined count, and the knob to raise. Exhaustion is now diagnosable
instead of surfacing as an unbindable port.
- Released ports carry per-key affinity: a port is offered back to the
process key that last held it before any other key. Process keys
(modelID#replica) and NodeModel rows (nodeID, modelName, replicaIndex)
are isomorphic, so a port that can only be re-bound by its previous
owner can only ever be named by that owner's row, which that key's
re-registration overwrites. Misrouting to a different model becomes
impossible by construction rather than by racing the quarantine timer.
Affinity is a preference, not a reservation: under range pressure an owned
port is stolen with a warning, because a guaranteed outage is worse than a
rare misroute window on a port long out of quarantine. Claiming a port
evicts its previous owner's entry, keeping ownership injective over ports
so the affinity map can never exceed the range width regardless of how
many distinct model keys the worker sees.
Ownership also expires. It is only load-bearing while a controller row
could still name the port, which the per-model reaper bounds at roughly
45s, so it lapses after five minutes and the port becomes ordinary free
space again. Holding it indefinitely would have made every distinct model
the worker ever served consume a port permanently: every release path is
keyed, so nothing would ever be unowned, the allocator would climb to the
end of its range on distinct-key count rather than concurrency, stealing
would become routine, and the steal warning would tell operators to widen
a range that was not the constraint. With expiry, reaching the steal
branch means the worker is genuinely out of concurrent capacity, so that
advice is correct when it appears.
Closes#10961Closes#10952
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(distributed): backend discovery hid worker-installed backends
Backend discovery endpoints filter on installed-state, which on a
distributed controller derives from the controller's own filesystem. A
backend lives on the worker node that runs it, so every backend an admin
installed on a GPU worker read as "not installed" and vanished from the
listing. #10947 fixed the sibling capability filter on the same endpoints,
so a fine-tuning-capable GPU worker now made the backend listable while
the installed-state filter still dropped it: the dropdown stayed empty.
The controller cannot derive this locally, but it already aggregates the
per-node view that GET /backends renders, so discovery reuses the active
BackendManager rather than growing a second path. Three surfaces shared the
root cause and route through the same helper now:
- GET /backends/available (Installed is now cluster-wide)
- GET /api/fine-tuning/backends
- GET /api/quantization/backends
The response stays a boolean rather than an installed-on-N-of-M count:
per-node install state is already served by GET /backends nodes[], and
per-node control by POST /api/nodes/:id/backends/install, so a summary is
all these dropdowns need.
A nil provider (single-node) leaves the local filesystem as the only source
and reproduces today's listing exactly, and a registry error degrades to
that same listing instead of blanking the catalog.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(worker): reap deleted backends and stop models that live on a worker
Three related backend-lifecycle defects, all reachable from the same
production incident on a Jetson/Thor worker: a deleted backend's gRPC
process survived ~40 minutes with its directory removed from disk, a later
model load was routed to that orphan and failed with a certifi path pointing
into the deleted directory, and the admin could not stop the model because
the frontend reported it as not loaded.
1. backend.delete orphaned the process it claimed to delete
------------------------------------------------------------
s.processes is keyed by `modelID#replicaIndex` (buildProcessKey), so the
backend name never appeared in a key and was recorded nowhere on the
process. backend.delete resolved its target via isRunning/stopBackend, whose
prefix path only matches a bare *modelID* - a delete keyed on a backend name
resolved to zero keys, the stop silently no-op'd, and the files were removed
out from under a live process.
The install fast path then handed that orphan back out: it returns any live
process for the (model, replica) slot without checking which backend started
it, so a reinstalled variant inherited the deleted backend's port.
- Record backendName on backendProcess, threaded installBackend ->
startBackend.
- Add resolveProcessKeysForBackend, matching the recorded name and resolving
alias <-> concrete via ListSystemBackends *before* DeleteBackendFromSystem
erases the metadata that carries the alias. Alias resolution failure
degrades to name-only matching so a delete never fails on it.
- backend.stop goes through resolveStopTargets, which accepts a backend
name, a model name, or an exact modelID#replica key. Its payload field is
named "backend" but is published with all three meanings: the admin UI
sends a backend name, UnloadRemoteModel sends a model name, and the
router's abandoned-load reap (#10948) sends an exact replica key.
Narrowing it to backend names alone would strand the latter two.
backend.delete stays strict - its identifier is unambiguously a backend.
- Gate the install fast path on processMatchesBackend so a slot held by a
different backend is restarted rather than reused. Processes with no
recorded name (pre-upgrade) are accepted, so rollout does not restart
every running backend.
- stopBackendExact reports a real stop failure - the process still being
alive afterwards, which is precisely what finishBackendStop already
detects to keep the entry and its port reserved - and backend.delete no
longer replies success when it knew about a process and could not kill it.
"No process was running" stays a success but is logged, so the orphan case
is visible rather than silent.
2. /backend/shutdown reported a running model as missing
---------------------------------------------------------
ModelLoader.deleteProcess short-circuits on a miss in this replica's
in-memory store. In distributed mode the authoritative record of "is this
model loaded" is the shared node registry: a frontend replica that never
served the model itself (load balancer picked a peer, or the replica
restarted) has no local entry. The remote unload path that pkg/model
documents ("when ShutdownModel is called for a model with no local process,
UnloadRemoteModel is called") sat behind that short-circuit, unreachable in
exactly the case it exists for. #10865 reworked this function but kept the
short-circuit at the top, so the gap survived that refactor.
- deleteProcess consults the remote unloader on a local-store miss, via a
shared unloadRemote helper so this branch and the existing
no-local-process branch both prefer #10865's RemoteModelContextUnloader,
preserving force propagation across the distributed boundary.
- UnloadRemoteModelContext reports ErrRemoteModelNotLoaded when no node has
the model; it previously returned nil, making a no-op stop
indistinguishable from a real one. The converse case (nodes have it, none
could be stopped) already errors since #10865 joined the per-node
failures, so that half of the original fix was dropped as redundant.
- Only when the model is absent locally AND cluster-wide does the endpoint
report not-found, now 404 naming both scopes rather than a bare 500.
- modelNotFoundErr becomes the exported ErrModelNotFound so the HTTP layer
can map it without string matching; watchdog's identity comparison becomes
errors.Is.
3. Coverage for the bounded Free() that #10865 shipped untested
----------------------------------------------------------------
The original branch also bounded the pre-stop Free(), but #10865 landed that
fix first (workerBackendFreeTimeout, applied in both stopBackendExact and
handleModelUnload). That production change is therefore DROPPED here as
superseded - master's version is strictly better, since it also releases the
supervisor mutex across the call and keeps the port reserved until
termination completes.
What #10865 did not ship is a test, and the bound is load-bearing: the
router-side reap in #10948 sends backend.stop for an abandoned load, and
against a wedged backend an unbounded Free would swallow that stop before it
reached the process. Nothing failed if the bound regressed.
The spec stands up a real gRPC backend server whose Free handler never
returns - what a Python backend looks like when its single worker thread
(PYTHON_GRPC_MAX_WORKERS=1 on 37 backends) is occupied by a stuck LoadModel.
A stub socket is not sufficient and was tried first: without a completed
HTTP/2 handshake, gRPC's own ~20s connect timeout ends the call, so that
version passed against the very bug it targets. With the connection READY,
only the caller's deadline can end it, so the spec hangs to its 60s limit if
the timeout is removed and passes with it.
Its fixture process is deliberately never started. go-processmanager v0.1.1
writes Process.pid from readPID() without synchronization, so a live process
races its own monitor goroutine under -race - reproducible with a bare
Run()+Stop() and unrelated to this spec. Since
scripts/model-lifecycle-conformance.sh runs this package with -race and is
fail-closed, starting one would turn that gate red on an upstream defect. An
unstarted process still proves the point: the stop is reached and the slot
released, which is exactly what an unbounded Free prevents.
Verified: make lint (new-from-merge-base origin/master) reports 0 issues;
scripts/model-lifecycle-conformance.sh passes all three stages including the
FizzBee liveness check (1458 states, IsLive: true).
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): keep remote unload idempotent, ask presence separately
2035a4d25 made UnloadRemoteModel return ErrRemoteModelNotLoaded when no node
holds the model, so ShutdownModel could answer 404 instead of a misleading
500. That narrowed a shared adapter contract to serve one caller and broke
the documented idempotent-unload guarantee, which CI caught on PR #10956:
[FAIL] Node Backend Lifecycle (NATS-driven) > NATS backend.stop events
should be no-op for models not on any node [Distributed]
Expected success, but got: model not loaded on any node
The spec name states the contract outright. The matching unit assertion was
updated in that commit; this e2e one was missed because it lives under
tests/e2e/ with no build tags and does not run in package-scoped test runs.
Caller audit - who breaks when an idempotent unload becomes an error:
- pkg/model/watchdog.go:902 (LRU memory reclaimer) is the serious one. It
untracks a model ONLY when shutdown returns nil or ErrModelNotFound. A new
error type means the model is never untracked, so the reclaimer keeps
re-selecting the same entry and never reclaims - a live wedge whenever a
local store entry outlives the remote model.
- core/services/galleryop/managers_local.go:43 (DeleteModel) would warn on
every deletion of an already-unloaded model.
- core/services/modeladmin/{state,config,remote_sync}.go stop instances
best-effort against models that are frequently not loaded.
- deleteProcess itself: the no-local-process branch returns the unload result
directly, so a stale local entry for a model no longer on any node turned a
previously-successful cleanup into a failure.
Only ShutdownModel wants the distinction, and only on the local-store-miss
path. So the distinction moves to the caller instead of the contract:
- UnloadRemoteModel/UnloadRemoteModelContext return nil again when no node
has the model, and ErrRemoteModelNotLoaded is removed.
- New optional RemoteModelPresenceChecker (HasRemoteModel) answers the
question directly. deleteProcess consults it BEFORE unloading, because an
idempotent unload cannot report afterwards whether anything was stopped.
Absent locally AND cluster-wide is the only case that reports 404.
- A failed registry lookup is surfaced rather than reported as absence: an
unreachable registry is not evidence a model is gone, and answering a
confident 404 off a failed lookup is how an operator gets told a running
model does not exist.
- Unloaders that predate the extension keep working - deleteProcess attempts
the unload rather than refusing it - and compile-time assertions in the
nodes package now pin all three optional interfaces, since both are
consumed by runtime type assertion where drift degrades behavior silently
instead of failing the build.
The contract is now pinned at both levels that disagreed, each spec pointing
at the other: "with no nodes returns nil" in unloader_test.go and "should be
no-op for models not on any node" in node_lifecycle_test.go.
Verified: full distributed e2e suite 233 passed / 0 failed (the suite that
failed 232/1 in CI); pkg/model and core/services/nodes green; make lint
new-from-merge-base reports 0 issues.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): drop replica rows when a worker stops a backend
A worker returns a stopped backend's gRPC port to its allocator as soon as
the process is confirmed dead, and hands it to the next backend that starts.
The controller's NodeModel row for the old address survives, and both
SmartRouter.probeHealth and the HealthMonitor per-model probe verify
liveness, not identity, so once an unrelated backend binds the recycled port
the stale row passes every check and the request is served by the wrong
backend instead of failing.
backend.delete is newly able to trigger this: before #10956 a delete never
actually stopped a process, so it never recycled a port. backend.upgrade has
the identical gap and always did — upgradeBackend force-stops every process
using the binary and starts none back up, while
DistributedBackendManager.UpgradeBackend never removes rows. model.unload is
the one path that gets this right today: it calls RemoveAllNodeModelReplicas
straight after StopBackend.
Report the process keys the worker terminated on the delete and upgrade
replies, and drop the matching rows in RemoteUnloaderAdapter, which already
holds a ModelLocator with RemoveNodeModel. All three call sites funnel
through that adapter, so no new interface, DB migration, or proto change is
needed. A key is reported only once its process is confirmed gone, so the
list stays trustworthy on the partial-failure replies too.
Old workers never populate the new fields. ReportsStoppedProcesses tells
"stopped nothing" apart from "does not report", so an old worker's silence
falls back to the pre-existing probe-based staleness recovery instead of
being mistaken for a completed cleanup.
Quarantine released ports for a short window as an interlock covering the
NATS round-trip between the worker freeing the port and the controller
dropping the row. It is deliberately not derived from HealthCheckInterval:
that cadence is operator-tunable and the per-model reaper can be disabled
outright, so coupling a worker-local constant to it would be silently wrong
on some clusters. Eager row removal is the fix; the delay only closes the
handoff gap.
Identity verification in probeHealth was considered and rejected: Health and
Status carry no backend identity, so it needs a proto change plus an
implementation in 36 Python and 4 C++ Health servicers, it is fail-open for
any backend not yet rebuilt, and the probeCache short-circuit means it would
not even execute during the 30s window where the misroute happens.
Fixes#10952
Refs #10954, #10956
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* chore(deps): bump go-processmanager, assert real backend termination
go-processmanager wrote Process.PID from readPID() with no synchronization
while its own monitor goroutine cleared the same field on exit, so a bare
Run()+Stop() tripped the race detector without any concurrent access from
the caller. LocalAI hit this on every backend stop.
Upstream fixed it in a94e2b7 by guarding PID with a mutex and adding
CurrentPID() as a race-safe accessor. The exported field was kept to avoid
a breaking change but is now deprecated: a direct read still races the
monitor. No tag carries the fix yet, so pin the pseudo-version.
GetGRPCPID reads through CurrentPID() instead of the field. The accessor
returns the same string under an RLock, so the empty-PID and strconv error
paths are unchanged; it is the only direct field read in the tree.
With the race gone, the Free-timeout spec no longer has to leave its
fixture process unstarted. It now runs a real child and asserts the child
genuinely exits, which is exactly what the earlier workaround gave up: the
spec could show the stop was reached and the slot released, but not that
SIGTERM ever landed. Termination is observed through Done(), which closes
only once the library has waited on the child. The pidfile-based liveness
helpers cannot serve here, because Stop() deletes the pidfile while
releasing the handle and so reports "not alive" even if no signal was sent.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update CrispStrobe/CrispASR
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(crispasr): initialize only declared submodules
The latest upstream commit contains an undeclared CrispASR gitlink that makes a blanket recursive submodule update fail. Limit initialization to the two submodules used by the backend build.
Assisted-by: Codex:gpt-5 [Codex]
* fix(crispasr): resolve vendored WebRTC from project root
CrispASR now builds a vendored WebRTC VAD, but its include paths assume CrispASR is the top-level CMake project. Extend the existing embedded-project rewrite to the shared third_party root.
Assisted-by: Codex:gpt-5 [Codex]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(gpu): detect GPUs via sysfs when no pci.ids database is present
ghw.GPU() calls pci.New() before it reads /sys/class/drm and fails
outright when it cannot find a pci.ids database file. jaypipes/pcidb
embeds no database and has network fetch disabled by default, so on an
image that ships no pci.ids, GPU enumeration returns an error and every
detection path downstream goes dark.
The Dockerfile installs pciutils only in the vulkan and cublas branches,
so the Intel image had no pci.ids. A correctly passed-through Arc A310
was reported as "No GPU detected" with zero VRAM even though clinfo and
sycl-ls both enumerated it inside the same container. NVIDIA and AMD
images were shielded by their nvidia-smi / rocm-smi binary fallbacks;
Intel has no equivalent, leaving it fully exposed.
Read PCI vendor IDs directly from /sys/class/drm/card*/device/vendor,
which needs no database, and consult that from DetectGPUVendor. The
same scan replaces the ghw-only guard in getIntelGPUMemory, which is
what had been blocking the working clinfo path and keeping VRAM at
zero. Install hwdata in the base image stage as well, so ghw stops
failing for every image variant rather than only Intel.
Also apply the documented NVIDIA > AMD > Intel priority to the ghw
path, which previously returned whichever card DRM enumerated first
and so reported "intel" on a machine with an Intel iGPU at card0 and
an NVIDIA dGPU at card1.
HasGPU() carried the same blindness plus one of its own: it matched
the requested vendor against ghw's card description with a
case-sensitive Contains, so "nvidia" never matched the pci.ids
spelling "NVIDIA Corporation". It only worked because that same
description embeds the lowercase kernel driver name ("nvidia",
"amdgpu"), and it returned false outright whenever ghw errored. Route
it through the shared vendor lookup so it matches case-insensitively
and falls back to sysfs. It feeds the GPU option and NGPULayers
defaults in core/config/gguf.go.
Fixes#10941
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* refactor(gpu): key vendor detection off the numeric PCI ID in both paths
The ghw and sysfs legs were identifying vendors by different means: ghw
by substring-matching the pci.ids vendor name, sysfs by the numeric PCI
vendor ID. ghw already exposes that same numeric ID via
DeviceInfo.Vendor.ID, read from the kernel's modalias rather than from
the database, so the name matching was both a duplicate mechanism and
the weaker of the two.
It is weaker because a card absent from an outdated pci.ids gets
Name: "unknown" while its ID is still correct. Detection then failed
even though ghw had enumerated the card successfully. Verified in a
container with a vendor-less pci.ids and an Arc's modalias: before,
DetectGPUVendor returned ""; after, "intel".
Both legs now resolve through the same pciVendorIDs table and share the
hex parsing, with the vendor name kept only as a fallback for devices
exposing no parseable ID.
ghwHasVendor is deliberately not a priority pick, unlike vendorFromGHW:
HasGPU("intel") must stay true on a hybrid-graphics host whose discrete
NVIDIA card outranks the integrated Intel one.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(gpu): silence the gosec G304 on the sysfs attribute read
gosec flags os.ReadFile with a non-literal path. The path here is the
DRM root (a package constant in production, a temp dir under test)
joined with a ReadDir entry name and a fixed attribute filename, so no
external input reaches it.
gosec's suggested autofix, os.Root, cannot be used: /sys/class/drm/cardN
is a symlink into the PCI device tree, and os.Root refuses to traverse
it ("path escapes from parent"), which would disable the whole scan.
Assisted-by: Claude:claude-opus-4-8 gosec golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
request.language usually carries an ISO 639-1 code (e.g. "de"), which
OpenAI-compatible clients such as Home Assistant / wyoming_openai send,
but qwen_asr.validate_language() only accepts full English names
("German") and raises ValueError otherwise. Normalize the requested
language: accept full names case-insensitively, translate ISO codes
(with optional region suffix like "de-DE") to the expected name, and
pass anything unrecognised through so qwen_asr still reports it clearly.
Fixes#10958
Signed-off-by: Tai An <antai12232931@outlook.com>
llama.cpp picks a random per-process media marker (ggml-org/llama.cpp#21962),
so LocalAI renders the prompt with a "<__media__>" sentinel and swaps in the
backend's real marker after probing ModelMetadata.
That probe was gated on an exact match against "llama-cpp", the gallery's meta
backend name. A model config pinning a concrete build ("vulkan-llama-cpp",
"cuda12-llama-cpp", "rocm-llama-cpp", ... and their -development counterparts)
runs the same llama.cpp gRPC server but skipped the probe, so MediaMarker
stayed empty, no substitution happened, and the prompt reached mtmd still
carrying the sentinel. mtmd_tokenize then counted zero markers against one
bitmap and every image request failed with "Failed to tokenize prompt".
The same early return also skipped thinking-mode detection and tool-format
marker extraction, so a pinned variant silently lost reasoning and native
tool-call parsing too.
Add IsLlamaCppBackend, which recognises the whole variant family (plus the
empty auto-detect name, which resolves to llama.cpp) while excluding
ik-llama.cpp, a separate engine that merely shares the suffix.
Fixes#10945
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): make the remote LoadModel deadline configurable
The router hardcoded a 5 minute gRPC deadline for the remote LoadModel
call. Staging finishes before the timer starts, so those five minutes
cover only the worker backend's own checkpoint load and pipeline init.
A cold load of meituan-longcat/LongCat-Video-Avatar-1.5 (~83 GB) on an
ARM64 Thor worker fails at exactly 302s with DeadlineExceeded while the
backend process is still making progress (CPU time accumulating, RSS
moving as weights are mapped), so the load was cut short rather than
wedged.
Add LOCALAI_NATS_MODEL_LOAD_TIMEOUT / --model-load-timeout mirroring the
existing backend-install timeout knob, defaulting to 5m so unset
clusters keep today's behaviour.
The cold-load hold ceiling (which bounds how long one load may hold the
per-model advisory lock) was derived from the install timeout alone, so
raising the load deadline past it would have been silently clipped.
Derive it from both budgets via ModelLoadCeilingFor:
max(install + load + 5m staging margin, 25m)
With the defaults that is 15m + 5m + 5m = 25m, identical to the previous
constant, and the 25m floor means shrinking either budget can never
tighten the ceiling below what clusters relied on before.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): reap the abandoned replica when a remote load times out
The gRPC deadline on the remote LoadModel call only cancels the client
side. A backend blocked in a synchronous weight load never observes its
cancelled handler context, so when scheduleAndLoad gave up it left the
worker loading with nobody waiting for the result.
Observed on an ARM64 Thor worker loading LongCat-Video-Avatar-1.5: the
client returned DeadlineExceeded at 302s, and the backend process was
still alive 30 minutes later having pulled ~57GB from HuggingFace. Every
retry stacked another multi-GB loader on the worker; they had to be
reaped by hand via POST /api/nodes/:id/models/unload.
Send backend.stop for the exact `modelID#replicaIndex` process key we
just abandoned. The exact key matters: a bare model ID stops every
replica on that node, including healthy ones serving traffic.
Only a deadline or cancellation triggers the reap. Any other LoadModel
failure is the backend answering, which means its handler returned and
the process is idle - stopping it there would discard a warm process and
its downloaded weights. The reap is best-effort and never replaces the
load error the caller is waiting on.
The `modelID#replicaIndex` format was already hand-rolled in two places
(the worker's buildProcessKey and pkg/model's log store). Rather than add
a third, export model.BackendProcessKey from pkg/model, the lowest common
dependency of both sides.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 golangci-lint
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(model-artifacts): materialize longcat-video checkpoints on the controller
longcat-video loads a checkpoint directory: its backend.py takes
request.ModelFile when os.path.isdir(request.ModelFile) and otherwise
falls back to snapshot_download. That places it in the same class as
transformers/vllm/diffusers/sglang, but the allow-list added in #10910
did not enumerate it, so PrimaryArtifactSpec returned no managed
artifact for a bare HuggingFace repo id.
The consequence in distributed mode: nothing was acquired on the
controller, ModelFileName fell through to the raw repo id, and staging
skipped the resulting phantom /models/<owner>/<repo> path. The worker
received a blank ModelFile, fell back to request.Model, and downloaded
~83GB from HuggingFace inside the remote LoadModel deadline - so the
load could only ever fail with DeadlineExceeded while an abandoned
backend process kept downloading.
Note this materializes the full repository. The backend restricts its
own snapshot_download with allow_patterns, and the avatar repo ships
both base_model/ and base_model_int8/ where only one is ever loaded;
inferred specs have no way to carry patterns today. Tracked separately.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): warn when staging skips a non-existent model path
stageModelFiles logs "Staging model files for remote node" up front, then
silently drops any path field that does not exist on the controller. The
skip itself is legitimate and must stay: a backend outside
managedArtifactBackends that takes a bare HuggingFace repo id gets an
optimistically constructed path (ModelFileName falls through to the raw
model reference) that was never materialized, and sources its own weights
on the worker. Erroring would break those configs.
But at debug level the operator is left with a reassuring staging line and
no trace of the skip, so a genuine controller-side acquisition gap is
indistinguishable from a healthy pass-through - it surfaces much later as
a remote LoadModel timeout, on a worker that is quietly downloading tens
of gigabytes. Raise the skip to warn and name the field, path, node and
tracking key. Behavior is unchanged.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(model-artifacts): allow a config to declare companion artifacts
A composed pipeline needs more than one HuggingFace snapshot.
LongCat-Video-Avatar-1.5 loads its own transformer but takes the
tokenizer, text encoder and VAE from the separate LongCat-Video base
repo, so a single-artifact config cannot express it and the backend is
left to fetch the second repo itself at load time.
Widen the artifact model to target: model plus any number of named
target: companion entries. Normalize accepts the new target and
constrains a companion name to [a-z0-9][a-z0-9_-]{0,63} because that
name is the option key the backend later receives; a companion may not
claim primary_file, which only means anything for a load target.
ModelConfig.Validate requires exactly one primary and requires it first,
since Artifacts[0] is what ModelFileName, size estimation and staging all
resolve from.
Both acquisition paths now loop instead of touching index 0 alone:
preloadOne for an already-installed config, bindPrimaryArtifact for a
gallery install. Failure policy differs by provenance. An inferred
primary keeps its warn-and-fall-back, because the legacy download path
still exists for it. Companions are explicit by construction, so they are
all-or-nothing: a config naming one is asserting the backend needs it,
and failing at the acquisition boundary is far more legible than a
missing-weights error surfacing later inside the backend.
The cache key is deliberately unchanged. It hashes source identity only,
never name or target, so every already-installed managed model still hits
its existing snapshot instead of silently re-downloading. Two specs pin
that: one proving a companion and a primary with identical sources agree
on the key, and one pinning the digest of a known primary outright.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(model-artifacts): hand resolved companion snapshots to the backend
A materialized companion is useless until the backend can find it, and
its location is a content-addressed cache key that does not exist until
the artifact resolves. A static gallery override cannot carry that, and
persisting it into the config YAML would rot the moment a re-resolve
produced a new key.
Synthesize it instead at load time: each resolved companion becomes
"<artifact name>:<snapshot path>" in ModelOptions.Options, reusing the
key:value convention backends already parse for options like
attention_backend. The value stays relative to the models directory so a
remote worker can resolve it under its own ModelPath once staging has
rewritten the model root. An option the author set explicitly always
wins, so pinning a companion to a local checkout still beats the managed
snapshot.
longcat-video resolves base_model through ModelPath, the same convention
qwen-tts, voxcpm, outetts and ace-step already use for companion assets.
Its sibling-directory heuristic is deleted: it looked for a LongCat-Video
directory next to the model, which cannot exist under the content
addressed .artifacts/huggingface/<key>/snapshot layout, so it was dead
code the moment the model became managed.
The gallery entry declares both repositories and restricts each with
allow_patterns. The avatar repo ships base_model/ and base_model_int8/
and only ever loads one, so fetching the whole repo would roughly double
the download. The patterns match the entry's own options (use_distill
true, use_int8 default false); enabling use_int8 here also requires
adding base_model_int8/**, which is called out in the entry.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): stage managed artifact trees from the models root
Staging anchored the worker's models directory on the primary snapshot
whenever a model was managed, so a companion snapshot could not reach the
worker at all.
frontendModelsDir was derived by stripping the Model relative path off
the end of ModelFile. For a managed artifact nothing matches: ModelFile
is .artifacts/huggingface/<key>/snapshot while Model stays a bare
HuggingFace repo id, so the strip was a no-op and the "models directory"
came out as the snapshot itself. Two consequences, both silent. Staging
keys lost the .artifacts/huggingface/<key>/snapshot prefix, so two
snapshots of one model were indistinguishable on the worker. And a
companion, which lives in a sibling snapshot directory outside the
primary, fell outside that directory entirely: StagingKeyMapper.Key
collapsed its files to bare basenames and resolveOptionPath could not
resolve the relative option at all, so it was skipped without a word.
Derive the models root from the artifact tree instead when the path runs
through it, and compute the worker's ModelPath from the file's path
relative to that root rather than from the Model field. The legacy layout
is unaffected: where Model really is the relative path, the new
derivation reduces to the old one, which a regression spec pins.
This deliberately changes an invariant that router_dirstage_test.go
pinned: for a managed primary, ModelFile and ModelPath were both the
snapshot directory, and staging keys were relative to it. Now ModelFile
is the snapshot, ModelPath is the models root above it, and keys keep the
full relative path. That spec is updated rather than accommodated, with
the reasoning recorded inline, because the old invariant is exactly what
made a sibling companion unreachable.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(ci): build Bonsai backend images
Register the Bonsai C++ source path with the backend matrix filter so changes select its image jobs. Also make shared llama.cpp changes rebuild the Bonsai and Turboquant fork images in the actual matrix, not only their test flags.\n\nAssisted-by: Codex:gpt-5 [Codex]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* fix(backends): list backends runnable on worker nodes in distributed mode
GET /backends/available filtered the gallery against the system state of
the host serving the request. In a distributed deployment that host is the
controller, which typically has no GPU, while the GPUs live on worker
nodes. Any meta backend whose capabilities map lacks a "default" (or "cpu")
key was therefore dropped from the listing entirely — longcat-video,
vllm-omni, ltx-video, parakeet, edgetam and qwentts were invisible in the
UI even though installing them by name on a GPU worker worked fine.
Workers now report their own meta-backend capability at registration and
the controller persists it on the node row. The controller cannot derive
it: OS-dependent capabilities (metal, darwin-x86, nvidia-l4t) and the CUDA
runtime refinements are only observable on the worker. Nodes registered
before this field existed fall back to a coarse capability derived from
their GPU vendor and VRAM.
Backend discovery then evaluates compatibility as the union over healthy
backend nodes, so a backend runnable on any node is offered while one no
node can run stays hidden. Each remote capability is evaluated through a
capability-pinned system state, otherwise a forced capability on the
controller image (LOCALAI_FORCE_META_BACKEND_CAPABILITY or
/run/localai/capability) would silently override every worker's verdict.
With no registered nodes the listing is byte-for-byte what it was, so
single-node deployments are unaffected.
Also fixes the same-root-cause misclassification in /api/operations, which
used the capability-filtered listing to decide whether an operation was a
backend or a model install. A GPU-only backend installing on a worker is
still a backend operation on the controller, so that lookup is now
unfiltered.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(backends): union worker capabilities in backend discovery
Implementation for the specs added in the previous commit, plus the two
remaining discovery endpoints.
Capability-filtered backend discovery evaluated compatibility against the
system state of the host serving the request. In a distributed deployment
that host is the controller, which typically has no GPU, while the GPUs
live on worker nodes. Any meta backend whose capabilities map lacks a
"default" (or "cpu") key was dropped entirely — longcat-video, vllm-omni,
ltx-video, parakeet, edgetam and qwentts were invisible in the UI even
though installing them by name on a GPU worker worked fine.
Workers now report their own meta-backend capability at registration and
the controller persists it on the node row. The controller cannot derive
it: OS-dependent capabilities (metal, darwin-x86, nvidia-l4t) and the CUDA
runtime refinements are only observable on the worker. Nodes registered
before this field existed fall back to a coarse capability derived from
their GPU vendor and VRAM.
Discovery then evaluates compatibility as the union over healthy backend
nodes, so a backend runnable on any node is offered while one no node can
run stays hidden. Each remote capability is evaluated through a
capability-pinned system state, otherwise a forced capability on the
controller image (LOCALAI_FORCE_META_BACKEND_CAPABILITY or
/run/localai/capability) would silently override every worker's verdict.
With no registered nodes the listing is byte-for-byte what it was, so
single-node deployments are unaffected.
Four surfaces shared this root cause and are all routed through the same
helper now:
- GET /backends/available
- GET /api/fine-tuning/backends
- GET /api/quantization/backends
- /api/operations backend-vs-model classification, which additionally
had no reason to filter by capability at all: a GPU-only backend
installing on a worker is still a backend operation on the
controller, so that lookup is now unfiltered.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
cuDNN 9 is a dispatcher (libcudnn.so.9) plus seven sublibraries the dispatcher
dlopen()s by bare soname. Only the dispatcher is ever a DT_NEEDED, so ldd finds
it and never the seven. The allowlist force-copied three of them
(libcudnn.so*, libcudnn_ops.so*, libcudnn_cnn.so*) into every CUDA backend,
which is wrong in both directions at once: too few libraries for a backend that
uses cuDNN, and too many for one that does not.
On an L4T fleet, ten of the eleven backends carrying cuDNN were in a broken end
state; the one that was correct was correct by accident, being BUILD_TYPE=cpu
so package_cuda_libs never ran for it.
longcat-video bundled 4 of 8 at 9.24.0 over a complete pip set at 9.20.0.48
in its venv. libbackend.sh puts lib/ on LD_LIBRARY_PATH, searched before
DT_RUNPATH, so the bundle won and the rest still came from the venv:
CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH.
Nine others bundled 3 of 8 and had no venv cuDNN. None bundled
libcudnn_graph, which libcudnn_cnn has a hard DT_NEEDED on, so it resolved
out of the runtime image and the process ran bundled 9.22.0 against system
9.23.2.
Five of those nine - llama-cpp, whisper, rfdetr-cpp, sam3-cpp,
stablediffusion-ggml - do not reference cuDNN at all. ggml goes through cuBLAS.
They were carrying ~57 MB of cuDNN with no consumer, and completing the family
for them would have taken that to ~576 MB for nothing.
Sizes overall: backends with no cuDNN consumer shed ~57 MB each (seven
instances on the fleet measured, plus longcat's ~60 MB), while the ones that
genuinely use cuDNN grow from ~57 MB to ~576 MB, because the five missing
sublibraries are ~517 MB, dominated by libcudnn_engines_precompiled. Net on
that fleet is an increase of roughly 570 MB. That growth is the bug being paid
off, not a regression: those backends only work today by silently borrowing the
missing five from the runtime image. Whether the engines set can be trimmed is
an open question, not addressed here.
So bundle per backend, by what that backend actually needs:
- venv has a complete pip cuDNN -> bundle nothing; $ORIGIN resolves the pip
set, which is the one its torch was built against (longcat-video)
- venv has no pip cuDNN -> bundle the complete family. Stays
conservative rather than detecting consumers: for a Python backend they sit
inside the venv (torch, ctranslate2, onnxruntime) where the sweep does not
look (vllm)
- no venv, nothing references cuDNN -> bundle nothing (llama-cpp, whisper,
rfdetr-cpp, sam3-cpp, stablediffusion-ggml)
- no venv, something references it -> bundle the complete family
(face-detect, voice-detect)
The no-venv case needs no new machinery. Go backends stage their own shared
object into package/lib, which IS the target dir, so sweep_transitive_deps
already pulls the dispatcher when it is a genuine dependency - that is exactly
how libcudnn_graph reached longcat. cuDNN simply comes off the force-copy list,
and complete_cudnn_family fills in the seven dlopen'd sublibraries around
whatever the sweep found. Detection is a string scan rather than ldd, so a
consumer that only dlopen()s cuDNN is seen too; over-matching costs an unused
library, under-matching costs a backend that cannot load.
Keeping bundled and pip versions in agreement instead is not viable: nothing
here pins nvidia-cudnn (zero occurrences), torch is unpinned for l4t13 except
longcat-video, and the fleet already runs five concurrent cuDNN versions -
9.19.0.56, 9.20.0.48, 9.22.0, 9.23.2, 9.24.0.
verify_cudnn_bundle asserts the end state: exactly one complete cuDNN visible to
whoever needs one - never both, never partial, and never zero for a backend that
references it. Zero is correct and common otherwise. It deliberately does not
accept the build image's system cuDNN as completing a partial bundle, which is
the shape that had been shipping silently; the build image is not the runtime
image. A version check alone would have missed longcat too, whose four bundled
libs were all 9.24.0 and mutually consistent.
Match per family for the other components for the same dlopen reason: TensorRT
(libnvinfer_plugin, libnvinfer_builder_resource), cuBLAS, cuFFT, cuSPARSE,
cuSOLVER, nvRTC. Exclusions bind inside copy_lib so they cover the sweep.
The packaging scripts' shell tests ran nowhere in CI. Add make
test-build-scripts and a lint workflow job so they gate every PR.
Fixes#10905
Assisted-by: Claude:claude-opus-4-8 golangci-lint shellcheck
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Pin vLLM protogen to grpcio-tools 1.78.0 so its generated code remains importable by protobuf 6.33.x, and remove stale generated artifacts before regeneration.
Closes#10940
Assisted-by: Codex:gpt-5 [Codex]
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Avoid holding the global loader lock across backend lifecycle waits and propagate forced shutdown through distributed workers. Track parallel requests with in-flight counters and reserve worker ports until process termination.
Add focused race tests and an authoritative FizzBee lifecycle model with a fail-closed conformance target.
Assisted-by: Codex:GPT-5 [FizzBee] [Ginkgo]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
The Vite build emitted path-absolute asset URLs (base: '/'). index.html
entry scripts and the favicon were rewritten to include the reverse-proxy
prefix in serveIndex, but two reference kinds are not in index.html and so
bypassed that rewrite:
- CSS `url()` font references (e.g. Font Awesome .woff2), which the browser
resolves relative to the stylesheet and which `<base href>` never affects
- lazily-imported route chunks, whose preload base came from the absolute
Vite base
Under a subpath mount (X-Forwarded-Prefix: /llm/) both were fetched from the
origin root, 404ing — missing-glyph "tofu" icons and broken lazy-loaded pages.
Switch Vite to a relative base ('./') so every generated URL resolves against
the file that references it: CSS fonts and route chunks now load from
`/llm/assets/...`, and index.html's now-relative entry refs resolve via the
`<base href>` serveIndex already injects on every response. Root deployments
are unaffected. The existing path-absolute rewrite in app.go still covers the
public `/favicon.svg`.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The weekly "Dependabot Updates" pip job for /backend/python/diffusers has been
failing with `private_source_authentication_failure` against the Jetson pip
index (https://pypi.jetson-ai-lab.io/jp6/cu129/), referenced by that backend's
requirements-l4t12.txt. diffusers is the only dependabot-configured pip
directory that pulls from that private index, so it is the only update job that
fails; the other backends update cleanly.
torch and transformers are deliberately pinned in this backend for
reproducibility (see backend/python/diffusers/requirements-*.txt and #9979), so
we do not want dependabot bumping them anyway. Ignoring both dependencies for
this directory stops dependabot from resolving them against the unreachable
Jetson index and keeps the weekly update job green, without removing update
coverage for the rest of the backend's dependencies.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The managed-artifact materializer stages a HuggingFace snapshot into a
directory (.artifacts/huggingface/<key>/snapshot/). That is the right load
target for directory-consuming backends (transformers, vLLM, diffusers, ...),
but PrimaryArtifactSpec inferred a managed artifact from ANY HuggingFace-shaped
model reference regardless of backend. A single-file backend such as llama.cpp
or whisper was therefore handed the snapshot directory instead of the weight
file and failed to load it.
The /import-model importer already guards this with a backend allow-list
(managedArtifactBackends), but the loader-side inference did not. Move the
allow-list into core/config as IsManagedArtifactBackend and apply it in
PrimaryArtifactSpec: only directory-consuming backends may have an artifact
inferred from a bare reference; every other backend stays on the legacy
download-to-file path. An explicit artifacts: block still bypasses the gate,
where single-file snapshot resolution handles the load path.
The importer now shares the same predicate, so both paths agree on which
backends auto-materialize.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(model-artifacts): load single-file HF snapshots from the file, not the dir
The managed Hugging Face artifact materializer (#10825) always pointed
backends at the snapshot *directory*
(.artifacts/huggingface/<key>/snapshot). For a single-file model
reference such as huggingface://nomic-ai/nomic-embed-text-v1.5-GGUF/nomic-embed-text-v1.5.f16.gguf,
the GGUF lives *inside* that directory, so llama.cpp was handed a
directory and failed with "gguf_init_from_reader: failed to read magic".
This has kept the tests-aio job red on master since the feature merged
(the embeddings e2e tests could not load text-embedding-ada-002).
Record the single file of a one-file snapshot as Resolved.PrimaryFile and
have ModelFileName() resolve to snapshot/<PrimaryFile> when it is set.
Multi-file snapshots (e.g. transformers repos consumed as a directory)
keep pointing at the snapshot directory. PrimaryFile is derived from the
resolved contents and is deliberately excluded from the artifact cache
key. estimateModelSizeBytes now derives the snapshot directory from the
cache key instead of ModelFileName(), so its manifest lookup is unaffected
by the file-vs-directory resolution.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The artifact progress bridge mapped every PhaseVerifying event to a flat
95%. The materializer emits PhaseVerifying once per file (from each file's
AfterDownload hook) and downloads run sequentially, so the first small file
to finish pinned the bar at 95% - and, because progress is monotonic, it
stayed at 95% for the entire remaining download (e.g. a 70GB checkpoint
reporting 95% at 410MB / 69.7GB).
Track per-file verify proportionally to the running aggregate bytes, the
same way downloading does. CurrentBytes already reflects "completed files +
this file", so the percentage advances honestly. The flat 95%/99% is now
reserved for the genuinely once-per-install Committing/Persisting phases.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The Operate → Traces "API Traces" panel already recorded who made each
request (user_id/user_name) but never showed it, and did not capture the
caller's network identity at all. Operators asked to see the requesting
user (#10886) and the client IP + user agent (#10887) so a trace can be
attributed to who/what issued it.
Backend: add ClientIP and UserAgent to APIExchange and populate them from
echo's c.RealIP() (honours X-Forwarded-For / X-Real-IP behind a trusted
proxy) and the request's User-Agent header. Both are omitempty and the
/api/traces swagger response is map[string]any, so this is additive.
UI: add a sortable "User" column to the API traces table and a metadata
block (User / Client IP / User Agent) at the top of the expanded row
detail. Fields render only when present, so older buffered traces and
unauthenticated/local requests degrade cleanly.
Adds an e2e spec covering the new column value and the expanded metadata.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The Operate -> Traces table rendered the request time with the time of day
only, so entries that span more than one day were ambiguous. Add a
formatDateTime helper (localized date + existing time-with-millis) and use it
for the Traces "Time" column, keeping the cell on a single line. The shared
formatTimestamp used by the log views is unchanged.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
One agent turn runs several internal LLM generations (tool selection,
reasoning, final answer) that all emit stream_event deltas over the same
per-agent SSE channel. The chat page accumulated every 'content' delta
into a single live bubble and ignored the 'done' boundary events, so the
internal generations' text (e.g. the English tool-selection rationale)
merged with — and visually corrupted — the streamed final answer.
Reset the accumulated content/reasoning on 'done': each generation gets
a clean live bubble, and the authoritative full answer still arrives via
the final json_message event as before.
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
chore(moss-transcribe-cpp): bump pin to CUDA K-quant embed fix
Bumps the moss-transcribe.cpp pin to 190a569c, which merges the
host-side embed-lookup fallback for K-quant token_embd tensors
(localai-org/moss-transcribe.cpp#2).
Before this, running a q5_K/q4_K/q6_K moss-transcribe GGUF on CUDA
(or any non-CPU backend) aborted in getrows.cu with
"unsupported src0 type: q5_K" because ggml's GET_ROWS op has no
K-quant implementation on GPU, killing the backend process on the
first request. The engine now dequantizes the needed rows on the
host when the backend cannot run GET_ROWS for the tensor type,
producing bit-identical results.
Fixes#10862
Assisted-by: Claude:claude-opus-4-8 [Bash] [Edit]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Add the q8_0 (default) and f16 gallery entries for the moss-tts-cpp backend, each
pulling the MOSS-TTS-Local v1.5 GGUF plus the MOSS-Audio-Tokenizer-v2 codec and
the text tokenizer from mudler/MOSS-TTS-Local-Transformer-v1.5-GGUF. The backend
auto-discovers the codec and tokenizer siblings; output is 48 kHz stereo with
reference-audio voice cloning.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(backend): add moss-tts-cpp text-to-speech backend
Add a Go + purego backend wrapping the moss-tts.cpp ggml port of the OpenMOSS
MOSS-TTS-Local v1.5 text-to-speech model (GPT-J local transformer decoded through
MOSS-Audio-Tokenizer-v2), producing 48 kHz stereo audio with optional
reference-audio voice cloning. Mirrors the qwen3-tts-cpp backend: dlopen the
static-ggml shared library, bind the moss-tts.cpp C-API via purego, and serve
the gRPC TTS method. A thin C shim holds the pipeline handle and copies engine
PCM into a Go-freeable buffer.
Wires the CI registration: backend-matrix.yml (CPU, CUDA 12/13, Intel SYCL
f16/f32, Vulkan, ROCm, NVIDIA L4T, plus Darwin metal), backend/index.yaml metas
and image entries pointing at mudler/MOSS-TTS-Local-Transformer-v1.5-GGUF, the
root Makefile build targets, and the changed-backends.js path mapping.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: list the moss-tts-cpp backend among the LocalAI-maintained engines
Add moss-tts.cpp to the README "Backends built by us" table, the
Text-to-Speech compatibility table, and the reference-audio voice-cloning
backend list, so the new backend is documented alongside its peers.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(moss-tts-cpp): pin moss-tts.cpp to the squashed single-commit release
moss-tts.cpp history was collapsed to a single commit; repoint MOSSTTS_CPP_VERSION
to ee722b8e9205ee9b1b1c398a4e87e4e393e9be41.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(moss-tts-cpp): add the moss-tts-cpp-development gallery meta
The gallery had the -development image entries but no matching -development
meta anchor (as locate-anything-cpp and depth-anything-cpp have), so the master
build was not installable as a gallery backend. Add moss-tts-cpp-development
mirroring the production meta with the -development capability image names.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Google shipped the Gemma 4 MTP drafter heads and llama.cpp merged native
support in ggml-org/llama.cpp#23398. LocalAI's pinned llama.cpp already
carries it, and the config plumbing (draft_model + core/config/mtp.go)
was built for exactly this path, but no official Gemma 4 gallery entry
wired it up.
Add llama.cpp draft-mtp speculative-decoding variants for the dense
sizes, sourced from the unsloth QAT GGUF repos (target UD-Q4_K_XL +
mtp-*.gguf drafter + BF16 mmproj):
- gemma-4-e2b-it-qat-mtp
- gemma-4-e4b-it-qat-mtp
- gemma-4-12b-it-qat-mtp
- gemma-4-31b-it-qat-mtp
These replace the previously commented-out attempts, which were disabled
because the Janvitos/boxwrench drafter GGUFs declared the architecture as
`gemma4_assistant` (underscore) and failed to load on stock llama.cpp.
The unsloth drafters use the upstream `gemma4-assistant` (hyphen) spelling
that mtp.go's isDraftOnlyAssistantArch expects, so they load without any
backend patch. The 26B-A4B MoE is intentionally omitted (the upstream PR
reports no meaningful MTP speedup for it).
Also fix gemmable-4-12b-mtp: it loaded the draft-only `-mtp` GGUF as the
main model with no draft_model set, which cannot run standalone. It now
loads the target as the model, wires the drafter via draft_model, enables
spec_type:draft-mtp, and downloads both files.
All sha256 pins were taken from the HuggingFace API lfs.oid (reliable
content hash even for Xet-backed repos).
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
StopP2P() read and wrote a.p2pCtx/a.p2pCancel without holding
a.p2pMutex, and StartP2P() reassigned both fields with no lock at
all -- including when RestartP2P() calls it from a background
goroutine after releasing the mutex. Both paths are reachable from
POST /api/settings (empty p2p_token -> StopP2P, non-empty ->
RestartP2P), so concurrent requests race on the same fields.
Take a.p2pMutex in StopP2P and around the field publication in
StartP2P, factor the shared teardown into stopP2PLocked() so
RestartP2P reuses it, and route the goroutine error path through
StopP2P instead of touching a.p2pCancel unlocked.
Signed-off-by: Anai-Guo <antai12232931@anaiguo.com>
Co-authored-by: Anai-Guo <antai12232931@anaiguo.com>
* ⬆️ Update CrispStrobe/CrispASR
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(crispasr): rewrite c2pa-audio submodule path for subproject builds
CrispASR a38cb89 adds a crispasr_c2pa_native static library whose sources
live in the new third_party/c2pa-audio git submodule, located via
CMAKE_SOURCE_DIR in src/CMakeLists.txt. That variable assumes CrispASR is
the top-level CMake project; LocalAI embeds it via add_subdirectory, so
the path resolved to backend/go/crispasr/third_party/c2pa-audio and every
build variant failed at CMake generate with 'Cannot find source file:
c2pa_native.cpp'.
Extend the existing talk-llama sed workaround to also rewrite the
c2pa-audio reference to PROJECT_SOURCE_DIR, which is correct both
standalone and as a subproject. The submodule itself is already checked
out by the recursive submodule init. Verified locally: the exact CI error
reproduces with CMAKE_SOURCE_DIR, and with the rewrite CMake configure,
crispasr_c2pa_native, and crispasr-lib all build cleanly on a CPU-only
fallback configuration.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The turboquant and bonsai backends copy backend/cpp/llama-cpp/ wholesale
into their build directories and reuse its Makefile/prepare.sh against
their own llama.cpp forks. When PR #10837 added
backend/cpp/llama-cpp/patches/0001-add-minimax-m3-support.patch, the
copied patches/ directory was mis-applied to the fork checkouts: the
fork trees diverge from upstream, hunks rejected, and because the
patch-apply loop in prepare.sh ran before set -e took effect the build
kept going and died much later with a confusing compile error
("'LLM_ARCH_MINIMAX_M3' was not declared in this scope"). This broke
tests-turboquant-grpc on that PR.
Two hardening changes:
- turboquant/bonsai Makefiles: delete the copied patches/ directory
right after the cp -rf of backend/cpp/llama-cpp/. Patches vendored
for upstream llama.cpp must never be applied to the forks; each fork
carries its own patch series under backend/cpp/<backend>/patches/,
applied by its apply-patches.sh.
- llama-cpp prepare.sh: run the patch-apply loop under set -e so a
rejecting patch fails fast and loudly at apply time instead of
surfacing as a downstream compile error. A missing or empty patches/
directory remains a no-op success, so all existing callers (the
llama-cpp Makefile targets and the turboquant/bonsai copies) are
unaffected when no patches ship.
Exposed by PR #10837.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(settings): add declarative runtime-settings field registry
One fieldSpec row per RuntimeSettings field, with a reflection
completeness spec so a field added without a registry row is a red
test instead of a silently-dropped setting (the #10845 bug class).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* refactor(settings): drive ToRuntimeSettings/ApplyRuntimeSettings from the field registry
Behavior-preserving: ~350 hand-written per-field lines become two loops
over runtimeSettingsFields, gated by a To->Apply->To round-trip spec.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* feat(settings): baseline-driven startup merge for persisted runtime settings
ApplyRuntimeSettingsAtStartup compares the live config against
DefaultRuntimeBaseline (option-less-run defaults incl. kong-injected
flag defaults) instead of per-field == 0 guards. Fixes persisted
lru_eviction_max_retries, tracing_max_items, agent_job_retention_days,
memory_reclaimer_threshold, galleries and autoload flags being
silently ignored at boot.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* fix(settings): registry-driven startup merge, applied before consumers
loadRuntimeSettingsFromFile becomes a thin wrapper over
ApplyRuntimeSettingsAtStartup and runs at the top of New(), before
model configs capture app-level defaults. WithThreads stops eagerly
resolving 0 so a persisted thread count survives restart while
LOCALAI_THREADS still wins (#10845); the physical-core fallback moves
after the merge.
Also: run.go now injects the memory-reclaimer threshold unconditionally
so the option-less boot matches DefaultRuntimeBaseline and a UI-saved
threshold survives restart.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* refactor(settings): file watcher delegates to the registry merge; shared API-key merge
Manual edits to runtime_settings.json now behave like a boot-time load
(env still wins) instead of the inverted diverged-from-startup guard
that ignored most manual edits. MergeAPIKeys dedups env keys in one
place for the endpoint and the watcher.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* docs(settings): document unified runtime-settings precedence
Document the single env/CLI > runtime_settings.json > defaults rule,
applied identically at boot, on POST /api/settings, and on manual file
edits, plus the two known limitations (default-valued env vars are
indistinguishable from unset; API-changed fields hot-apply on the next
restart only). Also add a completion debug log when the watcher applies
runtime_settings.json.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* test(settings): reset the global VRAM cap leaked by the round-trip spec
The round-trip spec applies vram_budget=12GiB, whose post-loop hook
installs a process-global default cap; without a reset every spec
ordered after it runs under that phantom budget. Also drop a stale
enumeration in the ApplyRuntimeSettings doc comment.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The gRPC configuration table only listed the two fields with a one-line
description each, without defaults, without explaining what the total
load window looks like, and without hinting when a user should adjust
them. In practice the default 20 attempts x 2 s = 40 s window is way
too tight for large NVFP4 / FP8 models on slow storage or first-run
CUDA-graph capture, and the resulting kill (exitCode=120, 'context
canceled') looks like a backend crash even though the backend is still
making legitimate forward progress.
Extend the section with:
- Defaults column (20 and 2) added to the table
- Prose explaining that these govern the readiness handshake between
LocalAI and a freshly spawned backend (Health polling loop)
- Total-load-window formula
- Concrete failure signature so users can recognize a timeout-kill
vs. a real backend crash
- Example configuration for a ~10 min cold-load window (grpc.attempts
140, attempts_sleep_time 5), with a note that inference-timeouts and
the watchdog are unaffected.
The sglang Python backend inherits the default Status RPC from
backend_pb2_grpc.BackendServicer, which raises NotImplementedError.
LocalAI's backend-monitor polls /backend.Backend/Status periodically on
every registered backend; when the call fails, /backend/monitor returns
HTTP 500 and downstream inference requests to the sglang backend are
blocked even though the model is loaded and answering directly via the
gRPC endpoint.
Add a minimal Status shim that mirrors the existing Health method and
returns StatusResponse{state=READY} unconditionally. This unblocks the
monitor path; a state-aware follow-up (UNINITIALIZED during load, BUSY
under active inference) is left for a subsequent change.
Reproduced on DGX Spark (GB10, arm64-l4t-cuda-13 image) with the sglang
v0.5.15 backend and Qwen3-Coder-Next-NVFP4-GB10; verified locally that
patching the shim in place immediately restores /backend/monitor and
inference across the sglang slot.
* fix(audio-transform): serialize WebSocket writes to avoid concurrent-write panic
AudioTransformStreamEndpoint writes to the same Gorilla WebSocket connection
from two goroutines: the backend-forwarding goroutine emits binary PCM frames
(and can call sendWSError on a backend recv error), while the read loop calls
sendWSError for malformed mid-stream JSON or a backend send failure. Gorilla
WebSocket permits only one concurrent writer, so these writers race and can
panic with "concurrent write to websocket connection", resetting the client
session; a -race build reports the data race directly.
Wrap the connection in a lockedConn that serializes WriteMessage behind a
mutex, mirroring the existing lockedConn used by the openresponses WebSocket
endpoint. Reads stay on the single read loop, so only writes need the lock.
Fixes#10844
Signed-off-by: Tai An <antai12232931@outlook.com>
* chore: empty commit to re-trigger checks
Signed-off-by: Anai-Guo <antai12232931@anaiguo.com>
---------
Signed-off-by: Tai An <antai12232931@outlook.com>
Signed-off-by: Anai-Guo <antai12232931@anaiguo.com>
Co-authored-by: Anai-Guo <antai12232931@anaiguo.com>
When the watchdog's busy-killer decides a backend has been busy past the
busy timeout, it shuts it down via ModelLoader.ShutdownModel -> deleteProcess,
which grabs ml.mu and then waits for IsBusy() to clear BEFORE stopping the
process. But a backend that exceeds the busy timeout is, by definition,
stuck on an in-flight gRPC call, so the graceful wait never returns, ml.mu
is held forever, and every other ml.Load blocks — including the shared
opus backend load at the start of every realtime (WebRTC) session. New
realtime connections then hang at "Connected, waiting for session..."
whenever the watchdog is enabled, while logs repeatedly print the
watchdog's busy / "active connection" line.
Fix: add a force shutdown path (ShutdownModelForce / deleteProcess(s,
force=true)) that stops the process FIRST — dropping the stuck call's
gRPC connection and unblocking it — instead of waiting on it. Route the
watchdog's busy-killer and busy LRU / group / memory evictions through
the force path; keep the graceful wait for idle and user-initulated
unloads. Graceful/unforced kills are unchanged.
Regression test: the watchdog busy-killer uses ShutdownModelForce.
Fixes#10391
Assisted-by: opencode:glm-5.2 [opencode]
Signed-off-by: Nandana Dileep <110280757+nandanadileep@users.noreply.github.com>
feat(bonsai): add PrismML llama.cpp fork backend + Bonsai gallery models
Adds a new `bonsai` backend that runs the PrismML fork of llama.cpp
(github.com/PrismML-Eng/llama.cpp, `prism` branch), which ships the Q1_0
(1-bit) and Q2_0 (ternary / 1.58-bit) weight-quantization kernels used by the
Bonsai and Ternary-Bonsai models. Stock llama.cpp cannot decode these quants.
Modeled on the turboquant backend: reuses backend/cpp/llama-cpp/grpc-server.cpp
against the fork's libllama via a thin wrapper Makefile, so the sub-2-bit models
are served with the same OpenAI-compatible API. No grpc-server allow-list patch
is needed (bonsai adds weight quants, transparent to the server, not KV-cache
types), and the reused server compiles cleanly against the fork with no skew
patches (validated locally via a CPU docker build; patches/ is present but empty
for any future re-pin skew).
Backend wiring: backend/cpp/bonsai/, .docker/bonsai-compile.sh,
backend/Dockerfile.bonsai, top-level Makefile targets, backend-matrix.yml build
rows (CPU, CUDA 12/13, L4T, SYCL f32/f16, Vulkan, ROCm/hipblas), backend/index.yaml
meta-backend + per-platform images, and a nightly bump_deps entry tracking the
`prism` branch.
Gallery: 8 entries across 4 families - bonsai-8b-1bit, ternary-bonsai-8b (+g64,
+pq2), bonsai-27b-1bit (vision), ternary-bonsai-27b (+pq2, +g64, vision). The 27B
models wire the mmproj vision tower; the DSpark speculative drafter GGUFs are not
wired (custom semi-autoregressive drafter, not a standard llama.cpp draft model).
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(watchdog): guard StopWatchdog with watchdogMutex to prevent double close
StopWatchdog checked, closed and cleared a.watchdogStop without holding
a.watchdogMutex, while startWatchdog and RestartWatchdog reassign and close the
same channel under that lock.
POST /api/settings dispatches to StopWatchdog or RestartWatchdog depending on
ApplicationConfig.WatchdogShouldRun(), so both are reachable concurrently. Two
callers can observe a non-nil watchdogStop and both close it, which panics with
'close of closed channel' and takes the server down.
Take the mutex, matching the other two writers. StopWatchdog is only called from
the settings handler, which holds no lock, so this cannot deadlock.
Fixes#10841
Co-authored-by: Anai Guo <antai12232931@anaiguo.com>
Finetune() compiled every model cutstrings/extract_regex entry via regexp.Compile
and called xlog.Fatal on failure, which terminates the entire local-ai process.
A single model config with an invalid regex (e.g. cutstrings: ["("]) turns one
/v1/chat/completions request into a process-level denial of service.
Log the compile error and skip the offending pattern instead. The mutex is
released before continuing, and skipping avoids dereferencing the nil regexp
that removing the fatal would otherwise leave behind.
Fixes#10843
Signed-off-by: Tai An <antai12232931@outlook.com>
ApplyRuntimeSettings persists the performance settings (threads,
context_size, f16) on the live /api/settings path, but the startup
loader loadRuntimeSettingsFromFile never read them back, so a value
saved via the Middleware UI was silently ignored on the next restart:
the model booted with the CLI/physical-core default and GET /api/settings
echoed that default instead of the saved value (#10845).
Threads needs special handling: unlike context_size/f16, WithThreads
eagerly resolves an unset (0) value to xsysinfo.CPUPhysicalCores() at
option-apply time, so options.Threads is never 0 in the loader and the
usual "== default" heuristic cannot tell an env/CLI value from the
physical-core fallback. Detect LOCALAI_THREADS/THREADS explicitly so the
env still wins over the persisted file value.
Signed-off-by: Anai-Guo <Anai-Guo@users.noreply.github.com>
Co-authored-by: Anai-Guo <Anai-Guo@users.noreply.github.com>
Add stable and development gallery variants for Linux and Darwin, and wire the backend build matrix so the referenced images are published.
Assisted-by: Codex:gpt-5 [yq]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Pairs unsloth/Qwen3.5-4B-GGUF (Q4_K_M target) with the
AtomicChat/Qwen3.5-4B-DFlash-GGUF Q8_0 drafter (quantized from
z-lab/Qwen3.5-4B-DFlash, upstream GGUF arch `dflash`), same shape as
the existing DFlash entries.
Assisted-by: Claude Code:claude-fable-5 [Bash] [Read] [Edit]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): add vrambudget primitive for per-node VRAM caps
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): apply default VRAM budget in xsysinfo aggregate getters
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): wire LOCALAI_VRAM_BUDGET flag to xsysinfo default budget
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): persist VRAM budget via runtime settings with live apply
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(vram): reset process-global VRAM budget after runtime-settings spec
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): add VRAM budget field to Settings page
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): store and enforce per-node VRAM budget in the node registry
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): apply per-node VRAM budget in router hardware defaults
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): report worker VRAM budget in node registration
The distributed worker now reports its operator-set VRAM budget string
(LOCALAI_VRAM_BUDGET) to the server on registration. The worker keeps
reporting RAW total/available VRAM and never sets the xsysinfo
process-global budget (that stays standalone-only); the server resolves
and enforces the budget uniformly (Task 6).
Also closes a Task 6 gap: on re-registration, a struct Updates zero-skips
an empty budget, so a worker that dropped LOCALAI_VRAM_BUDGET left the
stale cap in place. For non-admin-override nodes the budget columns are
now force-written (map Updates) even when empty, so removing the env var
clears the cap; admin overrides are preserved unchanged.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* style(vram): drop em dash from worker-clear comment
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): add node VRAM budget admin endpoints
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): add node VRAM budget control to the node UI
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): expose set_node_vram_budget MCP admin tool
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(vram): document LOCALAI_VRAM_BUDGET and node VRAM budget UI
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vram): avoid double-applying VRAM budget in GetResourceAggregateInfo
The GPU-branch aggregate returned by GetResourceInfo is sourced from
GetGPUAggregateInfo, which already caps total/free/used against the
process-wide VRAM budget. GetResourceAggregateInfo then applied the
budget a second time. For an absolute budget this is idempotent, but for
a percentage budget b.Apply resolves the ceiling as a fraction of its
input total, so a second pass yields P*(P*T) instead of P*T and distorts
UsagePercent (read by the memory reclaimer in pkg/model/watchdog.go).
Remove the redundant second application so the budget is applied exactly
once, against the raw physical totals, upstream in GetGPUAggregateInfo.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vram): implement SetNodeVRAMBudget on mcp assistant test stub
The LocalAIClient interface gained SetNodeVRAMBudget; the stubClient in
core/http/endpoints/mcp used by the assistant tests is a separate
implementer and needs the method too (broke golangci-lint typecheck and
both test jobs).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* test(core/http): make the suite's HTTP port overridable
app_test.go and openresponses_test.go hardcoded 127.0.0.1:9090. When
another service already listens on 9090 the suite does not fail fast:
the server goroutine logs the bind error and the specs then poll
whatever is squatting the port until Eventually times out. On machines
where 9090 is permanently taken this makes the pre-commit coverage gate
impossible to pass.
Introduce testHTTPAddr, defaulting to 127.0.0.1:9090 (what CI has
always used) and overridable via LOCALAI_TEST_HTTP_PORT for local runs.
Assisted-by: Claude:claude-fable-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): make per-node backend upgrade actually upgrade
The node detail page's Upgrade button reused the node-scoped install
path (POST /api/nodes/:id/backends/install). That fires NATS
backend.install with force=false, and the worker's install handler is
deliberately "ensure installed": when the backend binary already exists
on disk it short-circuits without touching the gallery. Since only an
installed backend can be upgraded, the whole chain was a guaranteed
successful no-op - the UI then toasted "backend upgraded" without even
waiting for the async job.
Route upgrades through the real force-reinstall path instead:
- BackendManager.UpgradeBackend now receives the ManagementOp (like
InstallBackend already did) so implementations can honor
op.TargetNodeID.
- DistributedBackendManager.UpgradeBackend scopes the backend.upgrade
fan-out to op.TargetNodeID when set, and errors when the target node
does not report the backend as installed.
- New POST /api/nodes/:id/backends/upgrade endpoint enqueues an
Upgrade=true node-scoped op (async 202 + jobID, mirroring install).
- NodeDetail UI calls the new endpoint and reports the dispatch
("Upgrading ... on this node...") instead of claiming success; the
Operations panel tracks the actual job.
Verified against a live local cluster (NATS + Postgres + two workers):
the target worker stops the running process, force-reinstalls from the
gallery and re-downloads the OCI image; the second worker receives no
backend.upgrade event; upgrading a backend missing from the target node
fails the job with a clear error.
Assisted-by: Claude:claude-fable-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
PyTorch 2.13 XPU pulls oneAPI 2026 libraries that conflict with the oneAPI 2025.3 backend image. Pin torch and torchaudio to the matching 2.11 XPU pair so the build resolves a coherent 2025.3 runtime.
Assisted-by: Codex:GPT-5 [uv]
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
SetDefaults injected the llama.cpp server options cache_reuse
(ApplyServingDefaults) and parallel (ApplyHardwareDefaults, re-applied
per selected node by the distributed router) onto every model config
regardless of backend. Every other backend ignores options it does not
understand, so this was harmless until longcat-video, which strictly
validates its options and fails LoadModel with
"unknown model option(s): cache_reuse, parallel".
Gate both injections behind a new UsesLlamaCppServingOptions allow-list
(llama-cpp plus the empty/auto-detect case that resolves to llama.cpp
from a GGUF file, mirroring how llamaCppDefaults is registered). This
follows the existing UsesLlamaSamplerDefaults precedent for llama-only
defaults. The typed NBatch field is deliberately left alone: it is a
proto field every backend simply ignores, which is why batch never
triggered the error.
Also harden the longcat-video backend to warn-and-ignore unknown model
options and request params through a testable select_known_options
helper, matching the other LocalAI Python backends, so a future
server-injected option cannot break loading again.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
handleRegenerate rebuilt the outbound message from the display-only
message.files metadata ({name, type: 'file'|'image'|..., content}),
which doesn't carry the base64/textContent payload sendMessage's
file-building loop expects. As a result, regenerating any answer whose
own question had an attachment silently dropped that attachment from
the resent message. This wasn't fork-specific, but forking a chat and
then regenerating an earlier (now non-last) answer is the natural way
to hit it.
Fix by reusing the original message's already-assembled `content`
verbatim (it already has the file text / image_url / audio_url /
video_url parts embedded from the first send) instead of trying to
reconstruct it from lossy display metadata.
Fixes#10806
Assisted-by: Claude:claude-sonnet-5
Signed-off-by: ajuijas <189517297+ajuijas@users.noreply.github.com>
Co-authored-by: ajuijas <189517297+ajuijas@users.noreply.github.com>
- Replace hardcoded text with useTranslation hook in UI components
- Add localization support for both English (en) and Indonesian (id) locales
Signed-off-by: Dedy F. Setyawan <dedyfajars@gmail.com>
* feat(ui): add voice library workflow
Give administrators a production-ready flow to record or upload consented reference audio, manage reusable profiles, inspect API usage, discover compatible models, and hand a saved voice directly to text-to-speech.
Assisted-by: Codex:gpt-5
* feat(voice): add managed voice cloning profiles
Make reusable reference voices manageable through the admin API instead of requiring model-directory and YAML edits. Discover compatible installed and gallery models from server-side backend capabilities, retain explicit model configuration controls, and stage saved references for supported backends.
Expose profile management through REST and MCP, document backend-specific behavior, and cover the workflow from profile creation through real Qwen3-TTS synthesis. Harden the agent-job HTTP test against completion racing cancellation.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(backends): add LongCat video and avatar generation
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] [web]
* refactor(config): declare model I/O modalities
Make model configs declare input and output modalities so capability discovery no longer branches on backend or checkpoint names. Complete the LongCat gallery and user documentation, make the SDPA patch apply to the pinned upstream revision, and stabilize the Agent Jobs race exposed by the required hook.
Assisted-by: Codex:GPT-5 [web]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Add four ready-to-run DFlash speculative-decoding entries for the
llama.cpp backend, now that upstream DFlash support (draft-dflash) is in
the pinned llama.cpp. Each entry bundles a full target model with its
small z-lab block-diffusion drafter and sets spec_type:draft-dflash,
spec_n_max:15, and flash attention (required by DFlash):
- qwen3-4b-dflash (Qwen3-4B + Qwen3-4B-DFlash drafter)
- qwen3.5-9b-dflash (Qwen3.5-9B + Qwen3.5-9B-DFlash drafter)
- qwen3.6-27b-dflash (Qwen3.6-27B dense + drafter)
- qwen3.6-35b-a3b-dflash (Qwen3.6-35B-A3B MoE + drafter)
The 4B pair uses the base Qwen3-4B target (not Qwen3.5-4B): its drafter
reports general.name "Qwen3 4B DFlash" and is the canonical pairing
documented upstream. All drafters were downloaded and verified to carry
GGUF architecture "dflash" (not the fork-only "dflash-draft" /
"DFlashDraftModel") so they load in the upstream backend, and every
drafter SHA256 was confirmed against the downloaded bytes.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* docs: refresh LocalAI homepage
Reframe the homepage around LocalAI's modular multimodal runtime, native inference engines, deployment range, and built-in platform capabilities. Remove the outdated video in favor of current product visuals and clearer paths into the documentation.
Assisted-by: Codex:gpt-5
* docs: give homepage a full-width canvas
Let the product homepage opt out of Relearn's persistent sidebar and duplicate title while preserving the documentation shell on interior pages. Tighten the responsive bounds for narrow screens.
Assisted-by: Codex:gpt-5
* docs: fit homepage to the Relearn content flow
Remove the full-width shell exception and use a single-column homepage inside the standard documentation layout. This avoids competing scroll containers and the compressed split hero.
Assisted-by: Codex:gpt-5
* docs: hide homepage scroll rail
Preserve Relearn's content scrolling while removing the visible scrollbar beside the landing-page hero.
Assisted-by: Codex:gpt-5
* docs: contain homepage sections within docs column
Prevent landing-page headings, figures, and section grids from widening Relearn's content pane or exposing overflow rails.
Assisted-by: Codex:gpt-5
* docs: remove nested homepage scrollbars
Wrap the quick-start command within its column and suppress component-level scrollbar tracks across the landing page.
Assisted-by: Codex:gpt-5
* docs: remove outdated gallery screenshot
Drop the stale Model Gallery image from the homepage until a current product visual is available.
Assisted-by: Codex:gpt-5
* docs: fix homepage architecture link
Point the homepage CTA at the generated reference/architecture route.
Assisted-by: Codex:gpt-5
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
When a model config has no explicit backend, the model loader greedily
probes every installed backend and binds to the first Load that
succeeds. opus and local-store were the only in-tree backends with no
model artefact to validate, so they accepted anything — an LLM
installed after them could silently bind to the audio codec or the
vector store and then fail at inference with "unimplemented"
(see #9287).
opus now accepts only its own name (what the realtime WebRTC path
sends) or none. local-store namespaces are arbitrary (router caches,
biometrics, user-named stores), so core's StoreBackend now marks
genuine store loads with a store:// prefix on the gRPC model name and
the backend refuses names without it; core and backend ship from the
same release, so the convention upgrades in lockstep.
Also repair the bit-rotted 'make test-stores' bootstrap (the suite
never registered external backends, so BACKENDS_PATH was dead weight)
and add the Load-validation rule to the adding-backends checklist.
Related: #9287
Assisted-by: Claude:claude-fable-5 golangci-lint
Signed-off-by: Richard Palethorpe <io@richiejp.com>
The vibevoice transcription e2e hangs until the go test timeout when
the HF CDN is slow: the ASR Q4_K model is >10 GB, downloadFile capped
every curl attempt at --max-time 600 (needs a sustained ~17 MB/s to
fit), and curl's --retry restarts from byte zero, so no attempt ever
makes forward progress. This killed the job twice on PR #10764 and
previously forced skipping it on release tags (#10567).
Replace the wall-clock cap with stall detection (--speed-limit 1 MiB/s
over --speed-time 120s) and resume from the bytes already on disk with
-C -, retrying from Go because curl does not re-evaluate the resume
offset on its internal retries. Resume against the HF Xet CDN was
verified by killing a transfer mid-flight and confirming the next
invocation appended (114 MB -> 235 MB, GGUF magic intact).
Also parameterize the suite timeout (BACKEND_TEST_TIMEOUT, default
30m) and raise it to 120m for the vibevoice transcription wrapper: a
10 GB download plus 25 specs does not fit in 30m even on a good day,
and the job-level GHA timeout there is already 150m.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vibevoice-cpp): true streaming TTSStream via vv_capi_tts_stream
Replaces the synth-to-tempfile TTSStream hack with a real streaming path:
binds the new vv_capi_tts_stream callback ABI via a single reusable purego
callback (CGO_ENABLED=0-safe, no runtime/cgo), copies each int16 PCM window
into the gRPC results channel after the streaming WAV header.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* test(vibevoice-cpp): real-model streaming integration test with TTFA measurement
Gated behind VIBEVOICE_IT=1, this Ginkgo spec dlopens the engine .so and
drives the exact Go->purego->C TTSStream/TTS path against the real
vibevoice-realtime-0.5B model. It measures time-to-first-audio for the
streaming path versus the batch path and asserts the streaming win:
44-byte WAV header first, >=2 PCM windows, non-silent audio, and
TTFA < total_stream. Without the env var the spec skips so CI and
normal go test are unaffected.
Measured: TTFA 2.38s vs batch deliver-time 39.96s (first audio in 5.9%
of the batch time, ~17x faster), 18 stream chunks, non-silent 24kHz PCM.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(vibevoice-cpp): pin streaming-decoder engine build
Bumps VIBEVOICE_CPP_VERSION to the streaming-decoder engine commit that
adds vv_capi_tts_stream (localai-org/vibevoice.cpp#8). Re-pin to the
merged master commit once that PR lands.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(vibevoice-cpp): re-pin to merged streaming-decoder commit
localai-org/vibevoice.cpp#8 merged to master as 000e372; move the pin
off the PR branch commit onto the merged master commit.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* test(vibevoice-cpp): check writer errors in TTFA report (errcheck)
golangci-lint errcheck flagged the unchecked fmt.Fprintf calls that
print the streaming TTFA headline. Build the report once with
fmt.Sprintf and write it per destination with an explicitly discarded
error, matching the GinkgoWriter reporting idiom used by the other
backend tests.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-fable-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
C++/ggml transcription + speaker diarization + timestamps backend. Purego
dlopens libmoss-transcribe.so (ggml statically linked) from moss-transcribe.cpp
and serves offline AudioTranscription, parsing the [start][Sxx]text[end] output
into segments with nanosecond timestamps. Adds the importer (surfaces in
GET /backends/known), backend-matrix (Linux + Darwin/metal), backend/index.yaml,
and a gallery entry (default q5_k GGUF from mudler/moss-transcribe.cpp-gguf).
Local L0 smoke (go build + go test ./... = 16 pass, golangci-lint 0 issues)
passed against the real libmoss-transcribe.so. The pre-commit coverage gate
(full pkg/core + tests/e2e) could not run in the authoring sandbox (no live
models, port 9090 held); CI must enforce it before merge.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Link the f5-tts library into the crispasr backend so CrispASR's native
F5-TTS runtime (SWivid F5-TTS, 22-layer DiT flow-matching + built-in Vocos
vocoder) is compiled in. The single self-contained GGUF auto-detects as
f5-tts through the session router, so no explicit backend selector is
needed. Add the f5-tts-crispasr gallery entry (cstr/f5-tts-GGUF) and an
env-gated e2e synthesis spec.
F5-TTS is voice-cloning only and has no baked speaker: it clones from a
reference WAV plus its transcript, supplied via the voice/voice_text
options. The gallery description documents this bring-your-own-reference
requirement.
Verified e2e on the pinned engine (278fb79): the GGUF auto-detects as
f5-tts, the reference voice loads, and synthesis produces a valid 24 kHz
mono WAV.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(schema): accept reasoning_content as inbound alias for reasoning
Interleaved-thinking clients (cogito, vLLM/DeepSeek-style) emit reasoning_content
on assistant turns. Accept it as an inbound alias so reasoning survives the
tool-result loop; canonical reasoning wins when both are present. Emission is
unchanged (still reasoning).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(schema): pin interleaved reasoning+tool_calls round-trip
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(openai): pin reachedTokenBudget truncation detection
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(anthropic): add thinking and signature fields to content blocks
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(anthropic): parse inbound thinking blocks into reasoning
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(anthropic): emit thinking blocks with synthetic signature on tool turns
Extract buildAnthropicContentBlocks so non-streaming content assembly is
unit-testable, and prepend a thinking block (with an opaque synthetic
signature) before text/tool_use blocks when the request opts into thinking.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(anthropic): stream thinking_delta and signature_delta before tool_use
Extract anthropicStreamSequence so the streaming block order is unit-testable,
and emit content_block_start(thinking) -> thinking_delta -> signature_delta ->
content_block_stop before the tool_use block sequence when thinking is enabled.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: add interleaved thinking with tool calls guide
Add a features guide describing interleaved thinking: an assistant turn
carrying reasoning and tool_calls together, the reasoning-round-trip
contract (including the reasoning_content inbound alias and Anthropic
thinking blocks with a synthetic signature), per-backend enablement
(reasoning_format for llama.cpp, reasoning_parser/tool_call_parser for
vLLM/SGLang plus the vLLM auto-config hook), a worked request/response
example, and known limitations. Cross-link from model-configuration,
text-generation, and openai-functions.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(backends): pin grpcio-tools to the installed grpcio in runProtogen
runProtogen installed grpcio-tools unpinned, so the protoc it bundles
stamped backend_pb2.py with the newest Protobuf gencode (7.35.0). When a
backend caps the protobuf runtime lower -- vLLM pins protobuf to 6.33.6 --
the import-time guarantee runtime >= gencode fails:
google.protobuf.runtime_version.VersionError: Detected incompatible
Protobuf Gencode/Runtime versions ... gencode 7.35.0 runtime 6.33.6
The backend crashes on `import backend_pb2` before it can serve, which
surfaces to the user as "grpc service not ready". It was mis-reported as a
ROCm/gfx1201 failure in #10718 but is not GPU-specific and affects every
vLLM variant (and any backend that caps protobuf below the latest gencode).
Pin grpcio-tools to the grpcio version the backend already installed --
they release in lockstep -- so the generated gencode stays in step with
the protobuf runtime. Falls back to unpinned when grpcio isn't present.
Closes#10718
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The OIDC verifier was built with a bare oidc.Config{ClientID: ...}, so
go-oidc applied its default of accepting RS256-signed ID tokens only. An
identity provider configured with an EC signing key (e.g. Authentik) issues
ES256-signed tokens, and the callback failed verification with:
failed to verify ID token: oidc: malformed jwt: unexpected signature
algorithm "HS256"; expected ["RS256"]
surfacing to the user as HTTP 500 "failed to fetch user info" (#10677; the
underlying cause became visible after the logging fix in #10679).
Set SupportedSigningAlgs to the standard asymmetric algorithms
(RS256/384/512, ES256/384/512, PS256/384/512, EdDSA). All are verified
against the provider's published JWKS. HS256 is intentionally excluded: it
is symmetric and would validate against the client secret, a different and
security-sensitive trust model.
Tested with a functional spec that signs an ES256 ID token and confirms it
verifies with the configured algorithms and is rejected under go-oidc's
RS256-only default (using oidc.StaticKeySet, no network).
Closes#10677
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Backend log capture into the per-model BackendLogStore (which feeds the
UI "Backend Logs" page and /api/backend-logs) was opt-in and off by
default in single mode, while worker/distributed mode force-enables it
via SetBackendLoggingEnabled(true). There was no CLI flag either, so the
only way to populate the store was the Settings UI toggle - and the page
was silently empty out of the box. Distributed "just worked"; single
mode looked broken.
Default EnableBackendLogging to true in NewApplicationConfig so single
mode matches worker mode. The store is a small in-memory ring buffer, so
the cost is negligible.
Now that the default is on, loadRuntimeSettingsFromFile's usual
"only flip false->true" merge would ignore a persisted false and revert
the UI toggle-off on every restart. There is no env var/CLI flag for
this setting, so an explicit persisted value is now authoritative in
both directions, letting the toggle-off survive a restart.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The image backends call PIL Image.save(request.dst) without a format, so
Pillow infers the encoder from the file extension. The core passes an
absolute staging path ending in .tmp (e.g. /staging/localai-output-*.tmp),
which Pillow can't map to a format, raising "unknown file extension: .tmp"
and crashing the worker right after a successful GPU inference.
Pass format="PNG" explicitly. LocalAI serves generated images as PNG
regardless of the temporary path, so this is always correct and no longer
depends on the extension of the destination the core happens to allocate.
diffusers is the reported backend (#10727); vllm-omni and tinygrad carry
the identical latent crash for any .tmp staging destination.
Closes#10727
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(transcription): honor model-config language/translate, add language form field
The /v1/audio/transcriptions endpoint read only input.Language /
input.Translate from the parsed request, and the request middleware never
populates those from a multipart upload -- nor did it read a `language`
form field. As a result the model config's parameters.language /
parameters.translate (a valid PredictionOptions field under `parameters:`)
were silently ignored, and multilingual models like canary defaulted to
translating into English even when the YAML set language: ru,
translate: false (#10655).
Resolve both with clear precedence: the request form field wins, then any
language on the parsed request, then the model config default. This also
makes the endpoint honor OpenAI's `language` form parameter, which was
not read before.
Applies to both the streaming and non-streaming paths (the resolved
values are built into the shared TranscriptionRequest). Note this ensures
the language/translate flags reach the backend; whether a given engine
acts on them is up to the backend.
Closes#10655
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
A client that keeps polling a model whose load fails (e.g. a backend that
crashes deterministically on init) triggered a fresh backend start on
every request: request -> load -> crash in ~10s -> 500, repeat on the
next poll. Each attempt could leak GPU/CUDA state, and under
LOCALAI_SINGLE_ACTIVE_BACKEND it kept stealing the active slot from
healthy models. The existing loading-coalesce map only dedups
*concurrent* loads, so sequential polls were never covered.
Track load failures per modelID in ModelLoader. After a load fails,
refuse fresh load triggers for that model until a cooldown elapses,
returning a typed ModelLoadCooldownError that the HTTP layer maps to 503
with a Retry-After header. The cooldown grows exponentially per
consecutive failure (base, doubling, capped at 5m) and resets on a
successful load. The coalesced follower-retry of an in-flight burst
bypasses the gate, so a genuinely concurrent burst still gets its one
retry -- only new, independent triggers are refused, matching the
report's "refuse new load-triggers" wording.
Configurable via --model-load-failure-cooldown /
LOCALAI_MODEL_LOAD_FAILURE_COOLDOWN (default 10s, 0 disables), plumbed
through ApplicationConfig and applied unconditionally at startup.
Closes#10719
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
InstallBackend extracted the artifact directly into the target directory
with no pre-clean, so a reinstall overlaid the new files onto the old
ones. Files present in a previous version but absent in the new artifact
(a stale .so, an orphaned package dir) survived and could shadow the new
build at import time -- e.g. an old vllm shared object lingering next to
a freshly pulled one. Only a failed download cleaned the directory.
Stage the download/extraction into a `<name>.install-tmp` dir, validate
run.sh is present, write metadata, then atomically swap it into place
(rename current -> .install-backup, staging -> current, drop backup),
rolling back on failure. This mirrors the atomic swap UpgradeBackend
already performs, so install and upgrade now leave identical on-disk
state with no orphaned files.
Reported as part of #10720.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The nvidia-l4t-cuda-13-arm64 vLLM backend left `vllm` unpinned, so the
prebuilt image drifted onto whatever aarch64 wheel was latest at build
time (0.23.x). On GB10 / DGX Spark (Grace Blackwell, unified memory),
0.23 crashes deterministically during cold model loads with an empty
"Engine core initialization failed" set and pins GPU memory until a host
reboot.
vLLM 0.24.0 carries vllm-project/vllm#45179 ("release cached device
memory under pressure on UMA GPUs during weight loading"), which the
reporter verified fixes the crash on GB10. Pin the L4T requirements to
0.24.0 to match the already-pinned cublas13 build
(requirements-cublas13-after.txt) and keep the image deterministic.
Editing this file also re-triggers the single-arch L4T image build via
the path filter, republishing the gallery image with 0.24.0 (the
single-arch matrix builds again after #10703).
Closes#10722
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Allow llama.cpp model configs to select the backend devices used for offload, matching upstream --device behavior so users can exclude a display or debug GPU.
Signed-off-by: rvmzes <rvmzes@rvmzess-MacBook-Pro.local>
Co-authored-by: rvmzes <rvmzes@rvmzess-MacBook-Pro.local>
* ⬆️ Update leejet/stable-diffusion.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(stablediffusion-ggml): pass chroma knobs via model_args after upstream API change
Upstream stable-diffusion.cpp bb849711 removed the dedicated
chroma_use_dit_mask / chroma_use_t5_mask / chroma_t5_mask_pad fields from
sd_ctx_params_t and now reads them from the generic model_args key=value
spec (parse_key_value_args). Assigning the old struct members broke the
gosd.cpp build. Emit the three options into model_args instead so the
existing chroma controls keep working. Verified by building
libgosd-fallback.so against the pinned upstream commit.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
GitHub Actions refuses to instantiate a matrix that would generate more
than 256 jobs. It does so silently: the job hangs forever at "Waiting for
pending jobs" and the whole run is marked `failure` while every other job
stays green. This is exactly what happened on the v4.6.1 tag build
(run 28786533892): the single-arch build matrix had grown to 268 entries,
so `backend-jobs-singlearch` (and its downstream merge) never produced a
single job, and the release build "failed" with no failing job to point at.
The single-arch list is the one that grows unbounded as backends are added,
so shard it across a fixed number of matrix jobs (SINGLEARCH_SHARDS=4,
~67 entries each today, headroom to ~1020 backends). Each merge shard
`needs:` only its matching build shard, preserving the "merge waits only on
its own build" property that keeps slow CUDA/ROCm builds from gating
multi-arch manifest assembly.
changed-backends.js now emits per-shard matrix/has-* outputs and throws
loudly if a shard ever reaches the 256 limit (telling the maintainer to
bump SINGLEARCH_SHARDS and add matching job blocks) instead of letting
GitHub drop the overflow silently. backend.yml and backend_pr.yml define
the four build + four merge shard jobs; multi-arch and darwin groups are
untouched.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(llama-cpp): cap single-pass embedding batch to fit VRAM
Embedding/score/rerank all decode or pool the whole input in one physical
batch, so EffectiveBatchSize sized the batch to the full context window. For
a large context that makes n_ubatch huge, and the per-device CUDA compute
buffer (forward-graph scratch, ~n_ubatch * n_ctx, NOT split across GPUs)
balloons into multi-GiB: a large-context embedding model then aborts on load
(exitCode=-1) even with plenty of free VRAM. Reproduced with qwen3-embedding-4b
(context 40960 -> n_batch 40960 -> abort) and qwen3-embedding-0.6b
(n_batch 8192); pinning batch:512 avoided it.
This is the same root cause as issue #10485 (a large context turns the batch
into multi-GiB of scratch that must fit on a SINGLE card), but the single-pass
path bypassed the VRAM headroom guard the config layer already had — it
returned the unbounded context as the batch with no GPU awareness.
Make the single-pass batch VRAM-aware: cap it to the largest batch whose
compute buffer fits the per-device VRAM headroom, clamped to
[DefaultPhysicalBatch, ctx], reusing the existing computeBufferBytesPerCell and
headroom-divisor math (no duplication). Unknown per-device VRAM (0) stays
conservative (DefaultPhysicalBatch, not the context) so a detection gap can't
OOM. The GPU is resolved through an injectable package var (config.LocalGPU,
backed by sync.Once-cached xsysinfo detection) so the per-request router call
stays cheap and tests inject a deterministic device. Explicit batch: still
wins. An input longer than the cap can no longer be pooled in one pass — the
accepted tradeoff, since a batch that OOMs the device processes nothing.
Assisted-by: Claude:claude-opus-4-8 golangci-lint go-test
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(config): single-pass batch follows context on unknown VRAM
The single-pass (embedding/score/rerank) batch cap must only shrink the batch
when the per-device VRAM ceiling is KNOWN. On unknown VRAM (CPU-only or a GPU
detection gap) SinglePassBatchForContext returned DefaultPhysicalBatch, which
under-sized the batch below the context — over-trimming score/embed/rerank
inputs (the modelTokenTrim middleware regression) with no OOM benefit on CPU
where the compute buffer lives in system RAM. Return the full context instead,
preserving the original single-pass behavior; the VRAM cap stays a downward
safety that only engages when VRAM is known.
Assisted-by: Claude:claude-opus-4-8 [go-test go-vet]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
When a model is imported without an explicit context_size, the GGUF
importer defaulted the model's context to its full trained window
(n_ctx_train). For long-context models (128k / 256k / 1M) that KV cache
cannot fit a consumer GPU, so the backend aborts on load (exitCode=-1)
even though the model file is perfectly fine. Reproduced live:
gemma-4-26b-a4b-it-qat-q4_0 defaulted to context=262144 and
qwythos-9b-claude-mythos-5-1m to 1048576, both aborting on a 20 GB card.
Instead of chasing the trained max, auto-derive a conservative default:
min(trainedMax, DefaultAutoContextSize=8192). A small model keeps its
trained window; a long-context model caps at 8k and users opt into more
via context_size. This cap applies always, including CPU / unknown-VRAM
hosts, so it never regresses those paths.
Per-device VRAM is used only as a DOWNWARD safety: when a per-device
ceiling is detected (xsysinfo.MinPerGPUVRAM) and even the 8k cap would
not fit it with headroom, step down through candidate contexts to the
largest that fits, floored at DefaultContextSize. When VRAM is unknown
(0) or no GPU is detected we do NOT clamp — the bug is GPU OOM and the
8k cap is already safe, so detection gaps must not shrink the window.
The footprint estimate reuses gpustack/gguf-parser-go's
EstimateLLaMACppRun at a given context with all layers offloaded, taking
the per-device NonUMA VRAM figure. The estimate and VRAM detection are
package vars so tests inject deterministic values. Explicit context_size
always wins (guessGGUFFromFile only acts when it is nil).
Assisted-by: Claude:claude-opus-4-8 [golangci-lint go-test]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The `--generated-content-path` and `--upload-path` defaults were the fixed
shared locations `/tmp/generated/content` and `/tmp/localai/upload`. On any
multi-user host these collide across accounts: macOS routes `/tmp` to the
shared `/private/tmp` for every user, so whichever account starts LocalAI
first creates the parent with 0750 perms and every other account then fails
startup with:
unable to create ImageDir: "mkdir /tmp/generated/content: permission denied"
unable to create UploadDir: "mkdir /tmp/localai/upload: permission denied"
The same happens on Linux once a stale root-owned `/tmp/generated` (e.g. from
a prior `sudo` run) is left behind. This bites the desktop launcher and any
app embedding the raw binary (Wingman, nib-desktop), which start `local-ai
run` with no path flags.
Default both paths under the OS temp dir (`os.TempDir()`, honoring `$TMPDIR`;
already per-user on macOS) namespaced by the current UID
(`TMPDIR/localai-<uid>/...`), so accounts never collide while the paths stay
ephemeral. Wired via new kong vars in main.go so every consumer of the raw
binary inherits the fix. All content subdirs (audio, images) derive from
`GeneratedContentDir`, so they are fixed transitively.
As defense in depth, the launcher also anchors these two paths under its own
per-user data directory (mirroring the #10610 fix for data/config), extracted
into a testable `BuildRunArgs`.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(reasoning): don't persist request-scoped reasoning_effort into model config
When a model sets `reasoning_effort: none` (or any default) in its YAML
without an explicit `reasoning.disable`, ApplyReasoningEffort resolves that
default at request time and sets ReasoningConfig.DisableReasoning on the
request-scoped config copy. The post-load thinking/marker probe then wrote
that request-scoped value back into the loader's persistent config via
UpdateModelConfig, making it look as though the operator had explicitly set
reasoning.disable=true. From then on, per-request `reasoning_effort` overrides
were silently ignored (an explicit operator disable wins over a request
asking to think).
DetectThinkingSupportFromBackend only fills reasoning slots that are still
nil, so a slot already set here came from ApplyReasoningEffort, not the probe.
Snapshot which slots were nil before the probe and only persist those, so the
probe's genuine backend detection is still saved while request-time reasoning
effort never leaks into the persistent config.
Fixes#10622
Signed-off-by: Tai An <antai12232931@outlook.com>
* test(reasoning): cover persist-guard added in this PR, extract for testability
ModelInference's post-probe persistence of ReasoningConfig.DisableReasoning /
DisableReasoningTagPrefill had no test: the guard logic lived inline in a
closure only reachable through a live gRPC backend. Extract it into
persistProbedReasoning (pure refactor, no behavior change) so it can be
exercised directly against a ModelConfigLoader, then add specs covering:
- a probe-filled slot (nil beforehand) gets persisted
- a slot that already carried a request-scoped value (e.g. from
reasoning_effort: none) is left alone, i.e. the #10622 regression stays
fixed
- an operator's explicit persisted disable is preserved when the guard is
false
- the media marker still persists unconditionally
Verified red/green: reverting persistProbedReasoning to the old unconditional
copy fails exactly the two guard specs.
Assisted-by: Claude:claude-sonnet-5 go vet
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(reasoning): ignore os.Remove error in temp file cleanup (errcheck)
Signed-off-by: Tai An <antai12232931@outlook.com>
* chore: empty commit to re-trigger flaky Agent Jobs CI test
Signed-off-by: Tai An <antai12232931@outlook.com>
---------
Signed-off-by: Tai An <antai12232931@outlook.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
Operators need a scrape-friendly signal for agent-turn health (completing,
erroring, cancelled, duration) — log-derived counters proved brittle (ANSI/
timezone parsing, restart gaps). Adds localai_agent_runs_total{agent,outcome}
and localai_agent_run_seconds histogram, recorded at the Chat() response
handoff (single choke point of the local execution path). Lazy meter init,
same pattern as the PII events counter (#10641).
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
The no-models getting-started wizard (`.home-wizard`) rendered
left-aligned instead of centered. `.home-page` is a column flexbox with
the default `align-items: stretch`; a child with `max-width: 48rem`
cannot be stretched past its max-width, so it falls back to the
cross-start (left) edge. The populated home branch never exposed this
because its children are full-width.
Add `margin: 0 auto` to `.home-wizard` so the max-width block centers
horizontally, for both the admin getting-started wizard and the
non-admin no-models hero.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Additive superset of /v1/models that enriches each model entry with the
capabilities it supports plus its input/output modalities
(text / image / audio / video). Clients that only understand /v1/models
are unaffected -- they simply never call the new route.
Audio and video *input* are derived from the model's multimodal limits
(vLLM limit_mm_per_prompt), which no single usecase FLAG expresses. That
gap is exactly why a plain capability list is insufficient and this
enriched endpoint exists: an attachment router can now decide whether an
image/audio/video file can go to the active model directly, or must be
converted/transcribed first.
Capability derivation lives in core/config as the single source of truth
(ModelConfig.Capabilities / InputModalities / OutputModalities /
VisionSupported / ...); the Ollama capability surface now delegates to
it instead of keeping a parallel copy. Vision is gated on
chat/completion capability so a MediaMarker hydrated onto a non-chat
model (e.g. a pure ASR/TTS backend) no longer reports a false vision
capability.
Read-only listing: no new FLAG_* flag, reuses the existing `models`
swagger tag, and intentionally exposes no MCP admin tool (there is
nothing to manage conversationally).
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The OAuth callback discarded the error returned by user-info resolution
before sending the generic 500, so real failures were completely opaque
in the logs: ID-token verification errors (e.g. issuer/audience mismatch
behind a reverse proxy), a missing id_token, claim-parse errors, or a
rejecting GitHub userinfo endpoint all collapsed into
"failed to fetch user info" with nothing logged.
Log the wrapped cause with xlog.Error (provider + error), matching the
code-exchange step just above it. The client-facing message is unchanged,
so no internal detail leaks to the browser.
Refs #10677
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
qwen3-tts-cpp, omnivoice-cpp, acestep-cpp and vibevoice-cpp shipped
rocm-* variants that silently ran on CPU ([Load] backend: CPU). Two
coupled defects:
- The Makefiles passed -DGGML_HIPBLAS=ON, but the vendored ggml only
understands -DGGML_HIP=ON (GGML_HIPBLAS was removed upstream), so the
ggml-hip backend target was never created and no GPU code was built.
- The CMake foreach that links the ggml GPU backends into the module
listed blas/cuda/metal/vulkan but not hip, so even a built ggml-hip
would not have been linked and its static backend registration would
never run.
CUDA users were unaffected because cublas passes the correct GGML_CUDA=ON
and the foreach already links cuda. Mirror the proven llama-cpp hipblas
block (ROCm clang CC/CXX + AMDGPU_TARGETS) and add hip to each foreach.
Upstream picks the best device via ggml_backend_init_best(), so no
runtime flag is needed once HIP is compiled and linked.
Assisted-by: Claude:claude-opus-4-8[1m] [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The ROCm packager copied rocBLAS kernel data (rocblas/library/*.dat) into the
bundled lib/ dir and run.sh pointed ROCBLAS_TENSILE_LIBPATH at it, but the
parallel hipBLASLt data dir (hipblaslt/library/TensileLibrary_lazy_gfx*.dat)
was never packaged and no HIPBLASLT_TENSILE_LIBPATH was set. The bundled
libhipblaslt.so therefore resolved its per-arch kernel data relative to itself,
found nothing, and silently fell back to slow generic kernels, logging:
rocblaslt error: Cannot read "TensileLibrary_lazy_gfx1201.dat": No such file or directory
rocblaslt error: Could not load "TensileLibrary_lazy_gfx1201.dat"
Fix, mirroring the existing rocBLAS handling:
- package-gpu-libs.sh: extract the rocblas data-dir copy into a reusable
copy_rocm_data_dir helper and call it for both rocblas and hipblaslt.
- llama-cpp/turboquant run.sh: export HIPBLASLT_TENSILE_LIBPATH when the
bundled hipblaslt/library dir exists.
The helper takes an optional ROCM_BASE_DIRS override so the copy is unit
testable without a real ROCm install; add a regression test that runs
package_rocm_libs against a fabricated ROCm tree and asserts both data dirs
are bundled.
Note: this bundles whatever gfx*.dat the build image's ROCm provides. If a
given arch's tensile data is absent from the shipped ROCm, that arch still
needs a ROCm bump; the packaging gap itself is fixed for every supported arch.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
POST /models/apply with an empty "id" fetches the attacker-supplied
"url" gallery config directly via http.Client, with no check that the
URL resolves to a public IP. In the default Docker deployment no API key
is configured, so any network-reachable client can coerce LocalAI into
issuing requests to internal services or cloud-metadata endpoints (and
exfiltrate a small slice of the response through the job error message).
Guard the config fetch chokepoints (GetGalleryConfigFromURL and
GetGalleryConfigFromURLWithContext, which back both the /models/apply
worker and gallery installs) with utils.ValidateExternalURL, matching
the protection already applied to the CORS proxy and image/video/audio
download paths. Only plain http(s) URLs are validated; non-network
schemes (huggingface://, github:, oci://, ollama://, file://) resolve to
fixed public services or local files and are left untouched.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The PII EventStore ring buffer is capacity-bound and meant for
recent-audit browsing via /api/pii/events; operators also want a
monotonic, scrape-friendly signal on /metrics — how many
detections/masks/blocks per hour, per origin, and whether the filter
stopped firing after a deploy (silent-failure class).
EventStore.Record is the single choke point every producer already goes
through (request middleware, response scrubbing, MITM proxy
connects/intercepts), so one lazily-initialised counter there covers all
paths without touching any producer:
localai_pii_events_total{kind, origin, action, direction}
Same lazy otel.Meter pattern as core/services/routing/billing, so the
counter lands on the Prometheus-backed global MeterProvider installed by
the monitoring service. No behaviour change; label cardinality is
bounded (enum-like fields only, no pattern IDs or user IDs).
Assisted-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
Realtime sessions previously lazy-loaded each pipeline sub-model (VAD,
transcription, LLM, TTS) on first use, so every cold session paid a
per-request model-load stall and load errors only surfaced mid-stream.
Warm the whole pipeline eagerly and blockingly at session start
(including the voice-gate speaker-recognition model, which an enforced
gate blocks each utterance on; compaction's summary_model stays lazy
since it only runs off the response path):
- Add backend.PreloadModel / PreloadModelByName as the single load path
for every modality (no transcription special-case; backend-omitted
configs are deprecated).
- The realtime session blocks on Model.Warmup and returns a
model_load_error to the client if any stage fails to load;
updateSession warms in the background. Opt out per pipeline with
pipeline.disable_warmup, exposed as a UI toggle via the
config-metadata registry.
Add a LocalAI-native POST /backend/load (and /v1/backend/load) that
pre-loads a model -- expanding realtime pipelines into their sub-models
-- as the inverse of /backend/shutdown. There is one preload engine
(backend.PreloadStages): the realtime Warmup methods, /backend/load and
the --load-to-memory startup flag all use it, so --load-to-memory now
also expands pipeline models and records load-failure traces. Pipeline
sub-model alias resolution is likewise shared
(ModelConfigLoader.LoadResolvedModelConfig). Surface the endpoint
everywhere an admin manages models:
- MCP admin tool load_model (httpapi + inproc clients, safety/catalog
prompts, catalog/dispatch tests).
- "Load into memory" action in the React models UI.
- Swagger regenerated; docs moved to the general backend-monitor page
since it is not realtime-specific.
Fix a Traces UI crash ("json: unsupported value: -Inf"): audio-snippet
RMS/peak now floor at a finite dBFS, and backend-trace data is sanitized
to drop non-finite floats before marshaling. The sanitizer is
copy-on-write -- it runs on every RecordBackendTrace, so containers are
only re-allocated on the paths that actually changed.
Migrate core/http/openresponses_test.go onto the prebuilt mock-backend
the rest of the http suite already uses -- it was the last spec still
pointing at a real HuggingFace model, so it 404'd wherever no vision
backend was built -- and fix its item_reference specs to send the
spec's "id" field instead of "item_id", which the handler never
accepted.
Assisted-by: Claude:claude-opus-4-8 Claude Code
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Two bugs broke OpenAI-style tool calling on the MLX backend (and any
Python backend sharing backend/python/common), reproduced end-to-end on
LocalAI v4.5.5 with the metal-mlx backend and
mlx-community/Qwen3.5-2B-MLX-8bit.
messages_to_dicts left each tool call's function.arguments as the raw
OpenAI-wire JSON string. HuggingFace chat templates (e.g. Qwen3.5)
iterate arguments as a mapping (.items()), so any request whose history
contained a prior assistant tool_calls message failed with HTTP 500
"Generation failed: Can only get item pairs from a mapping." — breaking
every agent loop on its second turn. Decode the string back into a dict
so the template sees a mapping.
split_reasoning returned ("", text) whenever the opening think tag was
absent. Models like Qwen3.5 open the assistant turn already inside
thinking, so the generated text carries only the closing </think>; the
whole chain-of-thought leaked into content. When the opener is missing
but the closer is present, treat everything before the closer as
reasoning.
Adds platform-independent unit tests under backend/python/common
(stdlib-only, no MLX/venv required, following parent_watch_test.py).
Assisted-by: Claude Code:claude-opus-4-8
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vllm): install ROCm vLLM from the AMD wheel index on Python 3.12
The rocm-vllm backend crashed at load with "No module named 'vllm'".
requirements-hipblas-after.txt requested a bare `vllm`, which resolves to
the CUDA-only PyPI wheel; that wheel is unusable on an AMD GPU. vLLM's
prebuilt ROCm wheels live on a dedicated index (https://wheels.vllm.ai/rocm/)
and are published only for CPython 3.12, so on the backend's default 3.10
the installer silently falls back to the CUDA wheel.
Add a hipblas branch to backend/python/vllm/install.sh that pins Python to
3.12 and installs vllm from the ROCm wheel index, hiding the bare-`vllm`
after-file so installRequirements installs only the base ROCm
torch/transformers first and does not pull the CUDA wheel.
Fixes#10642
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(vllm): drop the dead hipblas-after requirement and its hide dance
requirements-hipblas-after.txt (a bare `vllm`) is never installed for
hipblas: installRequirements only adds requirements-${BUILD_PROFILE}-after.txt
when BUILD_TYPE != BUILD_PROFILE, and for hipblas they are equal. So the file
was dead and the install.sh hide/restore of it was a no-op. Remove both. The
hipblas branch already installs vllm explicitly from the ROCm wheel index, so
deleting the bare-`vllm` file also removes a latent CUDA-wheel trap should the
installRequirements gap ever be closed.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): clone a chat into a new conversation (#10645)
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): retry any assistant answer, not just the last (#10645)
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): copy an entire chat to the clipboard (#10645)
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(ui): branch a new chat from any assistant answer (#10645)
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ui): send truncated history on mid-conversation retry (#10645)
Mid-conversation retry regenerated an answer with the downstream turns
still in the model's context. handleRegenerate truncated the DOM history
via updateChatSettings (a scheduled state update), but the synchronous
sendMessage that followed read the stale, pre-truncation history from its
closure to build the outbound API payload. Thread the intended base
history explicitly through sendMessage's options.baseHistory so the
request body matches the truncated view. Backward compatible: the normal
send path (no baseHistory) is unchanged.
Also guard two minor issues in Chat.jsx: the "Branch from here" button now
renders under !isStreaming to match the retry button, and the duplicate
toast only fires when forkChat returns a chat (not on a null result).
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(grpc): self-terminate backend workers when LocalAI dies non-gracefully
Symptom: a backend model-worker subprocess (the per-model gRPC server LocalAI
spawns) can be orphaned and linger — holding VRAM and its listen port — if the
LocalAI process is killed non-gracefully (e.g. a supervisor's graceful-shutdown
grace period elapses and LocalAI is SIGKILLed) before its own teardown runs.
Root cause: LocalAI's graceful teardown (pkg/signals/handler.go installs the
SIGINT/SIGTERM handler; core/cli/run.go registers app.Shutdown ->
ModelLoader.StopAllGRPC -> process.Stop in pkg/model/process.go) only runs when
LocalAI receives a catchable signal and survives long enough to run its
handlers. Backends are spawned via github.com/mudler/go-processmanager v0.1.1,
whose getSysProcAttr() sets Setpgid:true (own process group, so the group can be
signalled) but never PR_SET_PDEATHSIG/Pdeathsig, and exposes no Config field or
option for a caller to inject/extend SysProcAttr. LocalAI fully delegates
spawning to that library (it never builds the exec.Cmd itself), so it cannot set
a kernel parent-death signal at the spawn site. If LocalAI is SIGKILLed, nothing
tells the backend to exit and it is reparented to init.
Fix: add a best-effort, backend-side safety net at the one shared choke point
every out-of-process Go backend routes through — grpc.StartServer / RunServer in
pkg/grpc. On startup it captures getppid() and polls; when the process is
reparented (getppid changes / becomes 1 — the standard POSIX signal the original
parent died) it logs and self-terminates. getppid() reparent detection is
portable (Linux + macOS), unlike Linux-only PR_SET_PDEATHSIG. Toggle via
LOCALAI_BACKEND_PARENT_WATCH (default on; off on Windows) and
LOCALAI_BACKEND_PARENT_WATCH_INTERVAL. This is strictly a backstop alongside the
existing graceful SIGTERM->grace->SIGKILL teardown, which is unchanged.
Scope/limitations: covers Go-based backends (everything using pkg/grpc). The
C++ backends (e.g. llama-cpp) and Python backends do not route through
pkg/grpc and are not covered by this mechanism — they would each need an
equivalent parent-death check (follow-up). The fully general fix is for
go-processmanager to expose SysProcAttr injection so LocalAI can set Pdeathsig
at spawn for every backend regardless of language (suggested upstream follow-up;
out of scope for this LocalAI-only PR).
Test: pkg/grpc/parentwatch_test.go builds a real test -> middle -> grandchild
process tree, lets the middle process exit to orphan the grandchild running the
real watchParentDeath, and asserts it detects the reparent and self-terminates.
Unix-only (build-tagged), runs in CI (Linux).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(process): extend parent-death backstop to C++ and Python backends
The Go parent-death watcher (pkg/grpc/parentwatch.go, commit 772b435d5)
only protects backends that route through pkg/grpc. C++ and Python
backends don't, so the originally-reported case — the llama.cpp gRPC
worker surviving a non-graceful LocalAI death — was still uncovered.
Extend the same best-effort backstop to both languages, reusing the
exact mechanism and semantics:
- capture getppid() at startup, skip if already orphaned (<=1)
- a background thread polls getppid() and self-exits on reparenting
(getppid() != orig || == 1), portable across Linux/macOS, no-op on
Windows
- same env vars: LOCALAI_BACKEND_PARENT_WATCH (default on; falsy
false/0/no/off disable) and LOCALAI_BACKEND_PARENT_WATCH_INTERVAL
(default 2s; accepts Go-style durations like 500ms/2s/1m)
C++: implemented in backend/cpp/llama-cpp (the reported, most-used C++
backend) as a dependency-free header parent_watch.h, wired into
grpc-server.cpp's main() and copied at build time via prepare.sh. C++
backends have no shared server scaffolding, so other C++ backends
(ds4, ik-llama-cpp, privacy-filter, ...) are not yet covered and would
each need the same one-line include+call as follow-ups.
Python: implemented once in the shared common/parent_watch.py and armed
from common/grpc_auth.py's get_auth_interceptors() — the single helper
every one of the 35 Python backends invokes while building its gRPC
server — so all Python backends (and future ones) are covered with no
per-backend edits and no duplicated implementation.
Tests (real process-tree reparent detection, mirroring the Go test):
- backend/cpp/llama-cpp/parent_watch_test.cpp (via run-unit-tests.sh)
- backend/python/common/parent_watch_test.py (python -m unittest)
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* fix(backends): make backend install ops idempotent unless forced
POST /backends/apply hardcoded force=true through
LocalBackendManager.InstallBackend, so applying an already-installed
backend re-downloaded and re-extracted the whole artifact every time.
API clients that ensure a backend exists at startup paid a full OCI
image pull on every boot.
Backend install ops now default to non-forced — an installed, runnable
backend short-circuits (the orphaned-meta reinstall path in
InstallBackendFromGallery is preserved) — and reinstall stays available:
- ManagementOp gains a Force field; the local manager passes it through
instead of hardcoding true.
- /backends/apply accepts an optional "force" boolean in the body.
- The React UI install route keeps forcing, since its button doubles as
the explicit "Reinstall backend" action.
Distributed installs already behaved this way (workers skip when the
binary exists unless force is set); this aligns single-node behavior.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(backends): don't force-reinstall LOCALAI_EXTERNAL_BACKENDS on boot
The startup loop for LOCALAI_EXTERNAL_BACKENDS runs
InstallExternalBackend for each listed backend on every boot, and its
gallery-name path hardcoded force=true — so every start re-downloaded
and re-extracted each listed backend's OCI image even when it was
installed and runnable. Supervising apps that list several backends
paid several full OCI pulls per launch.
Give InstallExternalBackend an explicit force parameter (it only
affects the gallery-name fallback; URI installs always write) and pass:
- false from the boot loop and `local-ai backends install` (idempotent
ensure — `backends upgrade` is the refresh path),
- op.Force from the local manager's external-URI op,
- the request's force on the worker install path and true on its
upgrade path (behavior unchanged).
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Newest cloud reasoning models reject two parameters the cloud-proxy
backend currently sends:
- Anthropic (claude-opus-4-x) and OpenAI (gpt-5.x) return 400 when
temperature is present: "'temperature' is deprecated for this model".
OpenAI-compatible clients typically send only the server-side DEFAULT
sampling values rather than user intent, so the translators now forward
neither temperature nor top_p and let the upstream apply its own
defaults.
- OpenAI gpt-5.x rejects max_tokens ("Unsupported parameter: 'max_tokens'
... Use 'max_completion_tokens' instead"). The OpenAI translator now
serializes the token limit as max_completion_tokens, which current
chat-completions models accept.
Verified live against claude-opus-4-8, gpt-5.5 and gemini-3.1-pro
(Gemini OpenAI-compat endpoint). Tests updated to the new contract.
Assisted-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
* fix(distributed): don't let a dead worker pin the model-load advisory lock
In distributed mode a chat request could fail with:
failed to route model with internal loader: routing model ...:
loading model ...: advisorylock: acquiring lock <id>:
ERROR: canceling statement due to lock timeout (SQLSTATE 55P03)
Root cause is two independent defects in the cross-replica model-load path:
1. SmartRouter.Route holds a per-model PostgreSQL advisory lock for the whole
cold-load sequence, which includes installBackendOnNode -> InstallBackend,
a NATS request-reply with a 15m deadline (DefaultBackendInstallTimeout) that
ignored ctx. When the chosen worker died mid-install, the holder sat on the
lock for up to 15m. The detached loadCtx (WithoutCancel) had no deadline, so
nothing capped the hold.
2. The acquiring statement, pg_advisory_lock(), is subject to any deployment
global lock_timeout. A common operator setting (e.g. 10s) aborts the wait
with SQLSTATE 55P03, so every other replica's request for that model hard
-errored instead of waiting for the in-progress load and reusing it. For the
~15m window the model was effectively unroutable.
Fixes:
- advisorylock.WithLockCtx (postgres): SET lock_timeout = 0 on its dedicated
connection (RESET before it returns to the pool) so the Go context, not a
deployment-wide GUC, governs how long we wait. Waiters now block and then
re-check, reusing the model another replica just loaded.
- SmartRouter: bound the detached loadCtx with a single ModelLoadCeiling so the
lock is always released in bounded time even if a sub-step wedges. Default is
the configured backend.install deadline + 10m (staging + LoadModel margin),
so a legitimately slow load is never cut.
- installBackendOnNode: use singleflight.DoChan + select on ctx.Done() so the
install wait honors cancellation; the ceiling can then actually free a caller
pinned behind a dead worker. The shared install still coalesces via
singleflight.
Reproduced both defects as failing tests first (a real 55P03 against a
testcontainer with a short lock_timeout; a wedged install that blocks Route)
and confirmed green.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* fix(distributed): bound advisory-lock wait instead of disabling lock_timeout
Setting lock_timeout = 0 to override a deployment's short global lock_timeout
meant "wait forever" server-side. Safe for SmartRouter.Route (its loadCtx now
carries the model-load ceiling) but unsafe for the schema-migration callers
that pass context.Background(): a holder whose session never releases would
hang them indefinitely.
Derive the server-side lock_timeout from the caller's context instead: its
remaining budget plus a margin (so the Go context's cancellation still wins
with a clean error and the server bound is only a backstop), or a finite
30m backstop when the context has no deadline. Never zero - "wait forever"
is no longer possible, while a deployment's hostile short lock_timeout is
still overridden so legitimate cross-replica waits don't fail with 55P03.
Added a spec proving a deadline-less waiter gives up at the (shrunk) backstop
rather than hanging.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
fix(vllm): non-streaming tool-call regression after #10351 (native_streaming is a capability flag, not a state flag)
#10351 introduced native streaming via `parser.extract_tool_calls_streaming`
and gated the post-loop `extract_tool_calls` block on `native_streaming and
not native_streaming_error`. That works for streaming requests, but for
non-streaming requests the same flag is still True (it only means "the
parser can stream", not "we actually streamed"), so the block was skipped
and the `elif` cleared `content = ""` — the tool call was silently lost.
Symptom: non-streaming chat.completions with `tools=[...]` returns
`finish_reason: "stop"` with `content: ""` and no `tool_calls`. Streaming
requests are unaffected.
Fix: gate both branches on `streaming` too, so the extract_tool_calls
block runs for non-streaming requests (and for streaming requests that
fell back to the buffered path).
Reproduction (vLLM 0.24, Qwen3-Coder-Next-NVFP4, qwen3_coder parser):
curl -s -X POST http://localhost:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"coder","stream":false,
"messages":[{"role":"user","content":"7*8 via calc"}],
"tools":[{"type":"function","function":{"name":"calc",
"parameters":{"type":"object",
"properties":{"expression":{"type":"string"}}}}}]}'
Before: finish_reason: "stop", content: "", tool_calls: []
After: finish_reason: "tool_calls", tool_calls[0].function.name: "calc"
Streaming path re-verified in the same setup: delta.tool_calls arrives
token-by-token, finish_reason: "tool_calls", no raw XML in content.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
* fix(hipblas): symlink amdgpu.ids so ROCm backends find the ASIC ID table
ROCm's bundled libdrm_amdgpu looks up the GPU ASIC ID table at a
hardcoded fallback path, /opt/amdgpu/share/libdrm/amdgpu.ids, which is
only populated by AMD's full amdgpu-install (graphics/DKMS) stack. The
hipblas image is compute-only and doesn't have it, so every model load
logs "No such file or directory" and the GPU can't be identified.
Symlink it to the equivalent file already shipped by Ubuntu's
libdrm-amdgpu1 package.
Fixes#10624
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(hipblas): correct amdgpu.ids source package name in comment
Verified against the real rocm/dev-ubuntu-24.04:7.2.1 image with
hipblas-dev/hipblaslt-dev/rocblas-dev installed: /usr/share/libdrm/amdgpu.ids
is owned by libdrm-common, not libdrm-amdgpu1 as the comment said.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The launcher starts the server with run --models-path/--backends-path but
leaves --data-path and the dynamic config dir unset, so the server falls
back to its /data and /configuration defaults.
is kong.ExpandPath("."), i.e. the launcher process CWD
(commonly the user's home root), producing ~/data and ~/configuration
outside ~/.localai and an agent-pool stateDir under ~/data.
Pass --data-path and --localai-config-dir explicitly, rooted at the
launcher's own data directory (GetDataPath() -> ~/.localai), so data and
config stay consistent with --models-path/--backends-path.
* fix(watchdog): don't log optional Free() as an error when backend returns Unimplemented (#10602)
When the watchdog evicts a model, deleteProcess calls the backend's gRPC
Free() to release VRAM before stopping the process. Free is optional:
backends that don't override it -- the generated UnimplementedBackendServer
stub, many Python/external backends, or a federation proxy in distributed
mode -- return gRPC Unimplemented. That is expected, not a failure: VRAM is
reclaimed when the local process is stopped, or by the remote unloader for
remote backends. Logging it as "WARN Error freeing GPU resources" made a
benign, optional RPC look like a fault (the alarming line in #10602, seen
in distributed mode where the model is remote and Free hits a stub).
Treat gRPC Unimplemented from Free() as a no-op logged at Debug; genuine
failures still Warn. Free() is still attempted for every backend, so any
backend that does implement it is unaffected.
Add a reusable grpcerrors.IsUnimplemented helper following the package's
existing code-based detection idiom (prefer the typed status code, fall
back to the message across non-gRPC boundaries), with table tests.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
* fix(watchdog): log a non-Unimplemented Free() failure at error level
Per review: now that the expected gRPC Unimplemented case is split out and
logged at Debug, any remaining Free() error is a genuine failure to release
VRAM, so surface it at error level instead of warn.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
---------
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
The Q4_K_M quant degraded tool-call reliability for LFM2.5-8B-A1B.
Switch the gallery entry to the Q8_0 GGUF (sha256 verified via HF
x-linked-etag) while keeping the native jinja tool-parsing config.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
The backend.proto AudioTranscriptionLive bidirectional streaming RPC added
new required trait items (AudioTranscriptionLiveStream + audio_transcription_live)
on the generated Backend trait. The kokoros (TTS) backend did not implement
them, breaking its release build with E0046 (missing trait items).
kokoros is text-to-speech and has no live-ASR support, so stub the method to
return UNIMPLEMENTED, mirroring the existing audio_transcription_stream stub.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
LocalAI enables a cross-request prompt-prefix cache (cache_reuse, see
core/config/serving_defaults.go) so repeated prefixes — system prompts,
RAG context, agent scaffolds, multi-turn chat — are not reprocessed every
turn. For sliding-window-attention (SWA) models (Gemma 2/3, Cohere2,
Llama 4, ...) this silently does nothing: llama.cpp defaults to a reduced
SWA KV cache sized to the sliding window, and that reduced cache cannot
preserve a prompt prefix across requests, so every turn reprocesses the
whole prompt anyway.
llama.cpp's --swa-full (params.swa_full, already wired through the
LocalAI llama.cpp backend's `swa_full` option) keeps the full KV cache so
the shared prefix is reused. Enable it automatically, but only for models
that are actually SWA: detection reads the gguf-parser-normalized
`<arch>.attention.sliding_window` metadata (which also applies llama.cpp's
family rules, e.g. Phi-3 → not SWA), right where the GGUF is already
parsed for defaults. It is never applied to dense models (pure memory
waste) and never overrides an explicit user `swa_full`/`n_swa` choice.
Tradeoff: the full SWA cache scales with context_size, so it costs more
memory at large contexts — hence the SWA gating and the documented
`swa_full:false` opt-out.
Assisted-by: Claude:claude-opus-4-8 [Claude Code] golangci-lint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(watchdog): persist a UI-saved Check Interval across restarts (#10601)
The watchdog Check Interval saved via /api/settings reverted to 500ms on
every restart, while the idle/busy timeouts persisted correctly.
Root cause: NewApplicationConfig baseline-defaulted WatchDogInterval to
500ms, whereas the idle/busy timeouts default to 0. The startup loader
(loadRuntimeSettingsFromFile) applies a persisted runtime_settings.json
value only when the field is still at its zero default - its heuristic
for "this wasn't set by an env var". Because the interval was always
500ms at that point, the loader never read the persisted value back, so
the saved interval was silently discarded on each boot.
Fix: drop the non-zero baseline default so the interval behaves like the
sibling timeouts (0 = unset). The effective 500ms default is now supplied
at the watchdog layer: WithWatchdogInterval ignores a non-positive value
so DefaultWatchDogOptions' 500ms is preserved (and a 0 interval can never
turn the watchdog loop into a busy spin). Also mirror the interval in the
live config file watcher alongside idle/busy, and report the real 500ms
default (not the stale "2s") from ToRuntimeSettings.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Stapling only the dmg leaves the LocalAI.app bundle with no embedded
notarization ticket. Gatekeeper then falls back to an online notarization
check on first launch, so the app fails to open on a Mac that is offline or
behind a firewall, or once it has been copied out of the dmg — while it keeps
working on the (online) build host, which masks the problem.
Notarize and staple the .app before packaging it into the dmg so the bundle
verifies offline. Adds a `notarize-app` subcommand to
contrib/macos/sign-and-notarize.sh (zips the bundle for notarytool, then
staples + validates) and invokes it from dmg-launcher-darwin. Stays a no-op
when notary secrets are unset, so unsigned local/fork builds are unaffected.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: mudler <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
PR #10589 changed repo-root HuggingFace URI imports to name the model after
the selected GGUF file rather than the repository. The Open Responses API
integration test still requested the old repo-derived name
("Qwen3-VL-2B-Instruct-GGUF"), so every request 404'd on an unknown model and
the suite has failed on master since 1a4f68ed4.
Update testModel to the name the importer now registers for the default
q4_k_m quant ("Qwen3-VL-2B-Instruct-Q4_K_M") so the specs resolve the model
again. The #10589 behaviour change is intentional; only the stale test needed
updating.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Importing a model from a local directory (e.g. a HuggingFace checkout or an
LM Studio store) via a file:// URI produced a config whose model field kept
the scheme verbatim, e.g. model: file:///Users/u/.../Qwen3-4bit. The mlx and
vllm backends treat that field as a HuggingFace repo id or local path and
reject the file:// form with "Repo id must be in the form 'repo_name' or
'namespace/repo_name'", so the model imported fine but failed to load (issue
#7461).
Add a shared LocalModelPath helper that reduces a file:// URI to the bare
filesystem path it points at and leaves HuggingFace/HTTP URIs untouched, and
route the mlx, vllm, transformers and diffusers importers (all of which pass
details.URI straight into the model field for from_pretrained-style loading)
through it.
Cover the helper directly plus end-to-end file:// import specs for the mlx and
vllm importers.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
fix(functions): avoid quadratic-time debug logging in CleanupLLMResult/ParseFunctionCall
The streaming chat path (core/http/endpoints/openai/chat_stream_workers.go)
calls CleanupLLMResult / ParseFunctionCall once per delta chunk with the
*full accumulated* LLM result so far. Both functions xlog.Debug the entire
argument on entry and exit, so a single N-chunk stream emits roughly
chunk_size * N^2 bytes of debug output.
Under LOG_LEVEL=debug this was observed in a recent SGLang-via-LocalAI
session on a DGX Spark host (about 50K tokens, long streaming generation)
to drive container logs to ~96 GiB, which interacted with the streaming
hot loop on the same filesystem and contributed to a host-wide hard hang
once disk pressure built up. Workaround was setting LOG_LEVEL=info, but
the quadratic shape remains a foot-gun for anyone intentionally enabling
debug.
Replace the four result-content debug arguments with len(...) plus a
fixed-size head (200 bytes via a new truncForLog helper), bounding per-
call output to a constant. The debug signal stays useful: the first 200
chars are enough to identify which generation is in flight, and the
length lets you observe growth without paying for the payload itself.
No API change. No behaviour change for LOG_LEVEL != debug.
Signed-off-by: Poseidon <philipp.wacker@ibf-solutions.com>
Co-authored-by: Poseidon <philipp.wacker@ibf-solutions.com>
When importing a HuggingFace GGUF model from a repository-root URI (no file
component, e.g. hf://owner/repo) with the Model Name field left blank, the
importer named the model after the repository (filepath.Base(details.URI))
instead of the GGUF file it actually selected from the repo listing (issue
#10587).
Track whether the user supplied an explicit name; the URI base is now only a
fallback. In the HuggingFace branch, once the model group is picked, re-derive
the name from the selected GGUF via a new modelNameFromShardGroup helper that
uses ShardGroup.Base minus the .gguf extension. For sharded models this yields
a clean logical name (e.g. Qwen3-30B-A3B-Q4_K_M) rather than a shard filename
like ...-00001-of-00002. An explicit name preference still always wins, and the
.gguf/URL/OCI paths are unchanged.
Add network-free unit specs covering name-from-GGUF, clean-name-from-shard-base,
and explicit-name precedence, and update the live integration specs that had
encoded the previous repo-name behaviour.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>
fix(openai): stop max_tokens streaming retry loop on reasoning models
When a thinking model spends its entire max_tokens budget on the reasoning
block, the C++ autoparser clears the raw Response and delivers reasoning-only
ChatDeltas (no content, no tool calls). ComputeChoices' empty-response retry
then fires and regenerates from scratch up to maxRetries times, each
re-consuming the whole budget, instead of terminating with finish_reason
"length" (issue #9716).
Add a reachedTokenBudget helper and suppress both the built-in and
caller-driven retries when the completion count has reached the configured
max_tokens ceiling. Report finish_reason "length" instead of "stop" in the
streaming and non-streaming chat paths when the budget was exhausted.
Adds a deterministic regression test that counts backend invocations
(previously 6, now 1) plus boundary tests for the helper.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Dennisadira <dennisadira@gmail.com>
* feat(realtime): EOU-driven semantic_vad turn detection
Add a `semantic_vad` turn-detection mode to the realtime API that feeds
the transcription model live and decides "the user finished speaking"
from the `<EOU>` end-of-utterance token rather than from silence alone.
When EOU fires the turn commits immediately (~0.3s); otherwise it falls
back to an eagerness-scaled silence threshold (low/med/high = 8/4/2s).
Plumbing, bottom to top:
- proto: `AudioTranscriptionLive` bidirectional RPC (config-first oneof,
mono float PCM @16k, ready-ack / Unimplemented degrade signal) plus
`TranscriptResult.eou` for the unary retranscribe gate.
- pkg/grpc: client/server/base/embed scaffolding for the bidi stream,
modeled on AudioTransformStream; release stream conns on terminal Recv.
- parakeet-cpp: live transcription RPC with per-C-call engine locking
(one live stream per turn, finalize+free at commit); bump parakeet.cpp
to ABI v5 — incremental StreamingMel (no more quadratic per-feed mel
recompute that delayed EOU on long turns) and the <EOU>/<EOB> split;
strip the literal <EOU>/<EOB> from offline text and set Eou.
- core/backend: LiveTranscriptionSession wrapper + pipeline
`turn_detection:` config block (type/eagerness/retranscribe).
- realtime: semantic_vad integration — live input captions streamed as
transcription deltas while the user speaks, EOU-immediate commit with
eagerness fallback, optional retranscribe gate (batch re-decode must
also end in <EOU> to confirm), clause synthesis off the LLM token
callback, and per-turn live-transcription / model_load telemetry.
- UI: show the realtime pipeline components as a vertical list.
Docs and tests included; opt-in via the pipeline YAML or per-session
`session.update`. Non-streaming STT backends degrade to silence-only.
Assisted-by: Claude Code:claude-opus-4-8 [Read] [Edit] [Write] [Bash]
Assisted-by: Claude Code:claude-fable-5 [Read] [Edit] [Bash]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(realtime): explicit formally-verified state machines + parakeet streaming driver
The realtime API had several implicit state machines whose state was inferred
from scattered booleans, channels, and five separate mutexes, leaving
illegal/inconsistent states reachable. Make them explicit and keep the
implementation in step with a formal design; rework the parakeet streaming
backend along the same lines.
Realtime state machines (M1-M5). Each is a sealed sum-type State/Event/Effect
with a total, pure Next(state,event)->(state,[]effect) behind a single-writer
Coordinator:
M1 conncoord connection lifecycle: VAD toggle + once-only teardown
(replaces vadServerStarted + a `done` channel closed from
two sites).
M2 turncoord turn detection: collapses speechStarted and the live-stream
"turn open" flag into one state, so discardTurn can no longer
desync them and suppress the next onset.
M3 respcoord response coordination: serializes the dual-writer
start/cancel so at most one response is live; one
response.done per response.create.
M4 compactcoord conversation compaction: single-flight (replaces the
`compacting atomic.Bool` CAS).
M5 ttscoord TTS pipeline: open->closing->closed, idempotent wait(),
rejects enqueue-after-close (was a silent drop).
The Coordinator/Sink/Next plumbing — only the sealed types and Next differed
per machine — is extracted once into core/http/endpoints/openai/coordinator as
a generic Coordinator[S,E,F]; each machine keeps its public API via type
aliases, so no sink, call-site, or test moved.
Hierarchy. session_lifecycle.fizz models M1 as the parent region with its
children (M2/M3/M4) as one statechart and asserts ChildrenDieWithParent (conn
torn => all children terminal, none start after teardown). respcoord and
compactcoord gain an absorbing Terminated state + Shutdown event; conncoord's
teardown drives the children terminal. This closes a compaction teardown gap: a
fire-and-forget compaction could outlive a torn session — compactionSink now
takes a session-scoped cancellable context + WaitGroup and joins the in-flight
summarize+evict on shutdown.
Formal verification. formal-verification/ holds one authoritative FizzBee spec
per machine plus the composition spec, each with an always-assertion and a
documented one-line edit that makes the checker fail (verified non-vacuous).
scripts/realtime-conformance.sh is fail-closed: all Go conformance suites under
-race AND a model-check of every .fizz spec; a missing FizzBee is a hard error
(only the loud REALTIME_CONFORMANCE_SKIP_FIZZBEE=1 bypasses it, never in CI).
FizzBee is pinned by sha256 and installed via scripts/install-fizzbee.sh into
.tools/ (gitignored). Wired as make test-realtime-conformance, a CI workflow,
and a pre-commit path filter. Go conformance tests are Ginkgo/Gomega (per the
repo's forbidigo lint): transition tables + fixed-seed property walks +
concurrent/-race specs, no rapid dependency. Design map:
docs/design/realtime-state-machines.md.
Parakeet streaming backend. The same treatment applied to the parakeet-cpp
streaming paths:
- AudioTranscriptionStream returns codes.Unimplemented for non-streaming models
instead of decoding offline and emitting it as one delta + final. A client
that asked for streaming learns the model cannot stream rather than receiving
a batch result shaped like a stream. New grpcerrors.StreamTranscriptionUnsupported
carries that signal; the HTTP /v1/audio/transcriptions stream path surfaces it
as an SSE error event. Mirrors AudioTranscriptionLive, which already did this.
- utteranceBoundary (boundary.go): a single definition of the end-of-utterance
latch, replacing three open-coded finalEou toggles. Modelled as a two-valued
type so illegal states are unrepresentable.
- Shared decode driver (driver.go): streamFeedResult (one per-feed event) +
feedChunk (hides the ABI v4 JSON vs text-only split) + feedSlices + flushTail.
The feed loop is written once.
- AudioTranscriptionLive becomes a bidi adapter: it streams the per-feed
{delta,eou,eob,words} the realtime turn detector consumes and a terminal
FinalResult carrying only Text. Segments/duration/eou are offline-only and no
longer produced (nor read) on the live path; liveTraceState drops the terminal
eou and keeps the per-feed eou_events count.
- AudioTranscriptionStream + streamJSON merge into one driver-based function;
streamSegmenter is generalized to the unified event with a text-only fallback
that preserves the legacy (no-words) library's per-utterance segmentation.
Verified: build/vet/gofumpt clean, golangci-lint 0 issues, all coordinator and
parakeet packages under -race, the fail-closed conformance gate green, and
make test-realtime (12 e2e WS+WebRTC).
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
OpenAI wire format carries `function.arguments` as a JSON-encoded string,
but chat templates (e.g. Qwen3-Coder) iterate over it as a mapping. The
vllm backend already parses arguments before applying the chat template
(PR #10256); this mirrors that fix in the sglang backend.
Without this fix the second turn of any tool-using session (assistant
returns tool_calls, user posts `role:"tool"` result, model is invoked
with arguments still as a string) crashes inside transformers' Jinja
chat-template rendering with:
TypeError: Can only get item pairs from a mapping.
File ".../transformers/utils/chat_template_utils.py", in render_jinja_template
File ".../jinja2/filters.py", in do_items
raise TypeError("Can only get item pairs from a mapping.")
Reproduced on `lmsysorg/sglang:v0.5.14` via LocalAI v4.5.4 with
`saricles/Qwen3-Coder-Next-NVFP4-GB10` (W4A4 NVFP4 / compressed-tensors)
on NVIDIA DGX Spark (GB10, sm_121).
After the patch, a tool-call roundtrip (assistant tool_calls -> tool
result -> assistant final answer) returns http=200 with the expected
follow-up content; no behaviour change on requests that don't carry
tool_calls.
Signed-off-by: Poseidon <philipp.wacker@ibf-solutions.com>
Co-authored-by: Poseidon <philipp.wacker@ibf-solutions.com>
chore(recon): re-pin voice/face-detect to squashed release commits
The voice-detect.cpp and face-detect.cpp engine repos were squashed to a single
release commit, which orphaned the previous pins (voice 3d51077, face 06914b0).
Re-pin to the new single-commit SHAs (voice 1db1759, face e22260d).
These also fold in a real correctness fix: the persistent graph-cache fingerprint
now includes op_params, so two structurally identical GGML_OP_CUSTOM graphs (a
blocked 3x3 vs a blocked 1x1 strided conv) can no longer false-hit the cache and
replay the wrong kernel. voice CI was failing test_blocked/conv1x1_s2 with an
out-of-bounds write on the GGML_NATIVE=OFF build; both engine repos are now green
and WeSpeaker embed parity is 1.0 vs golden.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Installing large backend images (e.g. vLLM/vLLM-omni, several GiB) over
the Web UI could fail with "failed to download layer 0: unexpected EOF"
when a single connection to the registry dropped mid-stream. The whole
install then failed with no recovery, and since the download is not
resumable, retrying from the UI restarted from zero and usually hit the
same blip again - so users saw it as a consistent, size-correlated
failure (issue #10577).
The registry transport already retries manifest/digest fetches via
defaultRetryPredicate (GetImage/GetImageDigest), but the per-layer data
stream in DownloadOCIImageTar bypassed it entirely: layer.Compressed()
+ xio.Copy ran exactly once.
Extract the per-layer copy into downloadLayerToFile, which retries on the
same transient errors (unexpected EOF, EOF, EPIPE, ECONNRESET, connection
refused) with exponential backoff, truncating any partial data before
each retry. Non-retryable errors and context cancellation still fail
fast.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
On darwin arm64 the fish-speech editable install (pip install
--no-build-isolation -e) compiles the transitive `tokenizers` Python
package's Rust extension from source, because there is no prebuilt
manylinux wheel for that platform (Linux builds never compile it, so this
only breaks on macOS). The pinned tokenizers crate fish-speech's stack
resolves to contains a `&T` -> `&mut T` cast that the macOS CI runner's
newer Rust toolchain rejects via the now-deny-by-default
`invalid_reference_casting` lint:
error: casting `&T` to `&mut T` is undefined behavior ...
error: could not compile `tokenizers` (lib) due to 1 previous error
ERROR: Failed building wheel for tokenizers
This failed the fish-speech darwin/metal (mps) backend image build in the
v4.5.5 release CI while all Linux variants built fine.
Fix: export RUSTFLAGS with `-A invalid_reference_casting` (appended to any
existing value, not clobbering) before installRequirements so the
unchanged third-party crate compiles as it did under the older toolchain.
Version-agnostic and harmless on Linux, where no Rust compile happens.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(launcher): resume flaky downloads, drop redundant percent, fit dialogs
The binary upgrade/download flow had three rough edges:
- The status label printed "Downloading... N%" right next to a progress
bar already showing the percent. Replace it with a human-readable byte
readout ("Downloading... 12.3 MB / 45.6 MB").
- A failed download (GitHub releases are flaky) had no recourse and always
restarted from byte 0. Stream to "<dest>.part" and resume via a
"Range: bytes=N-" request (handling 206/200/416), renaming to the final
path only after checksum verification; on checksum failure the file is
discarded so the next attempt starts clean. Add a Retry button that
appears on failure and resumes from the partial file.
- Progress/install dialogs were hardcoded to oversized dimensions, leaving
a blank gap below "View Release Notes". Size each window to its content
with a sane minimum width.
Also unify the three near-identical download-progress popups into one
Launcher.showDownloadProgressWindow helper (and delete a dead unused copy
in ui.go) so the behaviour stays consistent across every entry point.
The progress callback now reports (downloaded, total) byte counts instead
of a single fraction. Resume/retry behaviour is covered by httptest-backed
unit tests in release_manager_test.go.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(launcher): resolve latest version via redirect to dodge GitHub API 403
On a fresh Linux start with no LocalAI installed, the download failed with
"failed to fetch latest release: status 403". The cause is the unauthenticated
api.github.com rate limit (60 requests/hour, per IP): on shared/NAT/CGNAT/cloud
addresses it is exhausted almost immediately and every request 403s.
Resolve the latest version by following the github.com "releases/latest"
redirect instead, reading the tag from the final ".../releases/tag/<tag>" URL.
That endpoint is not subject to the API rate limit. Only the version is ever
consumed by callers, so the tag is sufficient. The JSON API is kept as a
fallback, now honoring GITHUB_TOKEN and reporting rate-limit 403/429 clearly
instead of an opaque status code.
Covered by an httptest-backed unit test that asserts the redirect path is used.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The kokoro install.sh ends with `python -m spacy download en_core_web_sm`.
spaCy's CLI imports typer -> click, so click must be present at that point.
On the intel build profile, install.sh adds `--upgrade --index-strategy=unsafe-first-match`
against the Intel pip index. With that resolution strategy, click is not
resolved/installed, so the spacy CLI import fails with:
ModuleNotFoundError: No module named 'click'
make: *** [Makefile:3: kokoro] Error 1
Other profiles (cpu/cublas) pull click in transitively and build fine; only
the intel profile breaks. This surfaced in the v4.5.5 release CI as the
gpu-intel-kokoro backend image build failure.
Make click an explicit dependency in the base requirements.txt (installed for
every profile) so it is always present before `python -m spacy download` runs,
regardless of index resolution. Unpinned: spacy constrains the version.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(voice-detect): add Go purego backend for voice-detect.cpp
Add backend/go/voice-detect implementing the Backend gRPC voice subset
(VoiceEmbed/VoiceVerify/VoiceAnalyze) over libvoicedetect.so via purego,
mirroring the parakeet-cpp / omnivoice-cpp backends.
The flat voicedetect_capi C ABI is dlopen'd cgo-less; malloc'd string and
float-vector returns are owned by Go and released through the matching capi
free functions, with the per-ctx last error surfaced into Go errors. Calls are
serialized via base.SingleThread since the C context is not reentrant.
Proto field mapping:
- VoiceEmbed: VoiceEmbedRequest.audio (path) -> embed_path -> Embedding+Model.
- VoiceVerify: audio1/audio2 + threshold (<=0 falls back to the
verify_threshold option, default 0.25) -> verify_paths -> verified/distance/
threshold/confidence/model/processing_time_ms.
- VoiceAnalyze: audio (path) -> analyze_path_json; the JSON age/gender/emotion
document maps to a single VoiceAnalysis segment (start/end 0; gender "label"
-> dominant_gender with the remaining float scores as the gender map; emotion
label/scores -> dominant_emotion/emotion).
The Makefile pins voice-detect.cpp to 47546430, clones+builds libvoicedetect.so
with ggml static-linked (PIC, GGML_NATIVE off) so dlopen needs no external
libggml/libvoicedetect; ldd on the artifact shows only system libs. Ginkgo
tests cover option parsing and analyze-JSON mapping; embed/verify smoke specs
gate on VOICEDETECT_BACKEND_TEST_MODEL + VOICEDETECT_BACKEND_TEST_WAV.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* feat(voice-detect): wire backend into index, gallery and build
Register the voice-detect.cpp speaker-recognition + voice-analysis
backend (added in Voice-INT-A) into LocalAI's distribution surfaces,
mirroring the ced backend (the closest mudler C++/ggml audio analogue):
- backend/index.yaml: add the &voicedetect meta-backend (capabilities
platform map, no top-level uri) plus the full set of concrete per-arch
image entries (cpu/cuda12/cuda13/metal/rocm/sycl/vulkan/l4t and the
-development variants). Referential integrity audited - every alias
target resolves.
- gallery/index.yaml: add 5 model entries on backend voice-detect -
ECAPA-TDNN, WeSpeaker ResNet34, 3D-Speaker ERes2Net, CAM++ and the
wav2vec2 age/gender/emotion analyze model. The engine architecture is
read from GGUF metadata (voicedetect.arch) at load. GGUF artifacts are
not yet published: each files: entry points at the intended
mudler/voice-detect-gguf location with a TODO to fill sha256 after
upload (no fabricated hashes).
- .github/backend-matrix.yml: add the linux build matrix block + the
darwin metal entry mirroring ced.
- .github/workflows/bump_deps.yaml: track mudler/voice-detect.cpp via
VOICEDETECT_VERSION (pin 47546430, = 4754643).
- core/config/backend_capabilities.go: register voice-detect in the
backend capability map (VoiceVerify/VoiceEmbed/VoiceAnalyze ->
speaker_recognition), mirroring speaker-recognition.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* feat(face-detect): add purego Go backend for face-detect.cpp
Add the LocalAI Go backend that dlopens libfacedetect.so (the flat
facedetect_capi_* C-ABI) via purego, mirroring the sibling voice-detect
backend. Implements the Face subset of the Backend gRPC service:
- Embeddings(PredictOptions): Images[0] base64 -> temp file -> embed_path
-> L2-normalized ArcFace embedding.
- Detect(DetectOptions): src -> detect_path_json -> Detection boxes
(class_name "face", [x1,y1,x2,y2] -> x/y/w/h).
- FaceVerify(FaceVerifyRequest): two images + threshold + anti_spoof ->
verify_paths; best-effort img areas via detect.
- FaceAnalyze(FaceAnalyzeRequest): img -> analyze_path_json -> per-face
age + gender ("M"/"F" normalized to "Man"/"Woman").
The Makefile pins face-detect.cpp to 636a1963 and builds the shared lib
with ggml + vendored libjpeg-turbo static (PIC), so the .so is
ldd-clean (no libggml) and exports only facedetect_capi_* (no jpeg_
symbols). Gated Ginkgo e2e mirrors voice-detect.
Note for the gallery-wiring task: backend registration (index.yaml,
gallery, core/config/backend_capabilities.go) is intentionally not
touched here.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* fix(voice-detect): replace em dashes in net-new descriptions
Project style forbids em/en dashes. Replace the three U+2014 chars
introduced by the voice-detect gallery/index wiring with `-`/`:`.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* feat(face-detect): wire backend into index, gallery and build
Register the face-detect.cpp face detection / embedding / verification /
analysis backend (added in Face-INT-A) into LocalAI's distribution
surfaces, mirroring the voice-detect wiring (the closest mudler C++/ggml
recognition analogue):
- backend/index.yaml: add the &facedetect meta-backend (capabilities
platform map, no top-level uri to avoid the meta-backend gotcha) plus
the full set of concrete per-arch image entries (cpu/cuda12/cuda13/
metal/rocm/sycl-f16/sycl-f32/vulkan/l4t and the -development variants),
22 entries. Referential integrity audited: every alias target resolves.
- gallery/index.yaml: add 4 model entries on backend face-detect -
face-detect-buffalo-l/m/s (insightface SCRFD + ArcFace/MBF, NON-COMMERCIAL)
and face-detect-yunet-sface (OpenCV-Zoo YuNet + SFace, APACHE-2.0, the
commercial-friendly alternative). The detector/embedder architecture is
read from GGUF metadata (facedetect.arch) at load; only the real
verify_threshold option is set (0.35 buffalo, 0.363 sface). GGUF
artifacts are not yet published: each files: entry points at the
intended mudler/face-detect-gguf location with a TODO to fill sha256
after upload (no fabricated hashes).
- core/config/backend_capabilities.go: register face-detect in the
backend capability map (Embedding/Detect/FaceVerify/FaceAnalyze ->
face_recognition), mirroring insightface.
- .github/backend-matrix.yml: add the linux build matrix block + the
darwin metal entry mirroring voice-detect.
- .github/workflows/bump_deps.yaml: track mudler/face-detect.cpp via
FACEDETECT_VERSION (pin 636a1963).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* fix(recon): voice-detect metal build branch + face-detect gallery usecases
Add the missing metal BUILD_TYPE branch to the voice-detect Makefile
forwarding -DVOICEDETECT_GGML_METAL=ON, mirroring face-detect, so the
darwin metal CI artifact is built with the Metal backend instead of
CPU-only.
Expand the 4 face-detect gallery models' known_usecases to
[face_recognition, detection, embeddings] to match the backend
capabilities map and the mirrored insightface-buffalo entries, so
auto-selection for /v1/detect and /embeddings works.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* docs(recon): document voice-detect and face-detect ggml backends
Document the new standalone C++/ggml biometric backends as the
recommended/default option for face and voice recognition, keeping the
existing Python insightface / speaker-recognition backends framed as the
legacy path.
- features/face-recognition.md: add a face-detect (ggml) backend section
with the gallery entries (buffalo-l/m/s non-commercial, yunet-sface
Apache-2.0), licensing, and verify/detect/analyze quickstart.
- features/voice-recognition.md: add a voice-detect (ggml) backend
section with the gallery entries (ecapa-tdnn, wespeaker-resnet34,
eres2net, campplus speaker recognizers; emotion-wav2vec2 non-commercial
analyze head) and quickstart.
- reference/compatibility-table.md: add face-detect.cpp and
voice-detect.cpp rows to the Vision, Detection & Recognition table.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(gallery): publish recon backend GGUF uris + sha256
Fill in the published HuggingFace GGUF uris and verified sha256 for the
9 recon gallery entries (voice-detect-* and face-detect-*), and remove
the TODO publish markers. Correct the eres2net, campplus, and
emotion-wav2vec2 uris to the actual published filenames.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* feat(gallery): re-embed buffalo anti-spoof + add audeering age/gender voice model
Update the 3 buffalo face-detect GGUF sha256 (anti-spoof ensemble now
embedded and re-uploaded under the same filenames/uris) and note the
FaceVerify anti_spoof request flag in each description. Add a new
voice-detect-age-gender-wav2vec2 gallery entry mirroring the emotion
model.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* feat(gallery): add face-detect-buffalo-sc and antelopev2 packs
Add gallery entries for two newly-published insightface face packs on
the face-detect backend: buffalo_sc (smallest pack, SCRFD-500M + small
ArcFace) and antelopev2 (higher-accuracy, SCRFD-10G + ArcFace glint360k
R100, 512-d). Both are non-commercial research-only.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* feat(recon): honor LocalAI per-model threads in voice/face-detect backends
LocalAI spawns one backend process per model and serves requests
concurrently, so the engines' own min(hardware_concurrency, 8) default
can oversubscribe cores. Forward the per-model Threads value from the
gRPC LoadModel options into the engine via VOICEDETECT_THREADS /
FACEDETECT_THREADS (read at backend construction) before the capi load.
A non-positive Threads is treated as unset, leaving the engine default.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump backend pins to CPU-optimized engine commits
voice-detect.cpp -> 0d9c1b3 (radix-2 FFT FBank, threads, flash attn + cached
pos-conv); face-detect.cpp -> 523aee1 (thread-gated direct conv, threads).
Brings the CPU optimizations into the LocalAI backend builds. GGUF format and
parity unchanged, so the published HF GGUFs remain valid.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump backend pins to round-2 CPU-optimized engines
voice-detect.cpp -> fe7e6a3 (ERes2Net 1x1->mul_mat, CAM++ layout+context,
wav2vec2 conv-LN, ECAPA capture-drop, AVX512 dispatch opt-in); face-detect.cpp
-> 9c8adb7 (AVX2 Winograd F(2x2,3x3) for SCRFD/ArcFace 3x3 convs, ArcFace
BN-fold). Parity unchanged (cosine=1.0); GGUF format unchanged, HF GGUFs valid.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump backend pins to round-3 Winograd engines
voice-detect.cpp -> 45122ec (Winograd F(2x2,3x3) for WeSpeaker/ERes2Net 3x3
convs, -22%/-20% @8t); face-detect.cpp -> cd5c962 (Winograd F(4x4,3x3) for
SCRFD large maps, -22% @1t on top of F(2x2), more load-stable). Parity held
(cosine=1.0); GGUF format unchanged, HF GGUFs valid.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump backend pins to round-4 Winograd engines (CPU opt complete)
voice-detect.cpp -> d2839ca (CAM++ FCM 2D convs through Winograd, -15.5%/-10.3%);
face-detect.cpp -> c1db23d (AVX2-vectorized Winograd tile transforms, SCRFD
detect -14%/-9.6%). Final CPU optimization round; the conv-kernel lever class is
now exhausted (parity held cosine=1.0; GGUF/parity unchanged, HF GGUFs valid).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump face-detect pin to deep-kernel engine (7ae5c4d)
face-detect.cpp -> 7ae5c4d: register-blocked winograd-domain GEMM microkernel
(2.8x isolated GFLOP/s), AVX-512 zmm evolution behind runtime CPUID dispatch
(ship-safe, AVX2 fallback bit-identical), bias/relu fused into the winograd
output transform, and SFace Conv+BN fold + bias/PReLU fusion. SCRFD detect
~1.4x faster end-to-end vs the round-4 baseline; parity bit-exact; portable
single binary (function-multiversioned, no global -mavx512f).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump voice-detect pin to ECAPA operand-order win (e9c56ae)
voice-detect.cpp -> e9c56ae: weight-as-src0 mul_mat order in ECAPA's F32
conv1d_same (routes through tinyBLAS sgemm); ECAPA embed 1.67x @1t / ~1.3x @8t,
parity cosine=1.0. Isolated to encoder.cpp (ECAPA-only); ERes2Net/CAM++/WeSpeaker
do not call conv1d_same so are provably unaffected.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump pins to FMA-throughput engines (voice f7b9f89, face 2d2d5f0)
face -> 2d2d5f0: route ArcFace 3x3 body convs through the AVX-512 winograd
microkernel (kWinoMinSize 80->14); ArcFace 1.62x @1t, SCRFD detect to 0.966 of
MLAS @1t, no regression. voice -> f7b9f89: runtime-CPUID-dispatched AVX-512
winograd-GEMM microkernel (ship-safe, AVX2 fallback bit-identical); WeSpeaker
1.90x @1t. Parity cosine=1.0 throughout; portable single binaries.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump pins to MLAS-class direct-conv engines (voice 7ecfd07, face be22d67)
Hand-tuned nChw16c AVX-512 register-tiled direct-conv microkernel (~263 GFLOP/s,
within 6-7% of MLAS per-op efficiency), runtime-CPUID-dispatched + AVX2 fallback,
fused bias/relu. voice 7ecfd07: default 3x3-s1 kernel for WeSpeaker (+37%/+32%)
+ ERes2Net, CAM++ pinned to Winograd. face be22d67: shape-gated to the ArcFace
recognizer body (+25-27% @8t); SCRFD detector stays on Winograd (no regression).
Parity cosine=1.0 / detect <=1px on AVX-512 + AVX2 paths. Portable single binaries.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump voice pin to Phase-A blocked backbone (f4e7eef)
WeSpeaker ResNet34 runs as one nChw16c blocked island (2 reorders/forward vs
~60) on AVX-512, default; per-conv directconv fallback on AVX2. +2.9% @1t /
+17-19% @8t vs per-conv directconv, parity cosine=1.0. The conv microkernel is
already FMA-bound near peak (~0.86-0.98x MLAS-implied); residual to MLAS is
sub-peak edge + non-conv tail, documented in docs/cpu-optimization.md.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump pins to breadth blocked-backbone (voice 7f66871, face d80092b)
voice 7f66871: AVX2-vectorized (ymm) blocked island - AVX2-only hosts now run
the blocked backbone for WeSpeaker (2.3x over per-conv-AVX2, cosine=1.0);
ERes2Net stays per-conv (blocked regresses, opt-in only); CAM++ Winograd-pinned.
face d80092b: ArcFace recognizer blocked island, AVX-512 default (-13% @8t, ~0.90x
MLAS, the closest conv result), auto per-conv on AVX2; SCRFD untouched on Winograd
(0 island invocations during detect). Parity cosine=1.0 / detect <=1px throughout.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump pins to small-spatial + stem conv kernels (voice 99b1804, face 47fdab6)
Measured-gap-driven conv kernels: small-spatial (fill the register tile when
output width <= tile width) + small-IC stem + strided-1x1/downsample recovery.
ArcFace recognizer 0.57 -> 0.70x MLAS @1t (the closest conv model), WeSpeaker
0.65 -> 0.79x @1t. Parity cosine=1.0 / detect <=1px. The OC-block-sharing lever
was a measured dead-end (deep stride-1 is L3-weight-bandwidth bound, not
read-port bound) and was NOT shipped. Kernel ceiling reached; further gap needs
an algorithm-class change (cache-blocked weight-stationary GEMM, or q8 weights).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump pins to GPU persistent-graph + multi-model-safe cache (voice 45d2e6b, face 0a4799a)
GPU wins (CUDA/ggml backend, no CPU-path change): persistent per-shape graph+context
cache in Backend::compute() eliminates the per-call cudaGraph re-instantiation churn
-> wav2vec2 emotion+age-gender now AT GPU parity with torch-cuDNN on GB10 (0.97-0.98x),
CAM++ -5.7ms; bit-identical parity. Cache hardened multi-model-safe (invalidate-on-free
keyed by the ModelLoader weights buffer) so LocalAI multi-model hosting cannot stale-hit.
Conv models still trail cuDNN (im2col-materialization-bound) - cuDNN implicit-GEMM lever next.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump pins to cuDNN-conv-capable engines (voice b6e4356, face 6107a24)
Adds the opt-in cuDNN implicit-GEMM conv path (VOICEDETECT_GGML_CUDNN /
FACEDETECT_GGML_CUDNN, DEFAULT OFF -> zero build/runtime dep until enabled).
On GPU it kills the im2col-materialization bottleneck and reaches torch-cuDNN
parity on the spill-bound convs: SCRFD detect 14.8->6.4ms (2.3x, ~parity),
WeSpeaker ~parity, ERes2Net beats torch (1.10x); ArcFace/CAM++ neutral (no
spill). Parity exact (SCRFD <=1px, cosine=1.0). To USE it in LocalAI, the CUDA
backend build must enable the flag AND bundle libcudnn - deferred until a
cuDNN-bundled GPU image; flag stays OFF here.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* feat(recon): enable cuDNN conv path on arm64+CUDA13 recon backends
The voice-detect.cpp / face-detect.cpp engines have an opt-in cuDNN
implicit-GEMM conv path behind VOICEDETECT_GGML_CUDNN / FACEDETECT_GGML_CUDNN
(default OFF) that kills im2col on the GPU and reaches torch-cuDNN parity
(SCRFD 2.3x, WeSpeaker/ERes2Net parity), measured on the GB10
(arm64, CUDA 13, sm_121a).
Enable it for the CUDA build, but only where cuDNN actually ships: the
arm64 + CUDA 13 image (GB10/Jetson/L4T). x86 CUDA images carry no cuDNN,
so flipping it on globally for BUILD_TYPE=cublas would be a link failure.
The Makefiles gate on CUDA_MAJOR_VERSION=13 + arch (TARGETARCH from the
matrix/Docker build, uname -m fallback for local builds).
backend/Dockerfile.golang already installs the runtime libcudnn9-cuda-13
in the arm64+CUDA13 apt block; add the matching libcudnn9-dev-cuda-13 so
the build-time link resolves.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): bump voice-detect pin to ERes2Net blocked-default (30beecd)
Defaults VD_ERES2NET_BLOCKED ON: routes the ERes2Net Res2Net body through the
blocked nChw16c AVX-512 directconv island instead of the 1x1 mul_mat fast path
(CONT-transpose + skinny low-K GEMM). On the shipped GGML_NATIVE=OFF build (ggml
mul_mat is AVX2-only) this wins ~2x at every thread count (2.07x@1t, 2.2x@4t,
2.05x@8t); pure-AVX2 fallback still 1.3-1.62x. Parity exact (cosine=1.000000 vs
golden), so registered voices + verify/identify thresholds are unaffected. The
prior default-OFF rested on a stale comment whose 23pct regression only held on
the non-shipping GGML_NATIVE=ON build.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* docs(readme): announce native voice-detect + face-detect backends in Latest News
Add a Latest News entry for the new from-scratch C++/ggml biometric backends
(voice-detect.cpp + face-detect.cpp) that replace the Python insightface and
speaker-recognition backends: no Python/onnxruntime at inference, self-contained
GGUF, bit-exact parity, GPU cuDNN parity. Mirrors the parakeet.cpp /
locate-anything.cpp native-backend news entries. Refs PR #10441.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* chore(recon): re-pin to the squashed engine release commits
The voice-detect.cpp and face-detect.cpp histories were squashed to a single
release commit, which orphaned the previous pins (voice 30beecd, face 6107a24).
Re-pin to the new single-commit SHAs (voice 3d51077, face 06914b0); the tree is
identical, so the backend build is unchanged.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(ik-llama): port multimodal path to mtmd API and bump to f96eaddb (#10534)
The IK_LLAMA_VERSION bump to f96eaddba8bed6a9a5e628bbf6a566775c70b49c pulls in
upstream commit "Prune examples/llava", which deletes examples/llava (clip.* /
llava.*). The ik-llama backend's grpc-server.cpp built a local `myclip` library
from those files and called the removed clip/llava C API, so the bump no longer
builds.
ik_llama keeps its multimodal stack in the surviving `mtmd` library
(examples/mtmd/, public headers mtmd.h + mtmd-helper.h). This ports the backend's
multimodal path onto the high-level mtmd_* / mtmd_helper_* API in place, leaving
the text path (which still uses ik_llama's retained old common API) untouched:
- Makefile: bump IK_LLAMA_VERSION to f96eaddb.
- prepare.sh: drop the clip/llava source copy + sed block; mtmd is a library
target, no source copy needed.
- CMakeLists.txt: remove the `myclip` target; link `mtmd` and add its include
dir; build grpc-server as C++17 (mtmd headers require it).
- patches: drop 0002 (targeted the deleted examples/llava/clip.cpp; the mtmd
clip.cpp never calls ggml_quantize_chunk, so the fix is unneeded). Keep 0001
(verified still applies).
- grpc-server.cpp / utils.hpp: replace clip_model_load + clip_image_load_from_bytes
+ llava_image_embed_make_with_clip_img + the manual [img-N] prefix splitting and
per-image llava_embd_batch decode loop with mtmd_init_from_file (moved after the
model load, which it requires), mtmd_helper_bitmap_init_from_buf, mtmd_tokenize
and mtmd_helper_eval_chunks. Legacy [img-N] tags are translated, in order, into
mtmd media markers (mtmd_default_marker()); the post-image suffix text stays on
the normal token path so the sampling loop is unchanged.
Supersedes #10534.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* fix(ik-llama): align json alias to ordered_json to resolve mtmd.h conflict (#10534)
mtmd.h declares `using json = nlohmann::ordered_json` at global scope (and its
mtmd.cpp depends on it), while ik_llama's whole server/common stack also uses
ordered_json. Our grpc-server.cpp/utils.hpp kept a plain `nlohmann::json` alias,
which now collides with mtmd.h once it is included for the multimodal port:
"conflicting declaration 'using json = ...'". Switch our two aliases to
ordered_json to match; it is API-compatible (utils.hpp already used ordered_json
for its log helper) and our json never crosses into an unordered-json API.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* ⬆️ Update ggml-org/llama.cpp
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(llama-cpp): link server-stream.cpp TU into grpc-server for upstream 0ed235ea (#10536)
Upstream llama.cpp 0ed235ea added an SSE stream-resumption layer in a new
translation unit tools/server/server-stream.cpp, which defines
stream_session, stream_pipe_producer and the g_stream_sessions manager.
server-context.cpp (already #included into grpc-server.cpp) now calls into
it via spipe->cleanup(), stream_aware_should_stop() and
stream_session_attach_pipe(), so without the new TU the grpc-server link
fails on every arch with:
undefined reference to `stream_pipe_producer::cleanup()'
prepare.sh already copies every tools/server/* file into tools/grpc-server/,
so the source is present; the only missing piece was including its
definitions. Add an __has_include-guarded #include "server-stream.cpp"
before server-context.cpp, mirroring the existing server-chat.cpp and
server-schema.cpp guards, keeping the source compatible with older
pins/forks that predate the split. The file is self-contained (its only
external symbols come from server-common, already in the TU) so it adds no
new undefined references; the http route-handler factories it also defines
are unused in the grpc path but harmless.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* fix(llama-cpp): build renamed ggml-rpc-server target for upstream 0ed235ea (#10536)
Upstream renamed the RPC server CMake target and binary from `rpc-server`
to `ggml-rpc-server` (tools/rpc/CMakeLists.txt: `set(TARGET ggml-rpc-server)`),
so the RPC-enabled grpc build failed with "No rule to make target 'rpc-server'".
The grpc-server itself links fine after the server-stream.cpp fix; this only
updates the RPC target name and the binary path copied to llama-cpp-rpc-server.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The background=true resumable-stream path had two latent issues.
1. Unbounded resume buffer. AppendEvent grew StreamEvents without limit, so
a long-running or abandoned background generation could consume process
memory without bound. The store now caps the buffer (event count and total
bytes, mirroring llama.cpp's byte-capped slot ring), evicting oldest events
from the front and advancing a droppedThrough watermark. GetEventsAfter
returns ErrOffsetLost when the requested starting_after is below the
watermark, and handleStreamResume surfaces that as HTTP 409 before
committing to the SSE response, so a resuming client gets a clear error
instead of a silently truncated stream.
2. Missing ownership check (IDOR). GET /responses/:id, its stream resume, and
/cancel looked up responses purely by ID, letting any caller who knows or
guesses an ID read or cancel another caller's response. Responses now carry
the creating caller's identity (auth.GetUser), stamped at creation and
compared on read/cancel/resume; a mismatch returns 404 (not 403) so
existence is not leaked. Backward compatible: responses with no owner
(single-key / no-auth deployments) remain accessible.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
In distributed mode, even when the frontend and workers share the same
models directory via a shared volume mount, starting a model on a worker
re-staged (re-downloaded) it: stageModelFiles always uploads model files
into a tracking-key-namespaced subdir on the worker, and the staging probe
only checks that staged location, so a file already present on the shared
volume at the canonical path was never reused.
Add a config switch LOCALAI_DISTRIBUTED_SHARED_MODELS (default false). When
enabled, the operator asserts that all nodes mount the SAME models directory
at the SAME path, so staging is unnecessary: the frontend's absolute model
paths are already valid on the worker. In that mode stageModelFiles returns
the cloned opts unchanged without uploading, leaving the path fields pointing
at their canonical absolute paths so the worker loads them directly from the
shared volume.
The value is plumbed from DistributedConfig through SmartRouterOptions into
the SmartRouter. Docs and docker-compose.distributed.yaml updated.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Opening an AGENT-type worker node's detail page errored with
"failed to list backends on node" / NATS "nodes.<id>.backend.list:
no responders available". Agent workers only subscribe to agent.*,
jobs.*, mcp.* and <prefix>.backend.stop; they never subscribe to
backend.list, so the per-node ListBackendsOnNodeEndpoint request had
no responder and timed out.
The aggregate cluster-wide list already guards this in
managers_distributed.go (skip nodes whose NodeType is set and not
"backend"). The single-node endpoint lacked the same guard. Thread the
NodeRegistry into ListBackendsOnNodeEndpoint and short-circuit to an
empty (non-nil) list for non-backend node types before issuing the
doomed NATS request, mirroring the aggregate-list gate so both views
stay consistent.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
pickPreferredGroup matched a quant preference against the shard base
filename with strings.Contains. Because `f16` is a substring of `bf16`,
asking for the `F16` mmproj quant would wrongly satisfy a `BF16` file and
select it when its group came first.
Match the preference as a whole token instead: it must be delimited by a
non-alphanumeric character (or the string start/end) on both outer edges.
Separators inside the preference itself (e.g. `ud-q4_k_xl`) are left
untouched, and all occurrences are scanned before rejecting.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The `tests-vibevoice-cpp-grpc-transcription` job downloads the vibevoice ASR
model (`vibevoice-asr-q4_k.gguf`, ~10 GB) and decodes it through the
e2e-backends harness. On release tag pushes the detect step forces the full
matrix (run-all=true), so this job runs and consistently times out: the inner
`go test -timeout 30m` cannot pull a 10 GB file from HuggingFace's throttled
Xet CDN within budget (curl --max-time 600 x5 retries overruns the deadline),
leaving an orphaned curl and a 30m panic. It has been red on every release
(v4.5.3/4/5).
Guard the job's `if` with `!startsWith(github.ref, 'refs/tags/')` so it no
longer runs on tag/release builds. It still runs on PRs and branch pushes that
touch vibevoice-cpp, so real regressions are caught off the release path. A
proper fix (a small ASR test GGUF) can re-enable it on tags later.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The GGUF metadata parser (gpustack/gguf-parser-go) cannot read NVFP4-quantized
GGUFs at all: it errors with "read tensor info 0: This quantized type is
currently unsupported" because NVFP4 is a ggml tensor type it does not know.
When ParseGGUFFile errors, the llama-cpp defaults hook skips guessGGUFFromFile
entirely and the deferred fallback sets the context window to the conservative
GGUFFallbackContextSize (1024). The result: a model that trains to 262144
tokens runs with n_ctx=1024, and every prompt over ~1k tokens fails with
"request (N tokens) exceeds the available context size (1024 tokens)".
Two changes:
- Drop GGUFFallbackContextSize (1024) and fall back to DefaultContextSize
(4096) in both the GGUF run-estimate path (gguf.go) and the deferred hook
fallback (hooks_llamacpp.go). 1024 is a sensible floor for a tiny CPU GGUF
but a footgun for a large, long-context model whose header simply cannot be
parsed. Strengthen the existing "GGUF unreadable" test to assert the value.
- Set context_size explicitly on the four NVFP4 gallery entries
(qwen3.6-35b-a3b-nvfp4-mtp, qwopus3.6-27b-v2-mtp-nvfp4,
qwopus3.6-27b-coder-mtp-nvfp4, qwen3.6-27b-nvfp4-mtp) so the parser failure
is irrelevant for them. 32768 matches sibling Qwen entries and is safe on
memory; operators can raise it toward the 262144 train length.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The fish-speech metal-darwin-arm64 backend build has been failing on every
release (v4.5.3, v4.5.4, v4.5.5) and is a standing red on the darwin backend
matrix. fish-speech pulls `tokenizers` transitively from its upstream source
(`pip install -e fish-speech-src`), and on darwin/arm64 there is no prebuilt
wheel for the pinned old `tokenizers` version, so pip builds it from source.
Modern rustc rejects that old crate as a hard error:
error: casting `&T` to `&mut T` is undefined behavior ...
--> tokenizers-lib/src/models/bpe/trainer.rs:517:47
= note: `#[deny(invalid_reference_casting)]` on by default
error: could not compile `tokenizers` (lib) due to 1 previous error
This is deterministic, not a flake, and there is no clean fix that does not
either pin a stale Rust toolchain or downgrade a soundness lint guarding real
UB. Until upstream fish-speech moves to a tokenizers version that compiles on
current toolchains, drop darwin support so the release backend build stays
green. The Linux/CUDA/ROCm/Intel/L4T variants are unaffected.
Removes:
- the `-metal-darwin-arm64-fish-speech` entry from `includeDarwin` in
backend-matrix.yml
- the `metal:` capability mappings and the concrete `metal-fish-speech` /
`metal-fish-speech-development` gallery entries in backend/index.yaml
- the now-unused darwin-only requirements-mps.txt
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(distributed): add SyncedMap cross-replica in-memory state component
Introduce core/services/syncstate.SyncedMap[K,V]: a thread-safe in-memory map
that keeps itself consistent across frontend replicas via NATS, with an optional
pluggable durable Store and hydrate-from-source convergence.
Several features keep process-local state surfaced to the API (finetune/quant
jobs, agent tasks, model configs) and each hand-wired the same in-memory + NATS
broadcast + read-through-store legs - or forgot to, reintroducing cross-replica
staleness. SyncedMap makes that consistency a configuration choice:
- local writes mutate the map, write through the Store, then broadcast a delta;
- the apply path is memory-only and never re-publishes or re-writes the Store
(structural echo-loop guard, mirroring galleryop.mergeStatus);
- on Start and on NATS reconnect the map re-hydrates from the source (Store, else
Loader); an optional periodic Reconcile repairs silent drift;
- standalone mode (nil NATS client) is a strict in-memory no-op.
Reconnect re-hydrate is wired via a new *messaging.Client.OnReconnect callback,
consumed through an optional type-assertion so MessagingClient stays minimal.
Adds messaging.SubjectSyncStateDelta and a reusable testutil.FakeBus (synchronous
in-process MessagingClient with wildcard matching) for adopter tests.
Component only; service migrations follow in subsequent commits.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* refactor(finetune): back jobs with SyncedMap for cross-replica consistency
FineTuneService kept jobs in a process-local map and, although it wrote them to
Postgres, ListJobs/GetJob never read the store back and the wired natsClient was
never used - so in distributed mode a job created on one replica was invisible to
the others. Replace the map and the dead client with a syncstate.SyncedMap keyed
by job ID, value *schema.FineTuneJob (the exact REST shape, so responses are
unchanged).
- Add a Store adapter (core/services/finetune/syncstore.go) over FineTuneStore,
plus FineTuneStore.ListAll (global hydrate; per-user List kept) and an
idempotent Upsert (create-or-update; Create alone fails on dup key).
- Writes go through SyncedMap.Set/Delete (write-through + broadcast); reads use
List/Get. The on-disk state.json path becomes the standalone Loader, keeping
single-node restart recovery (stale->stopped / exporting->failed fixups).
- Fold SetNATSClient/SetFineTuneStore into NewFineTuneService; app.go passes the
distributed NATS client + store when distributed, nil otherwise.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* refactor(agentpool): back agent tasks with SyncedMap for cross-replica consistency
AgentJobService.ListTasks read the process-local tasks map only, while ListJobs
already read through the DB persister + dispatcher NATS - so in distributed mode
a task created on one replica was invisible to the others. Back tasks with a
syncstate.SyncedMap keyed by task ID (value schema.Task, the exact REST shape);
jobs are left untouched.
- Store adapter (task_syncstore.go) over the existing JobPersister
(LoadTasks/SaveTask/DeleteTask); reads svc.persister/userID live so a persister
swap needs no rebuild. No new persister methods required.
- Task reads -> SyncedMap.List/Get; create/update -> Set (write-through +
broadcast); delete -> Delete. The file persister now owns its own task set so
the write-through path does not re-enter the SyncedMap lock (deadlock guard).
- The distributed NATS client is not available at construction (start() precedes
initDistributed), so it is injected via SetTaskSyncNATS, which rebuilds the
still-empty map before Start/hydrate. Wired at the main, restart, and per-user
(UserServicesManager) distributed sites.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* refactor(quantization): back jobs with SyncedMap + durable QuantStore
QuantizationService kept jobs in a process-local map persisted only to a local
state.json, so in distributed mode jobs were neither visible across replicas nor
durable cluster-wide. Back jobs with a syncstate.SyncedMap keyed by job ID
(value *schema.QuantizationJob, the exact REST shape).
- New distributed.QuantStore (GORM, table quantization_jobs) mirroring
FineTuneStore: Create/Get/ListAll/Upsert(idempotent)/Delete, registered for
AutoMigrate via distributed.InitStores (Stores.Quant).
- New adapter (quantization/syncstore.go) over QuantStore implementing
syncstate.Store, with record<->schema conversion.
- Reads go through List/Get, writes through Set/Delete (write-through +
broadcast); state.json is kept as the standalone Loader for single-node restart
recovery (stale-job fixups preserved).
- app.go passes the distributed NATS client + QuantStore when distributed, nil
otherwise; Start/Close lifecycle mirrors finetune.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* fix(syncstate): annotate gosec G118 false positive on lifeCtx
gosec flagged the WithCancel in Start as "cancellation function not called"
because the returned cancel is stored on the struct rather than called/deferred
in scope. It is invoked in Close (covered by tests), and lifeCtx must outlive
Start to drive the reconnect/reconcile goroutines. Suppress the verified false
positive with a justified #nosec G118.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* test(distributed): e2e two-replica SyncedMap sync over real NATS + Postgres
Adds the real-infrastructure counterpart to the fake-bus unit tests, in the
existing distributed e2e suite (testcontainers NATS + PostgreSQL). Two SyncedMap
instances stand in for two frontend replicas - each with its OWN NATS connection
to a shared server and a SHARED Postgres store (the distributed-mode invariant) -
and assert, over the wire:
- a create on replica A is observed by replica B;
- an update and a delete propagate A -> B (delete prunes, which a reload cannot);
- a late-joining replica recovers a job it never received a delta for, via store
hydrate on Start (the at-most-once gap a fake bus cannot exercise);
- a local Set is written through to the shared Postgres store.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(backends): whisper darwin run.sh loads whichever fallback lib exists
The macOS branch hardcoded WHISPER_LIBRARY=$CURDIR/libgowhisper-fallback.dylib,
but the cmake build emits a Mach-O named libgowhisper-fallback.so on darwin, so
the Go loader panicked at runtime ("dlopen ...dylib: no such file") and the
backend exited ("grpc service not ready") — breaking e.g. the silero-vad-ggml
VAD on darwin. Pick whichever of .dylib/.so is present so it is robust to the
build's naming either way.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* fix(kokoros): implement new Backend RPCs to fix the build
The backend.proto grew six RPCs (SoundDetection, Depth, TokenClassify,
Score and the bidi-streaming Forward) that the kokoros gRPC service never
implemented, so the trait impl no longer satisfies `Backend`:
error[E0046]: not all trait items implemented, missing:
`sound_detection`, `depth`, `token_classify`, `score`,
`ForwardStream`, `forward`
kokoros is a TTS backend with no use for these, so add `unimplemented`
stubs (plus the `ForwardStream` associated type) matching the existing
pattern for every other unsupported RPC in this file.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* fix(fish-speech): add setuptools-rust for the editable source install
install.sh installs the fish-speech source tree editable with
`--no-build-isolation`, which means the build backends of its transitive
dependencies must already be present in the venv. One of them builds a
Rust extension and its metadata step fails with:
ModuleNotFoundError: No module named 'setuptools_rust'
Add setuptools-rust to requirements.txt so installRequirements provisions
it before the editable install runs.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* fix(llama-cpp-quantization): vendor convert_hf_to_gguf.py with conversion/
Upstream llama.cpp split the model-specific logic out of the single
convert_hf_to_gguf.py file into a sibling `conversion/` package, so the
script now starts with `from conversion import ...`. Downloading just the
one file therefore fails at runtime with:
ModuleNotFoundError: No module named 'conversion'
Clone the repo (reusing the clone already needed to build llama-quantize)
and copy both the script and the `conversion/` package into the backend
dir. Python puts the script's own directory on sys.path[0], so the package
resolves when it sits beside the script.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
* fix(sglang): pin the CPU source build to sglang v0.5.11
The CPU profile builds sgl-kernel from a `git clone` of sglang with no
ref, so it always tracks master. Recent master added CPU kernels (e.g.
mamba/fla.cpp) that fail to compile in our builder:
constexpr variable 'scale' must be initialized by a constant
static library kineto_LIBRARY-NOTFOUND not found
Pin the clone to v0.5.11, the same release the GPU path already floors on
(requirements-cublas12-after.txt). Overridable via SGLANG_VERSION so the
pin can be bumped deliberately.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
golang-darwin.sh's packaging check derived the launch binary by grepping every
$CURDIR/... reference in run.sh and taking the last one. Backends that pick a
runtime CPU variant assign it via unquoted `LIBRARY=$CURDIR/libgo<x>-avx512.so`
lines, so the heuristic returned `libgo<x>-avx512.so` — a variant Darwin never
builds (arm64 builds only fallback) — and the check then failed with
"package/libgo<x>-avx512.so not found ... refusing to package (#10267)",
breaking the darwin builds for whisper, sam3-cpp, vibevoice-cpp and friends.
Scan only the `exec` line(s) (the actual launch contract) and tolerate a
quoted `exec "$CURDIR"/<binary>`. parakeet-cpp's parakeet-cpp-grpc and the
quoted-only backends (sherpa/piper/opus) resolve correctly; no Linux change.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
2026-06-27 02:05:40 +02:00
892 changed files with 75188 additions and 7905 deletions
@@ -34,7 +34,7 @@ The build matrix is data-only YAML at `.github/backend-matrix.yml` (not inside `
**Without an entry here no image is ever built or pushed, and the gallery entry in `backend/index.yaml` will point at a tag that does not exist.** The `dockerfile:` field must point at `./backend/Dockerfile.<lang>` matching the language bucket from step 1 (e.g. `Dockerfile.python`, `Dockerfile.golang`, `Dockerfile.rust`). The `tag-suffix` must match the `uri:` in the corresponding `backend/index.yaml` image entry exactly.
**`scripts/changed-backends.js` registration — REQUIRED for any new dockerfile suffix.** This is the single most common omission, because it has no effect on the PR that adds the backend (when no prior path filter could catch it anyway) — it only breaks the *next* PR that touches your backend's directory, which then gets zero CI jobs and looks broken for unrelated reasons. Edit `scripts/changed-backends.js:inferBackendPath` and add a branch BEFORE the more-generic suffixes:
**Path-filter registration — REQUIRED for any new dockerfile suffix.** This is the single most common omission, because it has no effect on the PR that adds the backend (when no prior path filter could catch it anyway) — it only breaks the *next* PR that touches your backend's directory, which then gets zero CI jobs and looks broken for unrelated reasons. Edit `scripts/lib/backend-filter.mjs:inferBackendPath` and add a branch BEFORE the more-generic suffixes:
@@ -54,7 +54,9 @@ for (const e of m.include.filter(e => e.backend === '<your-backend>')) {
}"
```
A quick way to find the right insertion point: `grep -n 'item.dockerfile.endsWith' scripts/changed-backends.js`.
A quick way to find the right insertion point: `grep -n 'item.dockerfile.endsWith' scripts/lib/backend-filter.mjs`.
If your backend consumes a *shared* build input that lives outside its own directory (a new script under `scripts/build/`, a new file copied into every image), add a rule to `SHARED_BUILD_INPUTS` in the same file — the per-backend prefix match cannot see those, and a miss ships your change to no image at all. See `scripts/lib/backend-filter_test.mjs` for the pattern; `make test-ci-scripts` runs it.
**`bump_deps.yaml` registration — REQUIRED for any backend pinning an upstream commit.** If your backend's Makefile has a `*_VERSION?=<sha>` pin to a third-party repo, the daily auto-bump bot at `.github/workflows/bump_deps.yaml` won't notice it unless you register the backend in its matrix. The bot runs `.github/bump_deps.sh` which `grep`s for `^$VAR?=` in the Makefile you list — so the pin MUST live in the Makefile (not in a separate shell script). The bump for ds4 (#9761) had to walk this back because the original landed the pin in `prepare.sh`, which the bot can't see. Pattern (for `antirez/ds4`):
@@ -115,7 +117,7 @@ Wiring a backend into `includeDarwin:` is more than the matrix entry:
1.**`includeDarwin:` entry** — `tag-suffix: "-metal-darwin-arm64-<backend>"`, `build-type: "metal"`, `lang: "go"` for go+ggml backends; omit `build-type` for the bespoke C++ ones (llama-cpp / ds4 / privacy-filter). Match an existing entry of the same shape.
2.**`backend/index.yaml`** — add `metal:` to the backend's `capabilities` map (main and `-development`) and concrete `metal-<backend>` / `metal-<backend>-development` image entries pointing at the `-metal-darwin-arm64-<backend>` images.
3.**C/C++ backends only** — add an `inferBackendPathDarwin` case in `scripts/changed-backends.js` returning `backend/cpp/<backend>/` (the generic fallthrough assumes `backend/<lang>/`, which is wrong for a C++ source tree driven with `lang: go`), and give `run.sh` a Darwin branch that exports `DYLD_LIBRARY_PATH` instead of `LD_LIBRARY_PATH`. If the build is bespoke (single `grpc-server` + dylib bundling), model it on `scripts/build/ds4-darwin.sh` and add a `backends/<backend>-darwin` make target plus a gated step in `.github/workflows/backend_build_darwin.yml`.
3.**C/C++ backends only** — add an `inferBackendPathDarwin` case in `scripts/lib/backend-filter.mjs` returning `backend/cpp/<backend>/` (the generic fallthrough assumes `backend/<lang>/`, which is wrong for a C++ source tree driven with `lang: go`), and give `run.sh` a Darwin branch that exports `DYLD_LIBRARY_PATH` instead of `LD_LIBRARY_PATH`. If the build is bespoke (single `grpc-server` + dylib bundling), model it on `scripts/build/ds4-darwin.sh` and add a `backends/<backend>-darwin` make target plus a gated step in `.github/workflows/backend_build_darwin.yml`.
4.**C++ proto gotcha** — if the backend compiles the generated gRPC/protobuf in a separate CMake target (e.g. `hw_grpc_proto`), that target must link `protobuf::libprotobuf` + `gRPC::grpc++` so the Homebrew include dirs propagate; otherwise macOS fails with `google/protobuf/runtime_version.h not found` (Linux hides this because apt headers sit in `/usr/include`).
The CI path filter only builds a backend on a PR when a file under its directory changes, so a darwin-only YAML edit builds nothing — touch a file under `backend/<lang>/<backend>/` (a one-line comment is enough) in the same PR.
| `backendBuildTagPreferenceRules` | build tags (`cuda`, `rocm`, `metal`) | installed build directory names, as a substring | alias resolution in `ListSystemBackends` |
| `engineNamePreferenceRules` | engine names (`vllm`, `llama-cpp`, `mlx`) | a gallery entry's `backend:`, as a substring | gallery variant ranking |
| `servingFeaturePreferenceTokens` | serving features (`dflash`, `mtp`) | a gallery entry's `tags:`, compared whole and case-insensitively, and nothing else | gallery variant ranking, one rank below the engine |
**Putting a token in the wrong table matches nothing and does not error**: every
candidate scores equal and the next sort key decides, so the preference silently
stops existing. The block comment above all three tables spells the contract out.
The serving feature table is the odd one: it is not keyed by capability, because
no hardware prefers a plain build over an equivalent faster build of the same
weights. It reads a declared tag and nothing else. The entry name was the
original signal and is gone: a naming convention is not a contract, and names
are author-supplied free text where a short marker like `mtp` turns up inside
unrelated words or on weights whose entry enables nothing.
`overrides.options` was rejected for the mirror-image reason: `spec_type:` is
llama.cpp's config vocabulary, whereas a cross-backend ranking decision must
work the same for `ds4`'s `mtp_path:` and `sglang`'s `speculative_algorithm:`.
**If your backend can serve the same weights faster** (speculative decoding,
multi-token prediction), say so in the docs for its gallery entries so curators
tag them: the tagging rule and the per-backend evidence table live in
[adding-gallery-models.md](adding-gallery-models.md). A backend never needs to
appear in the token table itself; it ranks builds, not engines.
Leaving your backend out is a valid choice when no ordering can be justified for
it. It then ranks below every known engine and selection falls back to size,
which is the behaviour that predates preference.
**Leaving a whole capability out is not.** A missing row gives that host an
empty preference list, so size alone decides among everything that survives the
filters, and the filter will not save you: `IsBackendCompatible` derives hardware
support from the engine NAME, so `vllm` and `sglang` carry no darwin, cuda, rocm
or sycl token and are never dropped on a host with no GPU. That is why `default`
(no usable accelerator, including a GPU under the 4 GiB VRAM floor) and
`darwin-x86` both have rows putting `llama-cpp` first. Every capability
`getSystemCapabilities()` can return needs a row unless every engine really is
equally at home there. When you add one, enumerate the engines you are demoting
rather than relying on them falling through unmatched: unmatched engines all tie
with each other, so size decides among them.
## Documenting the backend (README + docs)
A backend is not "added" until it is discoverable. Update the user-facing docs:
@@ -243,7 +308,7 @@ After adding a new backend, verify:
- [ ] Backend directory structure is complete with all necessary files
- [ ] Build configurations added to `.github/backend-matrix.yml` for all desired platforms (per-arch entries with `platform-tag` for multi-arch; `builder-base-image` for llama-cpp / ik-llama-cpp / turboquant)
- [ ]**OS coverage considered**: added to `includeDarwin:` (macOS/Apple Silicon) if the backend can build there — with the `backend/index.yaml``metal:` capability + `metal-<backend>` image entries, a `run.sh` Darwin/DYLD branch and `inferBackendPathDarwin` case for C++ backends — or the PR explains why an OS is unsupported. Do not ship Linux-only by default.
- [ ]**OS coverage considered**: added to `includeDarwin:` (macOS/Apple Silicon) if the backend can build there — with the `backend/index.yaml``metal:` capability + `metal-<backend>` image entries, a `run.sh` Darwin/DYLD branch and `inferBackendPathDarwin` case (in `scripts/lib/backend-filter.mjs`) for C++ backends — or the PR explains why an OS is unsupported. Do not ship Linux-only by default.
- [ ] Meta definition added to `backend/index.yaml` in the `## metas` section
- [ ] Image entries added to `backend/index.yaml` for all build variants (latest + development)
- [ ] Tag suffixes match between workflow file and index.yaml
@@ -251,6 +316,8 @@ After adding a new backend, verify:
- [ ] No YAML syntax errors (check with linter)
- [ ] No Makefile syntax errors (check with linter)
- [ ] Follows the same pattern as similar backends (e.g., if it's a transcription backend, follow `faster-whisper` pattern)
- [ ]**`Load` validates its input and refuses models it can't serve.** When a model config has no explicit `backend:`, the model loader greedily probes *every* installed backend with the model's name and binds to the first `Load` that succeeds — an accept-anything `Load` will capture arbitrary LLMs (issue #9287). Backends that load a real artefact get this for free (the load fails); backends with no artefact must gate on the name: `opus` accepts only its own name (or none), `local-store` requires the `store.NamespacePrefix` namespace marker sent by `core/backend/stores.go`.
- [ ]**Gallery variant ranking considered**: if this backend should be preferred over another on some hardware, it is listed in `engineNamePreferenceRules` (NOT `backendBuildTagPreferenceRules`, NOT `servingFeaturePreferenceTokens`) in `pkg/system/capabilities.go`. A missing entry silently ranks it last and lets the next sort key decide.
- [ ] Documented: added to the category list in `docs/content/features/backends.md` (and any new endpoint/realtime capability documented under `docs/content/`)
- [ ] If it is an in-house native C/C++/GGML engine, added to the maintained-engines table in the top-level `README.md`
@@ -114,6 +114,24 @@ Both `backend.yml` (push) and `backend_pr.yml` (PR) generate their matrix dynami
- **Tag pushes**: `FORCE_ALL=true` is set from the workflow side (`startsWith(github.ref, 'refs/tags/')`) — releases rebuild every backend regardless of diff.
- **Schedule / `workflow_dispatch`**: no `event.before`, falls through to "run everything" automatically.
### Shared build inputs
The per-backend prefix match only sees files under a backend's own directory, so a change to shared build infrastructure would rebuild *nothing* — an empty matrix, every job green, and the change reaching no image. That silently un-shipped PR #10946 (a partial-cuDNN packaging fix in `scripts/build/package-gpu-libs.sh`), which merged 1h48m after the weekly cron and so sat unbuilt for a week.
`SHARED_BUILD_INPUTS` in `scripts/lib/backend-filter.mjs` closes that hole. Each rule maps a shared path to the narrowest set of matrix entries it can honestly invalidate, since a full matrix is 417 Linux + 56 Darwin builds:
| Changed path | Rebuilds |
|---|---|
| `backend/backend.proto` | everything (all languages compile or copy it) |
| `backend/Dockerfile.<x>` | the Linux entries whose `dockerfile:` names it |
| `backend/python/common/` | Python, Linux + Darwin |
| `scripts/build/package-gpu-libs.sh` | Python, Linux only |
| `scripts/build/<lang>-darwin.sh` | the Darwin entries that build target routes to |
| `.github/workflows/backend_build[_darwin].yml` | everything on that OS |
| anything else under `scripts/build/` (except `*_test.sh`) | everything — conservative default for unclassified packaging inputs |
Deliberately excluded: `backend/index.yaml` (gallery metadata, never enters an image), `.github/backend-matrix.yml` (adding a backend would rebuild all of them), `backend/Dockerfile.base-grpc-builder` (owned by `base-images.yml`), and the root `Makefile` (touched in ~11% of commits, and its backend-relevant edits arrive alongside the backend directory anyway). `make test-ci-scripts` pins all of this.
The Sunday 06:00 UTC cron on `backend.yml` exists specifically because path filtering can leave Python backends frozen on stale wheels. `DEPS_REFRESH` (below) only fires when the build actually runs, so an untouched Python backend would never re-resolve its unpinned deps. The weekly cron is the safety net.
## The `DEPS_REFRESH` cache-buster (Python backends)
@@ -65,6 +65,7 @@ This is enforced by `forbidigo` (see `.golangci.yml`): `http.DefaultClient` and
The project documentation is located in `docs/content`. When adding new features or changing existing functionality, it is crucial to update the documentation to reflect these changes. This helps users understand how to use the new capabilities and ensures the documentation stays relevant.
- **Docs-with-code rule**: When you change user-facing behavior (API endpoints, CLI flags, config keys, or features), update the corresponding page under `docs/content/` in the SAME change, not as a follow-up. A user-facing change without a matching docs update is incomplete. The PR template carries a checklist item for this.
- **Feature Documentation**: If you add a new feature (like a new backend or API endpoint), create a new markdown file in `docs/content/features/` explaining what it is, how to configure it, and how to use it.
- **Configuration**: If you modify configuration options, update the relevant sections in `docs/content/`.
- **Examples**: providing concrete examples (like YAML configuration blocks) is highly encouraged to help users get started quickly.
if echo "$staged" | grep -qE '\.go$'; then go_changed=1; fi
if echo "$staged" | grep -qE '^core/http/react-ui/'; then ui_changed=1; fi
if echo "$staged" | grep -qE '^(core/http/endpoints/openai/(coordinator|respcoord|turncoord|conncoord|compactcoord|ttscoord)/|formal-verification/)'; then rt_changed=1; fi
if [ "$go_changed" -eq 0 ] && [ "$ui_changed" -eq 0 ]; then
echo "pre-commit: no Go or React UI changes staged — skipping."
if [ "$go_changed" -eq 0 ] && [ "$ui_changed" -eq 0 ] && [ "$rt_changed" -eq 0 ]; then
echo "pre-commit: no Go, React UI, or realtime-spec changes staged — skipping."
exit 0
fi
@@ -57,4 +62,11 @@ if [ "$ui_changed" -eq 1 ]; then
b.WriteString("This is a proposal, not a decision. The gallery agent adds one build per model and never joins an existing family, so entries that are alternative builds of the same weights drift apart as the gallery grows. This job re-applies the grouping heuristics from the manual sweeps and asks a human to confirm.\n\n")
b.WriteString("Each family below lists the parent, the variants, and the evidence that they are the same weights. **Reject anything whose evidence you do not believe.**\n\n")
b.WriteString(fmt.Sprintf("To decline a family permanently, add one line to `%s` in this pull request and close it:\n\n",ledgerPath))
b.WriteString(fmt.Sprintf("### Declined by the ledger (%d)\n\n",len(r.Suppressed)))
iflen(r.Suppressed)==0{
b.WriteString("Nothing the heuristics found was already on the ledger.\n\n")
}else{
b.WriteString("Candidates the heuristics found and the ledger has already settled. They are listed so the ledger's effect stays visible rather than silently shrinking the job's output.\n\n")
b.WriteString(fmt.Sprintf("### Aliases, not variants (%d)\n\n",len(r.AliasSkipped)))
b.WriteString("These entries install byte for byte the same payload. An alias exists so clients can send a particular name; folding it under another entry would hide that name.\n\n")
b.WriteString("---\n\nOpened by `.github/ci/variantproposals`. Heuristics and the rejection ledger live there and in the ledger file; a wrong proposal is a bug in one of the two.\n")
returnb.String()
}
funcjoinSignals(signals[]Signal)string{
iflen(signals)==0{
return"inferred through another member of the family"
}
out:=make([]string,0,len(signals))
for_,s:=rangesignals{
out=append(out,"`"+string(s)+"`")
}
returnstrings.Join(out,", ")
}
funcdescribeEvidence(eEvidence)string{
varparts[]string
ife.SharedStem!=""{
parts=append(parts,fmt.Sprintf("same name once quantization markers are stripped: `%s`",e.SharedStem))
}
ife.SharedFile!=""{
parts=append(parts,fmt.Sprintf("same primary weight filename once quantization markers are stripped: `%s`",e.SharedFile))
// A parent that is itself somebody's variant would create a chain, which
// the gallery's own resolution refuses to install.
ifowner,claimed:=claimedBy[parentName];claimed{
returnnil,&Refusal{Members:names,Reason:fmt.Sprintf("the natural parent %q is already a variant of %q; proposing it as a parent would nest variants",parentEntry.Name,owner)}
returnnil,&Refusal{Members:names,Reason:fmt.Sprintf("the natural parent %q is already proposed as a variant of %q; proposing it as a parent would nest variants",parentEntry.Name,owner)}
}
// Adding a variants key to an anchor is inherited by every entry that
// merges it, silently grouping models nobody proposed. Handling that means
// editing each merging child too, which is a larger change than this job
// should make unsupervised, so it refuses and hands the reviewer the list.
Reason:fmt.Sprintf("the parent %q defines YAML anchor &%s, and a variants key added there is inherited by the %d entries that merge it (%s). Grouping this family by hand also means adding an explicit `variants: []` to each of those entries",
returnnil,&Refusal{Members:names,Reason:fmt.Sprintf("%q already offers variants of its own, so it cannot itself be a variant target",target.Name)}
}
if!target.Installable(){
returnnil,&Refusal{Members:names,Reason:fmt.Sprintf("%q has no url, config_file, overrides or files, so it is not independently installable",target.Name)}
returnnil,&Refusal{Members:names,Reason:fmt.Sprintf("%q is already a variant of %q; a target claimed by two parents is not something the gallery resolves predictably",target.Name,owner)}
return0,fmt.Errorf("more than one entry is named exactly %q (%s), so which one is the base build is a judgement this job will not make",familyStem,strings.Join(names,", "))
}
// No shared stem to be named after. An entry whose name every other member
// extends is still recognisably the base one, and this is the only handle
// left for families whose weights carry no readable quantization token at
// all, such as the ONNX builds.
ifprefix,ok:=uniquePrefixMember(ix,members);ok{
returnprefix,nil
}
best:=-1
bestWidth:=1<<20
for_,m:=rangemembers{
e:=ix.Entries[m]
width:=unknownWidth
ifprimary,ok:=e.PrimaryWeightFile();ok{
width=BuildWidth(primary)
}
// Members are visited in gallery order, so a strict comparison leaves
// the earliest entry holding a tie and the choice is deterministic.
ifwidth<bestWidth{
best,bestWidth=m,width
}
}
ifbest<0{
return0,fmt.Errorf("no member could be identified as the smallest build")
}
ifbestWidth==unknownWidth{
names:=make([]string,0,len(members))
for_,m:=rangemembers{
names=append(names,ix.Entries[m].Name)
}
return0,fmt.Errorf("no member declares a weight file whose quantization can be read (%s), so the smallest build cannot be identified",strings.Join(names,", "))
}
returnbest,nil
}
// uniquePrefixMember reports the single member whose name every other member's
stale-issue-message:'This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days.'
stale-pr-message:'This PR is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 10 days.'
@@ -39,9 +39,10 @@ LocalAI follows the Linux kernel project's [guidelines for AI coding assistants]
- **Logging**: Use `github.com/mudler/xlog` (same API as slog)
- **Go style**: Prefer `any` over `interface{}`
- **Comments**: Explain *why*, not *what*
- **Docs**: Update `docs/content/` when adding features or changing config
- **Docs (docs-with-code rule)**: When you change user-facing behavior (API endpoints, CLI flags, config keys, or features), update the corresponding page under `docs/content/` in the SAME change, not as a follow-up. A user-facing change without a matching docs update is incomplete. See also the documentation conventions in [.agents/coding-style.md](.agents/coding-style.md).
- **New API endpoints**: LocalAI advertises its capability surface in several independent places — swagger `@Tags`, `/api/instructions` registry, auth `RouteFeatureRegistry`, React UI `capabilities.js`, docs. Read [.agents/api-endpoints-and-auth.md](.agents/api-endpoints-and-auth.md) and follow its checklist — missing any surface means clients, admins, and the UI won't know the endpoint exists.
- **Admin endpoints → MCP tool**: every admin endpoint that an admin would manage conversationally (install/list/edit/toggle/upgrade) MUST also be exposed as an MCP tool in `pkg/mcp/localaitools/`. The LocalAI Assistant chat modality and the standalone `local-ai mcp-server` consume that package; drift between REST and MCP is a real risk. Read [.agents/localai-assistant-mcp.md](.agents/localai-assistant-mcp.md) — the `TestToolHTTPRouteMappingComplete` test fails until you wire the new tool and update the route map.
- **Build**: Inspect `Makefile` and `.github/workflows/` — ask the user before running long builds
- **Backend OS coverage**: a new backend must target every OS it can build for, not just Linux. `.github/backend-matrix.yml` has two matrices — `include:` (Linux) and `includeDarwin:` (macOS / Apple Silicon). Most C/C++/GGML and many Python backends build on Darwin too — wire the `includeDarwin` entry + `backend/index.yaml``metal:` entries, or say in the PR why an OS is unsupported. See the darwin checklist in [.agents/adding-backends.md](.agents/adding-backends.md).
- **Gallery variant ranking**: a gallery entry can declare `variants` (alternative builds of the same weights), and LocalAI ranks the ones a host can run by engine preference first, size second. A new backend that should be preferred on some hardware must be listed in `engineNamePreferenceRules` in `pkg/system/capabilities.go`; the sibling `backendBuildTagPreferenceRules` speaks build tags rather than engine names, and using the wrong table matches nothing without erroring. See [.agents/adding-backends.md](.agents/adding-backends.md).
- **UI**: The active UI is the React app in `core/http/react-ui/`. The older Alpine.js/HTML UI in `core/http/static/` is pending deprecation — all new UI work goes in the React UI
@@ -177,6 +177,7 @@ For more details, see the [Getting Started guide](https://localai.io/basics/gett
## Latest News
- **June 2026**: New native biometric backends from the LocalAI team: [voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp) for speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion) and [face-detect.cpp](https://github.com/mudler/face-detect.cpp) for face detection, recognition, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace). Both are from-scratch C++/ggml engines with no Python or onnxruntime at inference, self-contained GGUF weights, bit-exact parity with the reference, and GPU cuDNN parity, replacing the heavier Python `insightface` and `speaker-recognition` backends ([PR #10441](https://github.com/mudler/LocalAI/pull/10441)).
- **June 2026**: New [realtime voice assistant demo](https://github.com/localai-org/localai-realtime-demo) (a tiny Go client for the Realtime API with a full talk-back voice loop and tool calling), plus [streaming of the realtime LLM / TTS / transcription pipeline stages](https://github.com/mudler/LocalAI/pull/10176) and [configurable WebRTC ICE candidates](https://github.com/mudler/LocalAI/pull/10231).
- **June 2026**: Big speech push: the [parakeet.cpp](https://github.com/mudler/parakeet.cpp) ASR engine gains [NeMo-faithful segment timestamps](https://github.com/mudler/LocalAI/pull/10207), a [multilingual streaming Nemotron-3.5 model](https://github.com/mudler/LocalAI/pull/10199), [dynamic batching for concurrent transcription](https://github.com/mudler/LocalAI/pull/10112) and [CUDA graphs](https://github.com/mudler/LocalAI/pull/10273); the new [CrispASR backend](https://github.com/mudler/LocalAI/pull/10099) adds multi-architecture ASR + TTS, and [60 Piper TTS voices across 42 languages](https://github.com/mudler/LocalAI/pull/10296) land in the gallery (plus [per-request TTS instructions and params](https://github.com/mudler/LocalAI/pull/10172)).
- **June 2026**: New backends and models: [locate-anything.cpp](https://github.com/mudler/LocalAI/pull/10264) for open-vocabulary object detection via ggml, [Ideogram4 image generation](https://github.com/mudler/LocalAI/pull/10201) in stablediffusion-ggml, [llama.cpp video input](https://github.com/mudler/LocalAI/pull/10216), and the [Gemma 4 QAT family with MTP speculative-decoding pairs](https://github.com/mudler/LocalAI/pull/10215). Plus an [interactive CLI chat mode](https://github.com/mudler/LocalAI/pull/10226) and [RAG source citations in agent responses](https://github.com/mudler/LocalAI/pull/10228).
@@ -231,12 +232,17 @@ Most backends wrap a best-in-class upstream engine. A handful of them are native
| Backend | What it does |
|---------|-------------|
| [parakeet.cpp](https://github.com/mudler/parakeet.cpp) | C++/GGML port of NVIDIA NeMo Parakeet ASR (tdt/ctc/rnnt/hybrid), with cache-aware streaming transcription |
| [ced.cpp](https://github.com/mudler/ced.cpp) | C++/GGML port of the CED audio-tagging models: sound-event classification (527-class AudioSet) over REST and the realtime API for live recognition |
| [voxtral.c](https://github.com/mudler/voxtral.c) | Voxtral Realtime 4B speech-to-text in pure C |
| [moss-transcribe.cpp](https://github.com/localai-org/moss-transcribe.cpp) | C++/GGML port of OpenMOSS MOSS-Transcribe-Diarize: joint long-form transcription, speaker diarization and timestamping in a single pass |
| [moss-tts.cpp](https://github.com/mudler/moss-tts.cpp) | C++/GGML port of the OpenMOSS MOSS-TTS family: text-to-speech (MOSS-TTS-Local v1.5, 48 kHz stereo) with reference-audio voice cloning, through the MOSS-Audio-Tokenizer neural codec |
| [ced.cpp](https://github.com/localai-org/ced.cpp) | C++/GGML port of the CED audio-tagging models: sound-event classification (527-class AudioSet) over REST and the realtime API for live recognition |
| [voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp) | Speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion), replacing the Python speaker-recognition backend |
| [voxtral-tts.c](https://github.com/mudler/voxtral-tts.c) | Voxtral Realtime 4B speech-to-text in pure C |
| [vibevoice.cpp](https://github.com/mudler/vibevoice.cpp) | Native port of Microsoft VibeVoice for TTS (voice cloning) and long-form ASR with speaker diarization |
| [face-detect.cpp](https://github.com/mudler/face-detect.cpp) | Face detection, recognition, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace), replacing the Python insightface backend |
| [free-splatter.cpp](https://github.com/localai-org/free-splatter.cpp) | Pose-free 3D reconstruction (FreeSplatter): turns a handful of plain photos into 3D Gaussians, no camera poses or GPU required |
LOG_ERR("unable to load clip model: %s",params.mmproj.path.c_str());
returnfalse;
}
if(params.n_ctx<2048){// request larger context for the image embedding
params.n_ctx=2048;
@@ -512,10 +481,24 @@ struct llama_server_context
}
if(multimodal){
constintn_embd_clip=clip_n_mmproj_embd(clp_ctx);
constintn_embd_llm=llama_model_n_embd(model);
if(n_embd_clip!=n_embd_llm){
LOG("%s: embedding dim of the multimodal projector (%d) is not equal to that of LLaMA (%d). Make sure that you use the correct mmproj file.\n",__func__,n_embd_clip,n_embd_llm);
// mtmd_init_from_file requires the already-loaded text model, so it must
// run AFTER llama_init_from_gpt_params. It validates the projector
// against the model internally and returns nullptr on dim mismatch, so
// the explicit clip_n_mmproj_embd check is no longer needed.
or "projector." in name or "pre_mm_projector_norm" in name \
or "image_newline" in name or "view_seperator" in name \
or "patch_embed" in name or "patch_embedding" in name \
- or "patch_merger." in name or "model.connector." in name:
+ or "patch_merger." in name or "patch_merge_mlp" in name \
+ or "model.connector." in name:
return None
return super().filter_tensors(item)
@@ -1201,7 +1202,7 @@ class TextModel(ModelBase):
self.gguf_writer.add_embedding_length(n_embd)
logger.info(f"gguf: embedding length = {n_embd}")
- if (n_ff := self.find_hparam(["prefix_dense_intermediate_size", "intermediate_size", "n_inner", "hidden_dim"], optional=True)) is not None:
+ if (n_ff := self.find_hparam(["prefix_dense_intermediate_size", "dense_intermediate_size", "intermediate_size", "n_inner", "hidden_dim"], optional=True)) is not None:
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
