mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-14 15:18:03 -04:00
1cd7d63c7b0d0f5ae5dcfb84ee2c8d80ea9fdbe3
962
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 mudler's LocalAI [bot]andEttore Di Giacinto
|
1cd7d63c7b |
fix(distributed): reject wrong-model requests on the remaining modalities (#10990)
#10970 gave the four PredictOptions RPCs a model-identity check so a backend reached through a stale distributed route rejects the request instead of answering from whatever model it holds (#10952). Every other modality shares that exposure: the route is cached by host:port, a worker can recycle a stopped backend's port for another model's backend, and a liveness-only probe cannot tell a stale row from a valid one. Extends the same mechanism to the 21 remaining request messages that reach a backend through the router, using the pattern #10970 established rather than a parallel one: - proto: ModelIdentity on each modality request message. - controller: populated from ModelConfig.Model at the call site that also builds ModelOptions, so load-time and request-time values are equal by construction. - backends: one generic guard in pkg/grpc/server.go (27 Go backends), the method set in backend/python/common (36 Python backends), llama-cpp (AudioTranscription/Stream, Rerank, Score) and privacy-filter (TokenClassify). - reconcile already drops the stale row on IsModelMismatch; no change. TTSRequest and SoundGenerationRequest get a SEPARATE ModelIdentity field rather than reusing their existing `model`: FileStagingClient rewrites `model` to a worker-local path, so comparing it would reject valid requests in exactly the configuration this guards. AudioEncode/AudioDecode are deliberately left unguarded: the opus codec backend is loaded from a literal rather than a ModelConfig, so no value carries the equality guarantee the comparison depends on. The four bidirectional stream RPCs are out of scope; they bypass reconcile. Empty means skip on both sides, so an old controller, an old backend, and the bare request structs in tests/e2e-backends all keep working. Assisted-by: Claude Code:claude-opus-4-8 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
65bdbc4ee3 |
fix(http): make /readyz reflect startup readiness, plus gitignore and coverage-ratchet fixes (#10989)
* fix(http): make /readyz reflect startup readiness instead of always 200 /readyz was registered as a static handler returning 200 unconditionally, so it carried no information: it was green whenever it could be reached at all. Readiness could not distinguish "serving" from "still starting", and any future change that started the HTTP listener earlier would silently turn the probe into a lie. Track startup completion on the Application (atomic flag, flipped at the very end of New() on the success path only) and have the readiness handler consult it per request, returning 503 with a small JSON body while startup is in progress. A nil readiness source fails open so embedders keep the historical behaviour. /healthz is deliberately left readiness-independent. Liveness and readiness answer different questions, and failing liveness during a long preload makes an orchestrator restart the pod mid-download so the preload never finishes. This matters because since #10949 the startup preload materializes HuggingFace artifacts for managed backends: tens of GB for a large model (31 GB observed on a live cluster). Both probes stay in quietPaths and stay exempt from auth. Note the listener is still started only after New() returns, so today the not-ready state is not observable over HTTP. Moving the listener earlier is a separate, deliberate decision and is not made here. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(gitignore): anchor the mock-backend pattern so its source dir is traversable The bare `mock-backend` pattern matched the *directory* tests/e2e/mock-backend/, not just the binary built into it. Git will not descend into an ignored directory even for tracked files, so `git add tests/e2e/mock-backend/main.go` required -f. This was hit while working on #10970. Anchor it to the artifact's full path. The built binary stays ignored (it is also covered by tests/e2e/mock-backend/.gitignore) while the source directory becomes traversable again. Verified with `git check-ignore -v`: a new source file under tests/e2e/mock-backend/ is no longer ignored, and the binary produced by `make build-mock-backend` still is. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(coverage): raise the coverage ratchet from 48.5% to 54.2% The committed baseline had drifted well below reality: it still read 48.5% while a full instrumented run measures 54.2%. A stale-low baseline makes the gate meaningless — coverage could regress by more than 5 percentage points and still pass. Raising a ratchet is a deliberate act, not something to fold into an unrelated fix, so it gets its own commit. The headroom was earned by tests landed in #10946, #10947, #10948, #10949, #10956, #10967, #10968, #10970 and #10975. Measured with `make test-coverage` on this branch (the same instrumented run `make test-coverage-baseline` uses: ginkgo over ./pkg and ./core plus the in-process tests/e2e suite, --covermode=atomic, --coverpkg over core/... and pkg/..., generated protobuf excluded). The run completed with exit 0 and zero spec failures; the total was then written with the exact command the test-coverage-baseline target uses: go tool cover -func=coverage/coverage.out \ | awk '/^total:/{gsub(/%/,"",$NF); print $NF}' > coverage-baseline.txt Verified afterwards with scripts/coverage-check.sh, which reports OK. Note the measured figure includes the readiness specs added earlier on this branch, so it is a demonstrated floor rather than an estimate. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
83a0f16a21 |
feat(gallery): let one gallery entry offer several builds of the same model (#10943)
* feat(system): expose raw detected capability for model meta resolution Model meta gallery entries express hardware fallback through candidate ordering rather than a capability map, so they need the undecorated detected capability string without Capability's default/cpu fallback chain. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(system): drop duplicate capability accessor, cover DetectedCapability ReportedCapability was added with a body identical to the existing DetectedCapability. Keep one accessor and move the specs onto it, since DetectedCapability had no direct coverage of its no-fallback behavior. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): parse IEC binary size suffixes (KiB..PiB) ParseSizeString accepted only SI suffixes, so a "20GiB" floor was rejected outright. Model and VRAM sizes are conventionally quoted in IEC units, and silently reading GiB as GB would understate a floor by about 7%. Purely additive: these inputs previously returned an unknown-suffix error. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add Candidate type for meta model entries Candidate is one option in a meta entry's ordered variant list. It names a concrete gallery entry and declares when that entry suits the host. EffectiveMinVRAM resolves the VRAM floor, letting an authored min_vram win over a nightly-inferred one. An unparseable floor errors instead of being treated as absent: swallowing a typo would turn a constrained candidate into an unconstrained one and select a too-large variant rather than fail loudly. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add hardware-aware model variant resolver Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): allow gallery model entries to declare variant candidates A gallery entry with a non-empty candidates list is a meta entry: it names an ordered list of concrete entries and resolves to the first one the host can satisfy, instead of describing model files directly. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): resolve meta model entries to hardware-appropriate variants at install Meta gallery entries carry an ordered candidate list; at install time the first candidate the host satisfies is resolved and its payload installed under the meta's name, so the model keeps a stable name regardless of which variant backs it. The resolution is recorded in the installed gallery config so a reinstall honors a prior pin and operators can see the backing variant. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): key meta pin recall on the installed name and detach resolved entries Six review findings on the meta-entry install path. Pin recall was keyed on the gallery entry name while applyModel writes the record under the install name (req.Name when supplied), so a meta installed under a custom name with a pin lost that pin on reinstall and was silently re-resolved onto a different variant, possibly swapping its backend. Compute the install name with applyModel's own precedence before the recall. ResolveMetaModel returned a shallow struct copy, so the resolved entry's Overrides aliased the gallery entry's map and the install path's in-place mergo merge wrote the caller's request into the shared catalog. Detach Overrides, ConfigFile, AdditionalFiles, URLs and Tags. Not exploitable today only because this path re-unmarshals the gallery per call, which is a property nobody should have to rely on. Also: overlay the meta's name onto the persisted config for meta installs so the gallery file no longer records the variant's name; move the pinned-VRAM warning below the variant validation so a pin naming a nonexistent entry does not warn about VRAM before failing for an unrelated reason; and stop seeding config.URLs in the config_file branch, which duplicated every declared URL. Add seven network-free specs driving InstallModelFromGallery with a meta entry: variant payload wins over the meta's legacy url fallback, the resolution record round-trips to disk, a pin is recorded and honored on reinstall including under a custom install name, and the resolved entry does not alias the gallery's maps. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): deep-copy meta overrides and make two specs functional ResolveMetaModel detached the resolved entry's Overrides and ConfigFile with maps.Clone, which only copies the top level. Gallery overrides are nested in practice (parameters.model is near-universal) and the install path merges the caller's request with mergo.WithOverride, which recurses into nested maps and overwrites them in place, so the gallery entry's own inner maps were still reachable and still got rewritten by the last caller to install. Copy both maps all the way down instead, recursing through the container shapes a YAML decoder produces. ConfigFile is not mutated on the install path today, but it carries the same kind of nested payload and leaving it shallowly cloned would invite the bug back. Also fix two specs that passed whether or not their target fix was present: - "does not write the caller's overrides back into the gallery entry" re-read the catalog from disk, which re-unmarshals fresh structs and so cannot observe in-memory aliasing. It now asserts against the in-memory gallery entry and drives the real mergo merge. - "round-trips the resolution record to disk under the meta's name" asserted a name that is already correct in the config_file branch. It now drives the url branch via a file:// fixture, where the meta-name overlay actually applies. Both were verified red by reverting their fix. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(gallery): lint meta model entry invariants in index.yaml Adds Ginkgo specs that parse the shipped gallery/index.yaml and enforce the invariants that keep meta entries safe: a legacy url fallback equal to the final candidate's url, references only to existing non-meta entries, a min_vram floor on every candidate but the last-resort one, a capability drawn only from the vocabulary the system can report, and descending VRAM floors within a capability group. The capability check is the only compensating control for a typo there. Candidate matching is a case-sensitive exact comparison against SystemState.DetectedCapability(), so an unknown value never matches and falls through silently instead of erroring. The vocabulary therefore mirrors the raw return set of getSystemCapabilities(), which notably excludes "cpu": that is a fallback key inside Capability(capMap) on the meta backend path, never a reported capability. A CPU-only host reports "default". These pass vacuously until the pilot meta entry lands; the guard is intentionally in place before the thing it guards. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(gallery): close coverage gaps in the meta entry lint The ordering invariant grouped candidates by capability and asserted floors descend within a group. A candidate with an EMPTY capability matches every host, so it does not belong in its own group: it dominates every later candidate whose floor is at or above its own, across capability groups. Track a running minimum floor over the unconditional candidates instead, which subsumes the old same-group check for the empty capability. Every spec skipped non-meta entries, so with zero meta entries in the index all five bodies were no-ops. Aligning GalleryModel.IsMeta() with GalleryBackend.IsMeta(), whose semantics are deliberately opposite, would have made all of them pass while checking nothing. Extract each invariant into a helper over a slice of entries returning the violations it finds, and cover those helpers with synthetic fixtures so the logic stays tested at zero meta entries. The index-driven specs are now a thin application of already proven logic. Also assert the index parses non-empty, report every violation in one run rather than aborting on the first, and parse the index once for the suite. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(gallery): add nightly denormalization of meta model candidates Fills the read-only backend, quantization and inferred_min_vram fields on meta gallery candidates and opens a PR, modeled on the existing checksum_checker job. Computing these needs network access, so it happens nightly rather than at install time. An authored min_vram is never modified: a human who measured a real load knows more than a pre-download estimate does. The index is rewritten via yaml.Node rather than a document round-trip. A full round-trip reflows all ~26k lines of gallery/index.yaml, which would bury the computed values and make the nightly PR unreviewable. The rewrite touches only the three derived keys, so authored styling survives and a run that computes nothing leaves the file untouched. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ci): keep the gallery denormalize diff reviewable and self-healing The nightly denormalization job edits YAML nodes instead of round-tripping structs so its PR stays small enough for a human to review, but the write path undid that: yaml.Marshal re-encoded the node tree at yaml.v3's default 4-space indent and dropped the leading document marker, reflowing roughly 6000 lines around the handful of real changes. Encode through yaml.NewEncoder at the index's authored 2-space indent and restore the header. A write that changes three fields now changes three lines. Stale inferred_min_vram values were also never cleared. Both skip paths (an authored min_vram is present, or the candidate is the last resort) returned before touching the field, so a candidate that gained a floor or became the last resort after a reorder kept an inferred value that EffectiveMinVRAM reported as a real constraint, failing the meta lint with no way for the job to self-heal. Clear the field before both skips. The workflow discarded a whole night's work on any single failure: the program exits 1 when a candidate cannot be estimated, which aborted the job before the PR step, so one unreachable candidate blocked every other refresh indefinitely. Capture the status, open the PR with what was computed, mark the PR body as partial, and fail the run afterwards so the problem still surfaces. Also preserve the index's existing file mode instead of forcing 0644, and drop the redundant //go:build ignore tag, since Go already skips dot directories and the sibling modelslist.go carries no tag. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add nanbeige4.1-3b meta entry with hardware-resolved variants Adds the first real meta entry to the gallery index. It resolves to the Q8_0 build on hosts with at least 6GiB of VRAM and to the Q4_K_M build everywhere else, installing either payload under the stable name nanbeige4.1-3b. The entry carries a url equal to its final candidate's url. LocalAI releases that predate candidates support parse the index non-strictly and drop the key silently, so without that url they would list the entry and install nothing. A regression spec parses the index the way those releases do and asserts every meta entry stays installable for them. Also teaches core/schema/gallery-model.schema.json about candidates. The schema sets additionalProperties: false at the top level, so an author following CONTRIBUTING.md and adding the yaml-language-server comment would otherwise get a validation error on this entry. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): make candidate entries complete, installable entries Reworks hardware-resolved gallery variants after a design pivot. There is no longer a separate "meta" entry kind. A gallery entry is a normal, complete entry that may additionally carry candidates:, a list of hardware-gated upgrades over itself, and the entry is itself the last-resort candidate. The previous design relied on a bare url: as the fallback for LocalAI releases that predate candidates support. That fallback is empty in practice: none of the 80 gallery/*.yaml files carry a top-level files:, and 1216 of 1281 index entries carry their payload in the index entry itself, so a url alone yields a config template with nothing to download. Since every released LocalAI reads gallery/index.yaml live from master, merging a payload-less entry would have shown every existing user a model that installs to a broken state. Making the entry its own base candidate removes the problem at the root: old clients drop the candidates key and install the entry exactly as they do today. Resolution order is now explicit pin, then capability plus VRAM over the declared upgrades, then the entry itself. The entry ALWAYS installs: when its own min_vram or capability is unmet the installer warns and installs it anyway, because there is nothing below it and refusing would make the gallery behave worse the newer the client is. A pin naming the entry's own name is valid and is how an operator declines an upgrade. IsMeta() becomes HasCandidates(), ResolveMetaModel becomes ResolveVariant, and the persisted meta_name record key becomes entry_name. GalleryBackend.IsMeta() is a separate concept and is untouched. The lint drops the three rules the pivot makes wrong (url equality with the final candidate, no inline payload, unconstrained final candidate) and gains one: the entry's own floor must sit strictly below every candidate's, since a base that outranks a candidate makes that candidate unreachable. The pilot entry is now the existing nanbeige4.1-3b-q4, which gains a 2GiB floor of its own and a single 6GiB upgrade to nanbeige4.1-3b-q8, replacing the separate nanbeige4.1-3b entry added in |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
465d488c90 |
fix(distributed): reject wrong-model requests at the backend (#10970)
fix(distributed): reject wrong-model requests at the backend (#10952) In distributed mode the controller caches a NodeModel row naming a backend's host:port. A worker can recycle a stopped backend's gRPC port for a different model's backend, and probeHealth verifies liveness rather than identity, so the probe succeeds against whatever now occupies the port and the request is dispatched to the wrong backend. The caller gets a silent wrong-model answer. Nothing in the request could catch this: PredictOptions had no model field, so model identity crossed the wire only in ModelOptions.Model at LoadModel time, and the cached-hit path issues no LoadModel. Every backend's "model not loaded" guard checks a nil handle, which a process holding a different model passes, so the stale row was never dropped either. Add PredictOptions.ModelIdentity and enforce it at the point of use: - The controller populates it in gRPCPredictOpts from ModelConfig.Model, the same expression ModelOptions feeds to model.WithModel and therefore the same value the backend received as ModelOptions.Model. Both are read from one config value in one function, so they are equal by construction and the comparison cannot false-reject. - Backends compare it against what they loaded and return NOT_FOUND with a fixed sentinel. Enforced in pkg/grpc/server.go (27 Go backends), an interceptor in backend/python/common (all 36 Python backends, no per-backend change), and the llama-cpp / ik-llama-cpp / ds4 C++ servers. That is every backend with real exposure: kokoros answers all four RPCs with unimplemented and privacy-filter implements none of them. - The router's reconcile drops the stale replica row on a mismatch, so the next request reloads somewhere correct. Empty means "skip the check" on both sides: a controller that predates the field sends nothing, a backend loaded by such a controller has nothing to compare, and the C++ server synthesizes PredictOptions internally for ASR. That keeps upgrades working in both directions. Scoped to the four PredictOptions RPCs. TTSRequest.model and SoundGenerationRequest.model are deliberately NOT validated: FileStagingClient already rewrites them to worker-local absolute paths, so in distributed mode they already differ from the load-time value and comparing them would reject valid requests. IsModelMismatch requires both the NOT_FOUND code and the sentinel, unlike the neighbouring helpers which accept either. insightface's Embedding returns NOT_FOUND "no face detected" on a PredictOptions RPC, and a code-only check would drop a healthy replica row on every faceless image. Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Nandana Dileep
|
b5e4413eab |
feat: add MiniMax-M3 model support (#10837)
Adds inference parameter defaults for the minimax-m3 model family and includes a vendored patch of upstream llama.cpp PR #24523 to recognize the minimax-m3 architecture. Once the upstream PR merges, the patch can be removed and LLAMA_VERSION bumped normally. Changes: - backend/cpp/llama-cpp/patches/0001-add-minimax-m3-support.patch: vendored patch from ggml-org/llama.cpp#24523 (Preliminary MiniMax-M3 support). Applied by prepare.sh during the build; keeps the pinned LLAMA_VERSION pointing at the latest upstream tag. - core/config/inference_defaults.json: add minimax-m3 family entry (temperature=1.0, top_p=0.95, top_k=40, min_p=0.01, repeat_penalty=1.0, matching the existing minimax defaults) and register it in the patterns list before the shorter minimax-m2.7 entry for correct longest-match-first ordering. Upstream: depends on ggml-org/llama.cpp#24523 Closes: https://github.com/mudler/LocalAI/issues/10820 Signed-off-by: Nandana Dileep <110280757+nandanadileep@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
e55cc3e2a7 |
fix(worker): bound the gRPC port allocator and stop leaking dead backends' ports (#10968)
The worker's gRPC port allocator grew monotonically with no upper bound: nextPort started at the base port and incremented whenever freePorts was empty, and nothing checked 65535. Past that it handed out integers that cannot be bound, surfacing as an opaque "backend won't start". #10961 estimated this needed ~15,000 concurrent-peak allocations, i.e. effectively unreachable. It is not, because of a second defect: the "process died unexpectedly" branch in startBackend deleted the process map entry without releasing its port at all. That port was leaked, never quarantined and never reused. A crash-looping backend leaks one port per restart, so a backend dying every 30s walks 50051 to 65535 in about five days. The leak, not concurrent peak, is the realistic route to exhaustion. Fixing the leak alone would have been wrong. Releasing that port makes it re-bindable, and the death path is the one teardown path with no request/reply to carry StoppedProcessKeys back to the controller (#10952's eager row removal), so a stale NodeModel row could then resolve to a live listener belonging to a different backend. probeHealth verifies liveness, not identity, so the request is silently misrouted. The 15s port quarantine does not cover this: the only reaper is the per-model health check at ~45s, and it can be disabled outright. The residual was masked only because the port was never rebound. So both are fixed together: - The allocator takes an explicit [basePort, LOCALAI_GRPC_MAX_PORT] range and returns ErrNoFreePort naming the range, the live backend count, the quarantined count, and the knob to raise. Exhaustion is now diagnosable instead of surfacing as an unbindable port. - Released ports carry per-key affinity: a port is offered back to the process key that last held it before any other key. Process keys (modelID#replica) and NodeModel rows (nodeID, modelName, replicaIndex) are isomorphic, so a port that can only be re-bound by its previous owner can only ever be named by that owner's row, which that key's re-registration overwrites. Misrouting to a different model becomes impossible by construction rather than by racing the quarantine timer. Affinity is a preference, not a reservation: under range pressure an owned port is stolen with a warning, because a guaranteed outage is worse than a rare misroute window on a port long out of quarantine. Claiming a port evicts its previous owner's entry, keeping ownership injective over ports so the affinity map can never exceed the range width regardless of how many distinct model keys the worker sees. Ownership also expires. It is only load-bearing while a controller row could still name the port, which the per-model reaper bounds at roughly 45s, so it lapses after five minutes and the port becomes ordinary free space again. Holding it indefinitely would have made every distinct model the worker ever served consume a port permanently: every release path is keyed, so nothing would ever be unowned, the allocator would climb to the end of its range on distinct-key count rather than concurrency, stealing would become routine, and the steal warning would tell operators to widen a range that was not the constraint. With expiry, reaching the steal branch means the worker is genuinely out of concurrent capacity, so that advice is correct when it appears. Closes #10961 Closes #10952 Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
9d82c37f98 |
fix(distributed): backend discovery hid worker-installed backends behind the controller's filesystem (#10967)
fix(distributed): backend discovery hid worker-installed backends Backend discovery endpoints filter on installed-state, which on a distributed controller derives from the controller's own filesystem. A backend lives on the worker node that runs it, so every backend an admin installed on a GPU worker read as "not installed" and vanished from the listing. #10947 fixed the sibling capability filter on the same endpoints, so a fine-tuning-capable GPU worker now made the backend listable while the installed-state filter still dropped it: the dropdown stayed empty. The controller cannot derive this locally, but it already aggregates the per-node view that GET /backends renders, so discovery reuses the active BackendManager rather than growing a second path. Three surfaces shared the root cause and route through the same helper now: - GET /backends/available (Installed is now cluster-wide) - GET /api/fine-tuning/backends - GET /api/quantization/backends The response stays a boolean rather than an installed-on-N-of-M count: per-node install state is already served by GET /backends nodes[], and per-node control by POST /api/nodes/:id/backends/install, so a summary is all these dropdowns need. A nil provider (single-node) leaves the local filesystem as the only source and reproduces today's listing exactly, and a registry error degrades to that same listing instead of blanking the catalog. Assisted-by: Claude:claude-opus-4-8 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
f735cb24c0 |
fix(worker): reap deleted backends and stop models that live on a worker (#10956)
* fix(worker): reap deleted backends and stop models that live on a worker
Three related backend-lifecycle defects, all reachable from the same
production incident on a Jetson/Thor worker: a deleted backend's gRPC
process survived ~40 minutes with its directory removed from disk, a later
model load was routed to that orphan and failed with a certifi path pointing
into the deleted directory, and the admin could not stop the model because
the frontend reported it as not loaded.
1. backend.delete orphaned the process it claimed to delete
------------------------------------------------------------
s.processes is keyed by `modelID#replicaIndex` (buildProcessKey), so the
backend name never appeared in a key and was recorded nowhere on the
process. backend.delete resolved its target via isRunning/stopBackend, whose
prefix path only matches a bare *modelID* - a delete keyed on a backend name
resolved to zero keys, the stop silently no-op'd, and the files were removed
out from under a live process.
The install fast path then handed that orphan back out: it returns any live
process for the (model, replica) slot without checking which backend started
it, so a reinstalled variant inherited the deleted backend's port.
- Record backendName on backendProcess, threaded installBackend ->
startBackend.
- Add resolveProcessKeysForBackend, matching the recorded name and resolving
alias <-> concrete via ListSystemBackends *before* DeleteBackendFromSystem
erases the metadata that carries the alias. Alias resolution failure
degrades to name-only matching so a delete never fails on it.
- backend.stop goes through resolveStopTargets, which accepts a backend
name, a model name, or an exact modelID#replica key. Its payload field is
named "backend" but is published with all three meanings: the admin UI
sends a backend name, UnloadRemoteModel sends a model name, and the
router's abandoned-load reap (#10948) sends an exact replica key.
Narrowing it to backend names alone would strand the latter two.
backend.delete stays strict - its identifier is unambiguously a backend.
- Gate the install fast path on processMatchesBackend so a slot held by a
different backend is restarted rather than reused. Processes with no
recorded name (pre-upgrade) are accepted, so rollout does not restart
every running backend.
- stopBackendExact reports a real stop failure - the process still being
alive afterwards, which is precisely what finishBackendStop already
detects to keep the entry and its port reserved - and backend.delete no
longer replies success when it knew about a process and could not kill it.
"No process was running" stays a success but is logged, so the orphan case
is visible rather than silent.
2. /backend/shutdown reported a running model as missing
---------------------------------------------------------
ModelLoader.deleteProcess short-circuits on a miss in this replica's
in-memory store. In distributed mode the authoritative record of "is this
model loaded" is the shared node registry: a frontend replica that never
served the model itself (load balancer picked a peer, or the replica
restarted) has no local entry. The remote unload path that pkg/model
documents ("when ShutdownModel is called for a model with no local process,
UnloadRemoteModel is called") sat behind that short-circuit, unreachable in
exactly the case it exists for. #10865 reworked this function but kept the
short-circuit at the top, so the gap survived that refactor.
- deleteProcess consults the remote unloader on a local-store miss, via a
shared unloadRemote helper so this branch and the existing
no-local-process branch both prefer #10865's RemoteModelContextUnloader,
preserving force propagation across the distributed boundary.
- UnloadRemoteModelContext reports ErrRemoteModelNotLoaded when no node has
the model; it previously returned nil, making a no-op stop
indistinguishable from a real one. The converse case (nodes have it, none
could be stopped) already errors since #10865 joined the per-node
failures, so that half of the original fix was dropped as redundant.
- Only when the model is absent locally AND cluster-wide does the endpoint
report not-found, now 404 naming both scopes rather than a bare 500.
- modelNotFoundErr becomes the exported ErrModelNotFound so the HTTP layer
can map it without string matching; watchdog's identity comparison becomes
errors.Is.
3. Coverage for the bounded Free() that #10865 shipped untested
----------------------------------------------------------------
The original branch also bounded the pre-stop Free(), but #10865 landed that
fix first (workerBackendFreeTimeout, applied in both stopBackendExact and
handleModelUnload). That production change is therefore DROPPED here as
superseded - master's version is strictly better, since it also releases the
supervisor mutex across the call and keeps the port reserved until
termination completes.
What #10865 did not ship is a test, and the bound is load-bearing: the
router-side reap in #10948 sends backend.stop for an abandoned load, and
against a wedged backend an unbounded Free would swallow that stop before it
reached the process. Nothing failed if the bound regressed.
The spec stands up a real gRPC backend server whose Free handler never
returns - what a Python backend looks like when its single worker thread
(PYTHON_GRPC_MAX_WORKERS=1 on 37 backends) is occupied by a stuck LoadModel.
A stub socket is not sufficient and was tried first: without a completed
HTTP/2 handshake, gRPC's own ~20s connect timeout ends the call, so that
version passed against the very bug it targets. With the connection READY,
only the caller's deadline can end it, so the spec hangs to its 60s limit if
the timeout is removed and passes with it.
Its fixture process is deliberately never started. go-processmanager v0.1.1
writes Process.pid from readPID() without synchronization, so a live process
races its own monitor goroutine under -race - reproducible with a bare
Run()+Stop() and unrelated to this spec. Since
scripts/model-lifecycle-conformance.sh runs this package with -race and is
fail-closed, starting one would turn that gate red on an upstream defect. An
unstarted process still proves the point: the stop is reached and the slot
released, which is exactly what an unbounded Free prevents.
Verified: make lint (new-from-merge-base origin/master) reports 0 issues;
scripts/model-lifecycle-conformance.sh passes all three stages including the
FizzBee liveness check (1458 states, IsLive: true).
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): keep remote unload idempotent, ask presence separately
|
||
|
zjuzhongwen
|
864c84f48b |
chore: fix some comments to improve readability (#10960)
Signed-off-by: zjuzhongwen <zjuzhongwen@outlook.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
0e0221b0f5 |
fix(vision): probe the media marker for pinned llama.cpp backend variants (#10955)
llama.cpp picks a random per-process media marker (ggml-org/llama.cpp#21962),
so LocalAI renders the prompt with a "<__media__>" sentinel and swaps in the
backend's real marker after probing ModelMetadata.
That probe was gated on an exact match against "llama-cpp", the gallery's meta
backend name. A model config pinning a concrete build ("vulkan-llama-cpp",
"cuda12-llama-cpp", "rocm-llama-cpp", ... and their -development counterparts)
runs the same llama.cpp gRPC server but skipped the probe, so MediaMarker
stayed empty, no substitution happened, and the prompt reached mtmd still
carrying the sentinel. mtmd_tokenize then counted zero markers against one
bitmap and every image request failed with "Failed to tokenize prompt".
The same early return also skipped thinking-mode detection and tool-format
marker extraction, so a pinned variant silently lost reasoning and native
tool-call parsing too.
Add IsLlamaCppBackend, which recognises the whole variant family (plus the
empty auto-detect name, which resolves to llama.cpp) while excluding
ik-llama.cpp, a separate engine that merely shares the suffix.
Fixes #10945
Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
fb4c61d1c9 |
fix(distributed): configurable remote model-load timeout, and reap the load when it times out (#10948)
* fix(distributed): make the remote LoadModel deadline configurable
The router hardcoded a 5 minute gRPC deadline for the remote LoadModel
call. Staging finishes before the timer starts, so those five minutes
cover only the worker backend's own checkpoint load and pipeline init.
A cold load of meituan-longcat/LongCat-Video-Avatar-1.5 (~83 GB) on an
ARM64 Thor worker fails at exactly 302s with DeadlineExceeded while the
backend process is still making progress (CPU time accumulating, RSS
moving as weights are mapped), so the load was cut short rather than
wedged.
Add LOCALAI_NATS_MODEL_LOAD_TIMEOUT / --model-load-timeout mirroring the
existing backend-install timeout knob, defaulting to 5m so unset
clusters keep today's behaviour.
The cold-load hold ceiling (which bounds how long one load may hold the
per-model advisory lock) was derived from the install timeout alone, so
raising the load deadline past it would have been silently clipped.
Derive it from both budgets via ModelLoadCeilingFor:
max(install + load + 5m staging margin, 25m)
With the defaults that is 15m + 5m + 5m = 25m, identical to the previous
constant, and the 25m floor means shrinking either budget can never
tighten the ceiling below what clusters relied on before.
Assisted-by: Claude:claude-opus-4-8 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): reap the abandoned replica when a remote load times out
The gRPC deadline on the remote LoadModel call only cancels the client
side. A backend blocked in a synchronous weight load never observes its
cancelled handler context, so when scheduleAndLoad gave up it left the
worker loading with nobody waiting for the result.
Observed on an ARM64 Thor worker loading LongCat-Video-Avatar-1.5: the
client returned DeadlineExceeded at 302s, and the backend process was
still alive 30 minutes later having pulled ~57GB from HuggingFace. Every
retry stacked another multi-GB loader on the worker; they had to be
reaped by hand via POST /api/nodes/:id/models/unload.
Send backend.stop for the exact `modelID#replicaIndex` process key we
just abandoned. The exact key matters: a bare model ID stops every
replica on that node, including healthy ones serving traffic.
Only a deadline or cancellation triggers the reap. Any other LoadModel
failure is the backend answering, which means its handler returned and
the process is idle - stopping it there would discard a warm process and
its downloaded weights. The reap is best-effort and never replaces the
load error the caller is waiting on.
The `modelID#replicaIndex` format was already hand-rolled in two places
(the worker's buildProcessKey and pkg/model's log store). Rather than add
a third, export model.BackendProcessKey from pkg/model, the lowest common
dependency of both sides.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-4-8 golangci-lint
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
626ae4d51e |
fix(model-artifacts): materialize longcat-video on the controller, and support companion repos (#10949)
* fix(model-artifacts): materialize longcat-video checkpoints on the controller longcat-video loads a checkpoint directory: its backend.py takes request.ModelFile when os.path.isdir(request.ModelFile) and otherwise falls back to snapshot_download. That places it in the same class as transformers/vllm/diffusers/sglang, but the allow-list added in #10910 did not enumerate it, so PrimaryArtifactSpec returned no managed artifact for a bare HuggingFace repo id. The consequence in distributed mode: nothing was acquired on the controller, ModelFileName fell through to the raw repo id, and staging skipped the resulting phantom /models/<owner>/<repo> path. The worker received a blank ModelFile, fell back to request.Model, and downloaded ~83GB from HuggingFace inside the remote LoadModel deadline - so the load could only ever fail with DeadlineExceeded while an abandoned backend process kept downloading. Note this materializes the full repository. The backend restricts its own snapshot_download with allow_patterns, and the avatar repo ships both base_model/ and base_model_int8/ where only one is ever loaded; inferred specs have no way to carry patterns today. Tracked separately. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): warn when staging skips a non-existent model path stageModelFiles logs "Staging model files for remote node" up front, then silently drops any path field that does not exist on the controller. The skip itself is legitimate and must stay: a backend outside managedArtifactBackends that takes a bare HuggingFace repo id gets an optimistically constructed path (ModelFileName falls through to the raw model reference) that was never materialized, and sources its own weights on the worker. Erroring would break those configs. But at debug level the operator is left with a reassuring staging line and no trace of the skip, so a genuine controller-side acquisition gap is indistinguishable from a healthy pass-through - it surfaces much later as a remote LoadModel timeout, on a worker that is quietly downloading tens of gigabytes. Raise the skip to warn and name the field, path, node and tracking key. Behavior is unchanged. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(model-artifacts): allow a config to declare companion artifacts A composed pipeline needs more than one HuggingFace snapshot. LongCat-Video-Avatar-1.5 loads its own transformer but takes the tokenizer, text encoder and VAE from the separate LongCat-Video base repo, so a single-artifact config cannot express it and the backend is left to fetch the second repo itself at load time. Widen the artifact model to target: model plus any number of named target: companion entries. Normalize accepts the new target and constrains a companion name to [a-z0-9][a-z0-9_-]{0,63} because that name is the option key the backend later receives; a companion may not claim primary_file, which only means anything for a load target. ModelConfig.Validate requires exactly one primary and requires it first, since Artifacts[0] is what ModelFileName, size estimation and staging all resolve from. Both acquisition paths now loop instead of touching index 0 alone: preloadOne for an already-installed config, bindPrimaryArtifact for a gallery install. Failure policy differs by provenance. An inferred primary keeps its warn-and-fall-back, because the legacy download path still exists for it. Companions are explicit by construction, so they are all-or-nothing: a config naming one is asserting the backend needs it, and failing at the acquisition boundary is far more legible than a missing-weights error surfacing later inside the backend. The cache key is deliberately unchanged. It hashes source identity only, never name or target, so every already-installed managed model still hits its existing snapshot instead of silently re-downloading. Two specs pin that: one proving a companion and a primary with identical sources agree on the key, and one pinning the digest of a known primary outright. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(model-artifacts): hand resolved companion snapshots to the backend A materialized companion is useless until the backend can find it, and its location is a content-addressed cache key that does not exist until the artifact resolves. A static gallery override cannot carry that, and persisting it into the config YAML would rot the moment a re-resolve produced a new key. Synthesize it instead at load time: each resolved companion becomes "<artifact name>:<snapshot path>" in ModelOptions.Options, reusing the key:value convention backends already parse for options like attention_backend. The value stays relative to the models directory so a remote worker can resolve it under its own ModelPath once staging has rewritten the model root. An option the author set explicitly always wins, so pinning a companion to a local checkout still beats the managed snapshot. longcat-video resolves base_model through ModelPath, the same convention qwen-tts, voxcpm, outetts and ace-step already use for companion assets. Its sibling-directory heuristic is deleted: it looked for a LongCat-Video directory next to the model, which cannot exist under the content addressed .artifacts/huggingface/<key>/snapshot layout, so it was dead code the moment the model became managed. The gallery entry declares both repositories and restricts each with allow_patterns. The avatar repo ships base_model/ and base_model_int8/ and only ever loads one, so fetching the whole repo would roughly double the download. The patterns match the entry's own options (use_distill true, use_int8 default false); enabling use_int8 here also requires adding base_model_int8/**, which is called out in the entry. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): stage managed artifact trees from the models root Staging anchored the worker's models directory on the primary snapshot whenever a model was managed, so a companion snapshot could not reach the worker at all. frontendModelsDir was derived by stripping the Model relative path off the end of ModelFile. For a managed artifact nothing matches: ModelFile is .artifacts/huggingface/<key>/snapshot while Model stays a bare HuggingFace repo id, so the strip was a no-op and the "models directory" came out as the snapshot itself. Two consequences, both silent. Staging keys lost the .artifacts/huggingface/<key>/snapshot prefix, so two snapshots of one model were indistinguishable on the worker. And a companion, which lives in a sibling snapshot directory outside the primary, fell outside that directory entirely: StagingKeyMapper.Key collapsed its files to bare basenames and resolveOptionPath could not resolve the relative option at all, so it was skipped without a word. Derive the models root from the artifact tree instead when the path runs through it, and compute the worker's ModelPath from the file's path relative to that root rather than from the Model field. The legacy layout is unaffected: where Model really is the relative path, the new derivation reduces to the old one, which a regression spec pins. This deliberately changes an invariant that router_dirstage_test.go pinned: for a managed primary, ModelFile and ModelPath were both the snapshot directory, and staging keys were relative to it. Now ModelFile is the snapshot, ModelPath is the models root above it, and keys keep the full relative path. That spec is updated rather than accommodated, with the reasoning recorded inline, because the old invariant is exactly what made a sibling companion unreachable. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
b19afb192a |
fix(distributed): backend discovery hid GPU-only backends behind the controller's capability (#10947)
* fix(backends): list backends runnable on worker nodes in distributed mode GET /backends/available filtered the gallery against the system state of the host serving the request. In a distributed deployment that host is the controller, which typically has no GPU, while the GPUs live on worker nodes. Any meta backend whose capabilities map lacks a "default" (or "cpu") key was therefore dropped from the listing entirely — longcat-video, vllm-omni, ltx-video, parakeet, edgetam and qwentts were invisible in the UI even though installing them by name on a GPU worker worked fine. Workers now report their own meta-backend capability at registration and the controller persists it on the node row. The controller cannot derive it: OS-dependent capabilities (metal, darwin-x86, nvidia-l4t) and the CUDA runtime refinements are only observable on the worker. Nodes registered before this field existed fall back to a coarse capability derived from their GPU vendor and VRAM. Backend discovery then evaluates compatibility as the union over healthy backend nodes, so a backend runnable on any node is offered while one no node can run stays hidden. Each remote capability is evaluated through a capability-pinned system state, otherwise a forced capability on the controller image (LOCALAI_FORCE_META_BACKEND_CAPABILITY or /run/localai/capability) would silently override every worker's verdict. With no registered nodes the listing is byte-for-byte what it was, so single-node deployments are unaffected. Also fixes the same-root-cause misclassification in /api/operations, which used the capability-filtered listing to decide whether an operation was a backend or a model install. A GPU-only backend installing on a worker is still a backend operation on the controller, so that lookup is now unfiltered. Assisted-by: Claude:claude-opus-4-8 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(backends): union worker capabilities in backend discovery Implementation for the specs added in the previous commit, plus the two remaining discovery endpoints. Capability-filtered backend discovery evaluated compatibility against the system state of the host serving the request. In a distributed deployment that host is the controller, which typically has no GPU, while the GPUs live on worker nodes. Any meta backend whose capabilities map lacks a "default" (or "cpu") key was dropped entirely — longcat-video, vllm-omni, ltx-video, parakeet, edgetam and qwentts were invisible in the UI even though installing them by name on a GPU worker worked fine. Workers now report their own meta-backend capability at registration and the controller persists it on the node row. The controller cannot derive it: OS-dependent capabilities (metal, darwin-x86, nvidia-l4t) and the CUDA runtime refinements are only observable on the worker. Nodes registered before this field existed fall back to a coarse capability derived from their GPU vendor and VRAM. Discovery then evaluates compatibility as the union over healthy backend nodes, so a backend runnable on any node is offered while one no node can run stays hidden. Each remote capability is evaluated through a capability-pinned system state, otherwise a forced capability on the controller image (LOCALAI_FORCE_META_BACKEND_CAPABILITY or /run/localai/capability) would silently override every worker's verdict. With no registered nodes the listing is byte-for-byte what it was, so single-node deployments are unaffected. Four surfaces shared this root cause and are all routed through the same helper now: - GET /backends/available - GET /api/fine-tuning/backends - GET /api/quantization/backends - /api/operations backend-vs-model classification, which additionally had no reason to filter by capability at all: a GPU-only backend installing on a worker is still a backend operation on the controller, so that lookup is now unfiltered. Assisted-by: Claude:claude-opus-4-8 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
9c43b2da8f |
fix(model): make backend shutdown model-scoped (#10865)
Avoid holding the global loader lock across backend lifecycle waits and propagate forced shutdown through distributed workers. Track parallel requests with in-flight counters and reserve worker ports until process termination. Add focused race tests and an authoritative FizzBee lifecycle model with a fail-closed conformance target. Assisted-by: Codex:GPT-5 [FizzBee] [Ginkgo] Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
0389495388 |
fix(webui): use relative asset base so fonts and lazy chunks honor X-Forwarded-Prefix (#10889) (#10904)
The Vite build emitted path-absolute asset URLs (base: '/'). index.html
entry scripts and the favicon were rewritten to include the reverse-proxy
prefix in serveIndex, but two reference kinds are not in index.html and so
bypassed that rewrite:
- CSS `url()` font references (e.g. Font Awesome .woff2), which the browser
resolves relative to the stylesheet and which `<base href>` never affects
- lazily-imported route chunks, whose preload base came from the absolute
Vite base
Under a subpath mount (X-Forwarded-Prefix: /llm/) both were fetched from the
origin root, 404ing — missing-glyph "tofu" icons and broken lazy-loaded pages.
Switch Vite to a relative base ('./') so every generated URL resolves against
the file that references it: CSS fonts and route chunks now load from
`/llm/assets/...`, and index.html's now-relative entry refs resolve via the
`<base href>` serveIndex already injects on every response. Root deployments
are unaffected. The existing path-absolute rewrite in app.go still covers the
public `/favicon.svg`.
Assisted-by: Claude:opus-4.8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
2dade4a9f9 |
fix(model-artifacts): gate inferred artifact materialization by backend (#10910)
The managed-artifact materializer stages a HuggingFace snapshot into a directory (.artifacts/huggingface/<key>/snapshot/). That is the right load target for directory-consuming backends (transformers, vLLM, diffusers, ...), but PrimaryArtifactSpec inferred a managed artifact from ANY HuggingFace-shaped model reference regardless of backend. A single-file backend such as llama.cpp or whisper was therefore handed the snapshot directory instead of the weight file and failed to load it. The /import-model importer already guards this with a backend allow-list (managedArtifactBackends), but the loader-side inference did not. Move the allow-list into core/config as IsManagedArtifactBackend and apply it in PrimaryArtifactSpec: only directory-consuming backends may have an artifact inferred from a bare reference; every other backend stays on the legacy download-to-file path. An explicit artifacts: block still bypasses the gate, where single-file snapshot resolution handles the load path. The importer now shares the same predicate, so both paths agree on which backends auto-materialize. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
279f5b8a93 |
fix(model-artifacts): load single-file HF snapshots from the file, not the directory (#10909)
fix(model-artifacts): load single-file HF snapshots from the file, not the dir The managed Hugging Face artifact materializer (#10825) always pointed backends at the snapshot *directory* (.artifacts/huggingface/<key>/snapshot). For a single-file model reference such as huggingface://nomic-ai/nomic-embed-text-v1.5-GGUF/nomic-embed-text-v1.5.f16.gguf, the GGUF lives *inside* that directory, so llama.cpp was handed a directory and failed with "gguf_init_from_reader: failed to read magic". This has kept the tests-aio job red on master since the feature merged (the embeddings e2e tests could not load text-embedding-ada-002). Record the single file of a one-file snapshot as Resolved.PrimaryFile and have ModelFileName() resolve to snapshot/<PrimaryFile> when it is set. Multi-file snapshots (e.g. transformers repos consumed as a directory) keep pointing at the snapshot directory. PrimaryFile is derived from the resolved contents and is deliberately excluded from the artifact cache key. estimateModelSizeBytes now derives the snapshot directory from the cache key instead of ModelFileName(), so its manifest lookup is unaffected by the file-vs-directory resolution. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
a89d780707 |
fix(gallery): keep multi-file HF install progress proportional during verify (#10908)
The artifact progress bridge mapped every PhaseVerifying event to a flat 95%. The materializer emits PhaseVerifying once per file (from each file's AfterDownload hook) and downloads run sequentially, so the first small file to finish pinned the bar at 95% - and, because progress is monotonic, it stayed at 95% for the entire remaining download (e.g. a 70GB checkpoint reporting 95% at 410MB / 69.7GB). Track per-file verify proportionally to the running aggregate bytes, the same way downloading does. CurrentBytes already reflects "completed files + this file", so the percentage advances honestly. The flat 95%/99% is now reserved for the genuinely once-per-install Committing/Persisting phases. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
4be6e22b5f |
feat(webui): surface user, client IP and user agent in API traces (#10886, #10887) (#10907)
The Operate → Traces "API Traces" panel already recorded who made each request (user_id/user_name) but never showed it, and did not capture the caller's network identity at all. Operators asked to see the requesting user (#10886) and the client IP + user agent (#10887) so a trace can be attributed to who/what issued it. Backend: add ClientIP and UserAgent to APIExchange and populate them from echo's c.RealIP() (honours X-Forwarded-For / X-Real-IP behind a trusted proxy) and the request's User-Agent header. Both are omitempty and the /api/traces swagger response is map[string]any, so this is additive. UI: add a sortable "User" column to the API traces table and a metadata block (User / Client IP / User Agent) at the top of the expanded row detail. Fields render only when present, so older buffered traces and unauthenticated/local requests degrade cleanly. Adds an e2e spec covering the new column value and the expanded metadata. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
bf484c5181 |
feat(webui): show date alongside time in the Traces view (#10888) (#10905)
The Operate -> Traces table rendered the request time with the time of day only, so entries that span more than one day were ambiguous. Add a formatDateTime helper (localized date + existing time-with-millis) and use it for the Traces "Time" column, keeping the cell on a single line. The shared formatTimestamp used by the log views is unchanged. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
futurehua
|
d3ea65a112 |
refactor: replace Split in loops with more efficient SplitSeq (#10879)
Signed-off-by: futurehua <futurehua@outlook.com> |
||
|
walcz-de
|
6ccb1130d8 |
fix(agent-ui): reset streamed text at generation boundaries in agent chat (#10664)
One agent turn runs several internal LLM generations (tool selection, reasoning, final answer) that all emit stream_event deltas over the same per-agent SSE channel. The chat page accumulated every 'content' delta into a single live bubble and ignored the 'done' boundary events, so the internal generations' text (e.g. the English tool-selection rationale) merged with — and visually corrupted — the streamed final answer. Reset the accumulated content/reasoning on 'done': each generation gets a clean live bubble, and the authoritative full answer still arrives via the final json_message event as before. Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> |
||
|
Tai AnandAnai-Guo
|
45b8047736 |
fix(p2p): serialize access to p2pCtx/p2pCancel (#10839) (#10861)
StopP2P() read and wrote a.p2pCtx/a.p2pCancel without holding a.p2pMutex, and StartP2P() reassigned both fields with no lock at all -- including when RestartP2P() calls it from a background goroutine after releasing the mutex. Both paths are reachable from POST /api/settings (empty p2p_token -> StopP2P, non-empty -> RestartP2P), so concurrent requests race on the same fields. Take a.p2pMutex in StopP2P and around the field publication in StartP2P, factor the shared teardown into stopP2PLocked() so RestartP2P reuses it, and route the goroutine error path through StopP2P instead of touching a.p2pCancel unlocked. Signed-off-by: Anai-Guo <antai12232931@anaiguo.com> Co-authored-by: Anai-Guo <antai12232931@anaiguo.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
c1a891662c |
refactor(settings): single declarative registry for runtime settings (fixes the #10845 bug class) (#10864)
* feat(settings): add declarative runtime-settings field registry One fieldSpec row per RuntimeSettings field, with a reflection completeness spec so a field added without a registry row is a red test instead of a silently-dropped setting (the #10845 bug class). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 * refactor(settings): drive ToRuntimeSettings/ApplyRuntimeSettings from the field registry Behavior-preserving: ~350 hand-written per-field lines become two loops over runtimeSettingsFields, gated by a To->Apply->To round-trip spec. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 * feat(settings): baseline-driven startup merge for persisted runtime settings ApplyRuntimeSettingsAtStartup compares the live config against DefaultRuntimeBaseline (option-less-run defaults incl. kong-injected flag defaults) instead of per-field == 0 guards. Fixes persisted lru_eviction_max_retries, tracing_max_items, agent_job_retention_days, memory_reclaimer_threshold, galleries and autoload flags being silently ignored at boot. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 * fix(settings): registry-driven startup merge, applied before consumers loadRuntimeSettingsFromFile becomes a thin wrapper over ApplyRuntimeSettingsAtStartup and runs at the top of New(), before model configs capture app-level defaults. WithThreads stops eagerly resolving 0 so a persisted thread count survives restart while LOCALAI_THREADS still wins (#10845); the physical-core fallback moves after the merge. Also: run.go now injects the memory-reclaimer threshold unconditionally so the option-less boot matches DefaultRuntimeBaseline and a UI-saved threshold survives restart. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 * refactor(settings): file watcher delegates to the registry merge; shared API-key merge Manual edits to runtime_settings.json now behave like a boot-time load (env still wins) instead of the inverted diverged-from-startup guard that ignored most manual edits. MergeAPIKeys dedups env keys in one place for the endpoint and the watcher. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 * docs(settings): document unified runtime-settings precedence Document the single env/CLI > runtime_settings.json > defaults rule, applied identically at boot, on POST /api/settings, and on manual file edits, plus the two known limitations (default-valued env vars are indistinguishable from unset; API-changed fields hot-apply on the next restart only). Also add a completion debug log when the watcher applies runtime_settings.json. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 * test(settings): reset the global VRAM cap leaked by the round-trip spec The round-trip spec applies vram_budget=12GiB, whose post-loop hook installs a process-global default cap; without a reset every spec ordered after it runs under that phantom budget. Also drop a stale enumeration in the ApplyRuntimeSettings doc comment. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Tai AnandAnai-Guo
|
dc2cc4da43 |
fix(audio-transform): serialize WebSocket writes to avoid concurrent-write panic (#10857)
* fix(audio-transform): serialize WebSocket writes to avoid concurrent-write panic AudioTransformStreamEndpoint writes to the same Gorilla WebSocket connection from two goroutines: the backend-forwarding goroutine emits binary PCM frames (and can call sendWSError on a backend recv error), while the read loop calls sendWSError for malformed mid-stream JSON or a backend send failure. Gorilla WebSocket permits only one concurrent writer, so these writers race and can panic with "concurrent write to websocket connection", resetting the client session; a -race build reports the data race directly. Wrap the connection in a lockedConn that serializes WriteMessage behind a mutex, mirroring the existing lockedConn used by the openresponses WebSocket endpoint. Reads stay on the single read loop, so only writes need the lock. Fixes #10844 Signed-off-by: Tai An <antai12232931@outlook.com> * chore: empty commit to re-trigger checks Signed-off-by: Anai-Guo <antai12232931@anaiguo.com> --------- Signed-off-by: Tai An <antai12232931@outlook.com> Signed-off-by: Anai-Guo <antai12232931@anaiguo.com> Co-authored-by: Anai-Guo <antai12232931@anaiguo.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
bbe018c1a0 |
feat(bonsai): PrismML llama.cpp fork backend + Bonsai/Ternary-Bonsai gallery models (#10834)
feat(bonsai): add PrismML llama.cpp fork backend + Bonsai gallery models Adds a new `bonsai` backend that runs the PrismML fork of llama.cpp (github.com/PrismML-Eng/llama.cpp, `prism` branch), which ships the Q1_0 (1-bit) and Q2_0 (ternary / 1.58-bit) weight-quantization kernels used by the Bonsai and Ternary-Bonsai models. Stock llama.cpp cannot decode these quants. Modeled on the turboquant backend: reuses backend/cpp/llama-cpp/grpc-server.cpp against the fork's libllama via a thin wrapper Makefile, so the sub-2-bit models are served with the same OpenAI-compatible API. No grpc-server allow-list patch is needed (bonsai adds weight quants, transparent to the server, not KV-cache types), and the reused server compiles cleanly against the fork with no skew patches (validated locally via a CPU docker build; patches/ is present but empty for any future re-pin skew). Backend wiring: backend/cpp/bonsai/, .docker/bonsai-compile.sh, backend/Dockerfile.bonsai, top-level Makefile targets, backend-matrix.yml build rows (CPU, CUDA 12/13, L4T, SYCL f32/f16, Vulkan, ROCm/hipblas), backend/index.yaml meta-backend + per-platform images, and a nightly bump_deps entry tracking the `prism` branch. Gallery: 8 entries across 4 families - bonsai-8b-1bit, ternary-bonsai-8b (+g64, +pq2), bonsai-27b-1bit (vision), ternary-bonsai-27b (+pq2, +g64, vision). The 27B models wire the mmproj vision tower; the DSpark speculative drafter GGUFs are not wired (custom semi-autoregressive drafter, not a standard llama.cpp draft model). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Tai AnandAnai Guo
|
808312b4b9 |
fix(watchdog): guard StopWatchdog with watchdogMutex to prevent double close (#10841) (#10859)
fix(watchdog): guard StopWatchdog with watchdogMutex to prevent double close StopWatchdog checked, closed and cleared a.watchdogStop without holding a.watchdogMutex, while startWatchdog and RestartWatchdog reassign and close the same channel under that lock. POST /api/settings dispatches to StopWatchdog or RestartWatchdog depending on ApplicationConfig.WatchdogShouldRun(), so both are reachable concurrently. Two callers can observe a non-nil watchdogStop and both close it, which panics with 'close of closed channel' and takes the server down. Take the mutex, matching the other two writers. StopWatchdog is only called from the settings handler, which holds no lock, so this cannot deadlock. Fixes #10841 Co-authored-by: Anai Guo <antai12232931@anaiguo.com> |
||
|
Tai An
|
5fe48e4910 |
fix(backend): don't crash the whole process on an invalid cutstrings/extract_regex (#10855)
Finetune() compiled every model cutstrings/extract_regex entry via regexp.Compile
and called xlog.Fatal on failure, which terminates the entire local-ai process.
A single model config with an invalid regex (e.g. cutstrings: ["("]) turns one
/v1/chat/completions request into a process-level denial of service.
Log the compile error and skip the offending pattern instead. The mutex is
released before continuing, and skipping avoids dereferencing the nil regexp
that removing the fatal would otherwise leave behind.
Fixes #10843
Signed-off-by: Tai An <antai12232931@outlook.com>
|
||
|
Tai AnandAnai-Guo
|
688f904a10 |
fix(runtime-settings): apply persisted threads/context_size/f16 at startup (#10853)
ApplyRuntimeSettings persists the performance settings (threads, context_size, f16) on the live /api/settings path, but the startup loader loadRuntimeSettingsFromFile never read them back, so a value saved via the Middleware UI was silently ignored on the next restart: the model booted with the CLI/physical-core default and GET /api/settings echoed that default instead of the saved value (#10845). Threads needs special handling: unlike context_size/f16, WithThreads eagerly resolves an unset (0) value to xsysinfo.CPUPhysicalCores() at option-apply time, so options.Threads is never 0 in the loader and the usual "== default" heuristic cannot tell an env/CLI value from the physical-core fallback. Detect LOCALAI_THREADS/THREADS explicitly so the env still wins over the persisted file value. Signed-off-by: Anai-Guo <Anai-Guo@users.noreply.github.com> Co-authored-by: Anai-Guo <Anai-Guo@users.noreply.github.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
8cec22c3b7 |
feat(vram): per-node VRAM allocation budget (LOCALAI_VRAM_BUDGET) (#10833)
* feat(vram): add vrambudget primitive for per-node VRAM caps Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): apply default VRAM budget in xsysinfo aggregate getters Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): wire LOCALAI_VRAM_BUDGET flag to xsysinfo default budget Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): persist VRAM budget via runtime settings with live apply Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(vram): reset process-global VRAM budget after runtime-settings spec Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): add VRAM budget field to Settings page Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): store and enforce per-node VRAM budget in the node registry Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): apply per-node VRAM budget in router hardware defaults Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): report worker VRAM budget in node registration The distributed worker now reports its operator-set VRAM budget string (LOCALAI_VRAM_BUDGET) to the server on registration. The worker keeps reporting RAW total/available VRAM and never sets the xsysinfo process-global budget (that stays standalone-only); the server resolves and enforces the budget uniformly (Task 6). Also closes a Task 6 gap: on re-registration, a struct Updates zero-skips an empty budget, so a worker that dropped LOCALAI_VRAM_BUDGET left the stale cap in place. For non-admin-override nodes the budget columns are now force-written (map Updates) even when empty, so removing the env var clears the cap; admin overrides are preserved unchanged. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * style(vram): drop em dash from worker-clear comment Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): add node VRAM budget admin endpoints Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): add node VRAM budget control to the node UI Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): expose set_node_vram_budget MCP admin tool Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(vram): document LOCALAI_VRAM_BUDGET and node VRAM budget UI Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vram): avoid double-applying VRAM budget in GetResourceAggregateInfo The GPU-branch aggregate returned by GetResourceInfo is sourced from GetGPUAggregateInfo, which already caps total/free/used against the process-wide VRAM budget. GetResourceAggregateInfo then applied the budget a second time. For an absolute budget this is idempotent, but for a percentage budget b.Apply resolves the ceiling as a fraction of its input total, so a second pass yields P*(P*T) instead of P*T and distorts UsagePercent (read by the memory reclaimer in pkg/model/watchdog.go). Remove the redundant second application so the budget is applied exactly once, against the raw physical totals, upstream in GetGPUAggregateInfo. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(vram): implement SetNodeVRAMBudget on mcp assistant test stub The LocalAIClient interface gained SetNodeVRAMBudget; the stubClient in core/http/endpoints/mcp used by the assistant tests is a separate implementer and needs the method too (broke golangci-lint typecheck and both test jobs). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
3601174ce0 |
fix(distributed): make per-node backend upgrade actually upgrade (#10838)
* test(core/http): make the suite's HTTP port overridable app_test.go and openresponses_test.go hardcoded 127.0.0.1:9090. When another service already listens on 9090 the suite does not fail fast: the server goroutine logs the bind error and the specs then poll whatever is squatting the port until Eventually times out. On machines where 9090 is permanently taken this makes the pre-commit coverage gate impossible to pass. Introduce testHTTPAddr, defaulting to 127.0.0.1:9090 (what CI has always used) and overridable via LOCALAI_TEST_HTTP_PORT for local runs. Assisted-by: Claude:claude-fable-5 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): make per-node backend upgrade actually upgrade The node detail page's Upgrade button reused the node-scoped install path (POST /api/nodes/:id/backends/install). That fires NATS backend.install with force=false, and the worker's install handler is deliberately "ensure installed": when the backend binary already exists on disk it short-circuits without touching the gallery. Since only an installed backend can be upgraded, the whole chain was a guaranteed successful no-op - the UI then toasted "backend upgraded" without even waiting for the async job. Route upgrades through the real force-reinstall path instead: - BackendManager.UpgradeBackend now receives the ManagementOp (like InstallBackend already did) so implementations can honor op.TargetNodeID. - DistributedBackendManager.UpgradeBackend scopes the backend.upgrade fan-out to op.TargetNodeID when set, and errors when the target node does not report the backend as installed. - New POST /api/nodes/:id/backends/upgrade endpoint enqueues an Upgrade=true node-scoped op (async 202 + jobID, mirroring install). - NodeDetail UI calls the new endpoint and reports the dispatch ("Upgrading ... on this node...") instead of claiming success; the Operations panel tracks the actual job. Verified against a live local cluster (NATS + Postgres + two workers): the target worker stops the running process, force-reinstalls from the gallery and re-downloads the OCI image; the second worker receives no backend.upgrade event; upgrading a backend missing from the target node fails the job with a clear error. Assisted-by: Claude:claude-fable-5 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
bcc41219f7 |
feat: materialize Hugging Face model artifacts (#10825)
* feat(config): add model artifact source contract Assisted-by: Codex:GPT-5 [Codex] * feat(downloader): add authenticated raw-byte progress Assisted-by: Codex:GPT-5 [Codex] * feat(huggingface): resolve immutable snapshot manifests Assisted-by: Codex:GPT-5 [Codex] * feat(models): add artifact storage primitives Assisted-by: Codex:GPT-5 [Codex] * feat(models): materialize pinned Hugging Face snapshots Assisted-by: Codex:GPT-5 [Codex] * feat(models): bind managed snapshots at runtime Assisted-by: Codex:GPT-5 [Codex] * feat(gallery): materialize model artifacts during install Assisted-by: Codex:GPT-5 [Codex] * feat(gallery): declare managed Hugging Face artifacts Assisted-by: Codex:GPT-5 [Codex] * feat(models): preload managed model artifacts Assisted-by: Codex:GPT-5 [Codex] * fix(gallery): retain shared artifact caches on delete Assisted-by: Codex:GPT-5 [Codex] * feat(models): report artifact acquisition progress Assisted-by: Codex:GPT-5 [Codex] * refactor(backends): load managed models from ModelFile Assisted-by: Codex:GPT-5 [Codex] * refactor(backends): load staged speech model snapshots Assisted-by: Codex:GPT-5 [Codex] * refactor(backends): use staged snapshots in engine backends Assisted-by: Codex:GPT-5 [Codex] * test(distributed): cover staged artifact snapshots Assisted-by: Codex:GPT-5 [Codex] * docs: explain managed model artifacts Assisted-by: Codex:GPT-5 [Codex] * docs: add product design context Assisted-by: Codex:GPT-5 [Codex] * feat(ui): show model artifact download progress Assisted-by: Codex:GPT-5 [Codex] * Eagerly materialize Hugging Face artifacts Materialize HF-backed model references as managed GGUF artifacts during load, with lazy download retained only as fallback. Assisted-by: Codex:GPT-5 [shell] * Refactor HF downloads through a shared executor Assisted-by: Codex:GPT-5 [shell] * drop Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
b224c96db6 |
fix(config): only inject llama.cpp serving options on the llama.cpp path (#10822)
SetDefaults injected the llama.cpp server options cache_reuse (ApplyServingDefaults) and parallel (ApplyHardwareDefaults, re-applied per selected node by the distributed router) onto every model config regardless of backend. Every other backend ignores options it does not understand, so this was harmless until longcat-video, which strictly validates its options and fails LoadModel with "unknown model option(s): cache_reuse, parallel". Gate both injections behind a new UsesLlamaCppServingOptions allow-list (llama-cpp plus the empty/auto-detect case that resolves to llama.cpp from a GGUF file, mirroring how llamaCppDefaults is registered). This follows the existing UsesLlamaSamplerDefaults precedent for llama-only defaults. The typed NBatch field is deliberately left alone: it is a proto field every backend simply ignores, which is why batch never triggered the error. Also harden the longcat-video backend to warn-and-ignore unknown model options and request params through a testable select_known_options helper, matching the other LocalAI Python backends, so a future server-injected option cannot break loading again. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ijasandajuijas
|
a5aa56db81 |
fix: preserve uploaded file content when regenerating a non-last answer (#10819)
handleRegenerate rebuilt the outbound message from the display-only
message.files metadata ({name, type: 'file'|'image'|..., content}),
which doesn't carry the base64/textContent payload sendMessage's
file-building loop expects. As a result, regenerating any answer whose
own question had an attachment silently dropped that attachment from
the resent message. This wasn't fork-specific, but forking a chat and
then regenerating an earlier (now non-last) answer is the natural way
to hit it.
Fix by reusing the original message's already-assembled `content`
verbatim (it already has the file text / image_url / audio_url /
video_url parts embedded from the first send) instead of trying to
reconstruct it from lossy display metadata.
Fixes #10806
Assisted-by: Claude:claude-sonnet-5
Signed-off-by: ajuijas <189517297+ajuijas@users.noreply.github.com>
Co-authored-by: ajuijas <189517297+ajuijas@users.noreply.github.com>
|
||
|
Dedy F. Setyawan
|
b10e330590 |
feat(react-ui): localize MediaHistory, AudioTransform, and Sound components (#10802)
- Replace hardcoded text with useTranslation hook in UI components - Add localization support for both English (en) and Indonesian (id) locales Signed-off-by: Dedy F. Setyawan <dedyfajars@gmail.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
4056283aa4 |
[voice] feat: add managed voice cloning profiles (#10799)
* feat(ui): add voice library workflow Give administrators a production-ready flow to record or upload consented reference audio, manage reusable profiles, inspect API usage, discover compatible models, and hand a saved voice directly to text-to-speech. Assisted-by: Codex:gpt-5 * feat(voice): add managed voice cloning profiles Make reusable reference voices manageable through the admin API instead of requiring model-directory and YAML edits. Discover compatible installed and gallery models from server-side backend capabilities, retain explicit model configuration controls, and stage saved references for supported backends. Expose profile management through REST and MCP, document backend-specific behavior, and cover the workflow from profile creation through real Qwen3-TTS synthesis. Harden the agent-job HTTP test against completion racing cancellation. Assisted-by: Codex:gpt-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
b00422e45f |
feat(backends): add LongCat video and avatar generation (#10792)
* feat(backends): add LongCat video and avatar generation Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] [web] * refactor(config): declare model I/O modalities Make model configs declare input and output modalities so capability discovery no longer branches on backend or checkpoint names. Complete the LongCat gallery and user documentation, make the SDPA patch apply to the pinned upstream revision, and stabilize the Agent Jobs race exposed by the required hook. Assisted-by: Codex:GPT-5 [web] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
1f9fda7138 |
fix(backends): refuse foreign model loads in opus and local-store (#10769)
When a model config has no explicit backend, the model loader greedily probes every installed backend and binds to the first Load that succeeds. opus and local-store were the only in-tree backends with no model artefact to validate, so they accepted anything — an LLM installed after them could silently bind to the audio codec or the vector store and then fail at inference with "unimplemented" (see #9287). opus now accepts only its own name (what the realtime WebRTC path sends) or none. local-store namespaces are arbitrary (router caches, biometrics, user-named stores), so core's StoreBackend now marks genuine store loads with a store:// prefix on the gRPC model name and the backend refuses names without it; core and backend ship from the same release, so the convention upgrades in lockstep. Also repair the bit-rotted 'make test-stores' bootstrap (the suite never registered external backends, so BACKENDS_PATH was dead weight) and add the Load-validation rule to the adding-backends checklist. Related: #9287 Assisted-by: Claude:claude-fable-5 golangci-lint Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
94bdc825dc |
feat(backend): add moss-transcribe-cpp backend (MOSS-Transcribe-Diarize) (#10756)
C++/ggml transcription + speaker diarization + timestamps backend. Purego dlopens libmoss-transcribe.so (ggml statically linked) from moss-transcribe.cpp and serves offline AudioTranscription, parsing the [start][Sxx]text[end] output into segments with nanosecond timestamps. Adds the importer (surfaces in GET /backends/known), backend-matrix (Linux + Darwin/metal), backend/index.yaml, and a gallery entry (default q5_k GGUF from mudler/moss-transcribe.cpp-gguf). Local L0 smoke (go build + go test ./... = 16 pass, golangci-lint 0 issues) passed against the real libmoss-transcribe.so. The pre-commit coverage gate (full pkg/core + tests/e2e) could not run in the authoring sandbox (no live models, port 9090 held); CI must enforce it before merge. Assisted-by: Claude:claude-opus-4-8 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Dedy F. Setyawan
|
294487eb61 |
fix(ui): prevent table container from breaking flexbox layout (#10754)
Signed-off-by: Dedy F. Setyawan <dedyfajars@gmail.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
5569b2de56 |
feat(config): context_size: -1 to auto-use model's full trained context (#10752)
* feat(config): clamp negative context_size to default in EffectiveContextSize Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * feat(config): resolve context_size=-1 to model trained max with VRAM warn Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * fix(config): treat negative context_size as unset when GGUF is unparseable Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * docs(config): document context_size=-1 auto-max Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * docs(backend): drop em dashes from EffectiveContextSize comment Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
40dae953f4 |
feat: interleaved thinking with tool calls (reasoning_content alias + Anthropic thinking blocks) (#10744)
* feat(schema): accept reasoning_content as inbound alias for reasoning Interleaved-thinking clients (cogito, vLLM/DeepSeek-style) emit reasoning_content on assistant turns. Accept it as an inbound alias so reasoning survives the tool-result loop; canonical reasoning wins when both are present. Emission is unchanged (still reasoning). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(schema): pin interleaved reasoning+tool_calls round-trip Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(openai): pin reachedTokenBudget truncation detection Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(anthropic): add thinking and signature fields to content blocks Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(anthropic): parse inbound thinking blocks into reasoning Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(anthropic): emit thinking blocks with synthetic signature on tool turns Extract buildAnthropicContentBlocks so non-streaming content assembly is unit-testable, and prepend a thinking block (with an opaque synthetic signature) before text/tool_use blocks when the request opts into thinking. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(anthropic): stream thinking_delta and signature_delta before tool_use Extract anthropicStreamSequence so the streaming block order is unit-testable, and emit content_block_start(thinking) -> thinking_delta -> signature_delta -> content_block_stop before the tool_use block sequence when thinking is enabled. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add interleaved thinking with tool calls guide Add a features guide describing interleaved thinking: an assistant turn carrying reasoning and tool_calls together, the reasoning-round-trip contract (including the reasoning_content inbound alias and Anthropic thinking blocks with a synthetic signature), per-backend enablement (reasoning_format for llama.cpp, reasoning_parser/tool_call_parser for vLLM/SGLang plus the vLLM auto-config hook), a worked request/response example, and known limitations. Cross-link from model-configuration, text-generation, and openai-functions. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
a0ed395cf9 |
fix(auth): accept EC/PS/EdDSA-signed OIDC ID tokens, not just RS256 (#10736)
The OIDC verifier was built with a bare oidc.Config{ClientID: ...}, so
go-oidc applied its default of accepting RS256-signed ID tokens only. An
identity provider configured with an EC signing key (e.g. Authentik) issues
ES256-signed tokens, and the callback failed verification with:
failed to verify ID token: oidc: malformed jwt: unexpected signature
algorithm "HS256"; expected ["RS256"]
surfacing to the user as HTTP 500 "failed to fetch user info" (#10677; the
underlying cause became visible after the logging fix in #10679).
Set SupportedSigningAlgs to the standard asymmetric algorithms
(RS256/384/512, ES256/384/512, PS256/384/512, EdDSA). All are verified
against the provider's published JWKS. HS256 is intentionally excluded: it
is symmetric and would validate against the client secret, a different and
security-sensitive trust model.
Tested with a functional spec that signs an ES256 ID token and confirms it
verifies with the configured algorithms and is rejected under go-oidc's
RS256-only default (using oidc.StaticKeySet, no network).
Closes #10677
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
LocalAI [bot]andEttore Di Giacinto
|
40c29db8c4 |
fix(logs): capture backend logs by default in single mode (#10742)
Backend log capture into the per-model BackendLogStore (which feeds the UI "Backend Logs" page and /api/backend-logs) was opt-in and off by default in single mode, while worker/distributed mode force-enables it via SetBackendLoggingEnabled(true). There was no CLI flag either, so the only way to populate the store was the Settings UI toggle - and the page was silently empty out of the box. Distributed "just worked"; single mode looked broken. Default EnableBackendLogging to true in NewApplicationConfig so single mode matches worker mode. The store is a small in-memory ring buffer, so the cost is negligible. Now that the default is on, loadRuntimeSettingsFromFile's usual "only flip false->true" merge would ignore a persisted false and revert the UI toggle-off on every restart. There is no env var/CLI flag for this setting, so an explicit persisted value is now authoritative in both directions, letting the toggle-off survive a restart. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
github-actions[bot]andmudler
|
0ae84be362 |
chore: bump inference defaults from unsloth (#10741)
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
cd65a1f645 |
fix(transcription): honor model-config language/translate + OpenAI language form field (#10731)
fix(transcription): honor model-config language/translate, add language form field The /v1/audio/transcriptions endpoint read only input.Language / input.Translate from the parsed request, and the request middleware never populates those from a multipart upload -- nor did it read a `language` form field. As a result the model config's parameters.language / parameters.translate (a valid PredictionOptions field under `parameters:`) were silently ignored, and multilingual models like canary defaulted to translating into English even when the YAML set language: ru, translate: false (#10655). Resolve both with clear precedence: the request form field wins, then any language on the parsed request, then the model config default. This also makes the endpoint honor OpenAI's `language` form parameter, which was not read before. Applies to both the streaming and non-streaming paths (the resolved values are built into the shared TranscriptionRequest). Note this ensures the language/translate flags reach the backend; whether a given engine acts on them is up to the backend. Closes #10655 Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
97175f4b5a |
feat(model): debounce model loads after a failure to stop retry-storms (#10728)
A client that keeps polling a model whose load fails (e.g. a backend that crashes deterministically on init) triggered a fresh backend start on every request: request -> load -> crash in ~10s -> 500, repeat on the next poll. Each attempt could leak GPU/CUDA state, and under LOCALAI_SINGLE_ACTIVE_BACKEND it kept stealing the active slot from healthy models. The existing loading-coalesce map only dedups *concurrent* loads, so sequential polls were never covered. Track load failures per modelID in ModelLoader. After a load fails, refuse fresh load triggers for that model until a cooldown elapses, returning a typed ModelLoadCooldownError that the HTTP layer maps to 503 with a Retry-After header. The cooldown grows exponentially per consecutive failure (base, doubling, capped at 5m) and resets on a successful load. The coalesced follower-retry of an in-flight burst bypasses the gate, so a genuinely concurrent burst still gets its one retry -- only new, independent triggers are refused, matching the report's "refuse new load-triggers" wording. Configurable via --model-load-failure-cooldown / LOCALAI_MODEL_LOAD_FAILURE_COOLDOWN (default 10s, 0 disables), plumbed through ApplicationConfig and applied unconditionally at startup. Closes #10719 Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
1d5139f0a0 |
fix(gallery): make backend (re)install a clean replace instead of an overlay (#10726)
InstallBackend extracted the artifact directly into the target directory with no pre-clean, so a reinstall overlaid the new files onto the old ones. Files present in a previous version but absent in the new artifact (a stale .so, an orphaned package dir) survived and could shadow the new build at import time -- e.g. an old vllm shared object lingering next to a freshly pulled one. Only a failed download cleaned the directory. Stage the download/extraction into a `<name>.install-tmp` dir, validate run.sh is present, write metadata, then atomically swap it into place (rename current -> .install-backup, staging -> current, drop backup), rolling back on failure. This mirrors the atomic swap UpgradeBackend already performs, so install and upgrade now leave identical on-disk state with no orphaned files. Reported as part of #10720. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
weifanglab
|
a6cf67cc6b |
refactor: use slices.Contains to simplify code (#10702)
Signed-off-by: weifanglab <weifanglab@outlook.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
85f5267ed2 |
fix(llama-cpp): cap single-pass embedding batch to fit VRAM (#10695)
* fix(llama-cpp): cap single-pass embedding batch to fit VRAM Embedding/score/rerank all decode or pool the whole input in one physical batch, so EffectiveBatchSize sized the batch to the full context window. For a large context that makes n_ubatch huge, and the per-device CUDA compute buffer (forward-graph scratch, ~n_ubatch * n_ctx, NOT split across GPUs) balloons into multi-GiB: a large-context embedding model then aborts on load (exitCode=-1) even with plenty of free VRAM. Reproduced with qwen3-embedding-4b (context 40960 -> n_batch 40960 -> abort) and qwen3-embedding-0.6b (n_batch 8192); pinning batch:512 avoided it. This is the same root cause as issue #10485 (a large context turns the batch into multi-GiB of scratch that must fit on a SINGLE card), but the single-pass path bypassed the VRAM headroom guard the config layer already had — it returned the unbounded context as the batch with no GPU awareness. Make the single-pass batch VRAM-aware: cap it to the largest batch whose compute buffer fits the per-device VRAM headroom, clamped to [DefaultPhysicalBatch, ctx], reusing the existing computeBufferBytesPerCell and headroom-divisor math (no duplication). Unknown per-device VRAM (0) stays conservative (DefaultPhysicalBatch, not the context) so a detection gap can't OOM. The GPU is resolved through an injectable package var (config.LocalGPU, backed by sync.Once-cached xsysinfo detection) so the per-request router call stays cheap and tests inject a deterministic device. Explicit batch: still wins. An input longer than the cap can no longer be pooled in one pass — the accepted tradeoff, since a batch that OOMs the device processes nothing. Assisted-by: Claude:claude-opus-4-8 golangci-lint go-test Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(config): single-pass batch follows context on unknown VRAM The single-pass (embedding/score/rerank) batch cap must only shrink the batch when the per-device VRAM ceiling is KNOWN. On unknown VRAM (CPU-only or a GPU detection gap) SinglePassBatchForContext returned DefaultPhysicalBatch, which under-sized the batch below the context — over-trimming score/embed/rerank inputs (the modelTokenTrim middleware regression) with no OOM benefit on CPU where the compute buffer lives in system RAM. Return the full context instead, preserving the original single-pass behavior; the VRAM cap stays a downward safety that only engages when VRAM is known. Assisted-by: Claude:claude-opus-4-8 [go-test go-vet] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |