mirror of
https://github.com/mudler/LocalAI.git
synced 2026-07-30 09:57:57 -04:00
chore/bump-inference-defaults
7296 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
mudler
|
a6e9f47476 | chore: bump inference defaults from unsloth | ||
|
Tai An
|
efb43776ba |
fix(chatterbox): pin cublas12 torch/transformers and setuptools so the backend loads (fixes #11070) (#11074)
fix(chatterbox): pin cublas12 torch/transformers and setuptools so the backend loads
The cuda12-chatterbox gallery backend fails to load on a fresh install
because several deps in requirements-cublas12.txt are unpinned:
- torch/torchaudio: unlike requirements-cublas13.txt and
requirements-cpu.txt, this file has no --extra-index-url, so pip pulls
a wheel whose CUDA runtime (cu130) is newer than the host driver
supports ("NVIDIA driver on your system is too old"). Add the cu124
index and pin torch/torchaudio 2.6.0+cu124.
- transformers: resolves to 5.x, which dropped LlamaConfig.rope_theta
that chatterbox-tts 0.3.1's T3 config still reads. Cap to <5.
- setuptools: 81+ dropped pkg_resources, which perth imports under a
bare try/except and silently sets PerthImplicitWatermarker=None,
making ChatterboxTTS.__init__ raise 'NoneType' object is not callable.
Cap to <81 in requirements.txt.
Fixes #11070
Signed-off-by: Tai An <antai12232931@anaiguo.com>
Co-authored-by: Tai An <antai12232931@anaiguo.com>
Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>
|
||
|
mudler's LocalAI [bot]
|
d8a1e3c2e4 |
fix(realtime): echo response.metadata on response.created and response.done (#11198)
response.create accepts a metadata map and ResponseCreateParams has carried the field all along, but triggerResponse never copied it onto the Response it emits, so both terminals went out with metadata omitted. That field is the only thing tying a terminal event back to the response.create that asked for it. Our own doc comment on ResponseCreateEvent says so — "the metadata field is a good way to disambiguate multiple simultaneous Responses" — and it is what makes an out-of-band response (conversation: "none") usable at all: a client running one alongside the spoken conversation has no way to tell its own answer from the conversation's, so it waits for a reply it already received and gave away. Found from the client side: a headless text turn injected into a live session was answered correctly in about a second, and the caller still blocked until its own two-minute timeout because it could not recognise the answer. Carry the map on liveResponse so all three terminals (in_progress, cancelled, completed) report it, and leave it omitted when response.create sent none. Assisted-by: Claude:claude-opus-5 gofmt Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-bot
|
9bfd71387b |
feat(stores): add Valkey Search vector store backend (#11196)
* feat: add Valkey Search vector store backend Add a new built-in Go gRPC store backend 'valkey-store' that implements the four Stores RPCs (Set/Get/Delete/Find) against the Valkey Search module (FT.*) using the pure-Go github.com/valkey-io/valkey-go client. It is selected via the existing per-request 'backend' field on /stores, so there is no proto or HTTP API change, and it mirrors the in-memory local-store while adding persistence across restarts and opt-in HNSW. Each vector is a Valkey HASH keyed by hex(little-endian float32); the index is created lazily on first Set (FLAT+COSINE by default), cosine similarity is derived as 1-distance, and namespaces get a collision-resistant token. Includes unit tests (valkey-go mock) and env-gated integration tests against valkey/valkey-bundle, plus build/matrix/gallery wiring and docs. Assisted-by: Kiro:claude-opus-4.8 golangci-lint Signed-off-by: Daria Korenieva <daric2612@gmail.com> * Address review feedback: recover persisted index dimension, harden Find - Load now recovers the persisted vector DIM from FT.INFO (not just index existence), so a post-restart Set/Find validates against the real DIM instead of silently re-learning a wrong one and dropping mismatched vectors from the index. This also restores Find's dimension check after a restart. - StoresFind treats a dropped/missing index as an empty store (empty result, no error) and clears the stale indexCreated flag, matching local-store's empty-store behaviour. - StoresSet reuses checkDims for its per-key length check so the four RPCs share one dimension-guard implementation. - Add unit tests for FT.INFO dimension recovery, loadIndexState, and the dropped-index Find path. Assisted-by: Kiro:claude-opus-4.8 Signed-off-by: Daria Korenieva <daric2612@gmail.com> * Address review feedback: TLS ServerName/CA, Find nil-check, config fail-fast Addresses external review comments on the valkey-store backend: - StoresFind now rejects a nil/empty query Key before dereferencing it, so a malformed gRPC request can no longer panic the backend. - TLS: derive ServerName (SNI) from the VALKEY_ADDR host so certificate verification works for IP-addressed endpoints, and add VALKEY_TLS_CA_CERT (custom CA bundle) and VALKEY_TLS_SKIP_VERIFY (testing-only) knobs. - Config integer parsing now fails fast on a malformed value (e.g. VALKEY_HNSW_M=1x6) instead of silently defaulting, matching the fail-fast behaviour of the index-algo/distance-metric validation. - Add VALKEY_DB (SELECT n) support for logical-DB isolation. - Cap the human-readable part of a namespace token at 64 chars so a very long model name cannot produce an unbounded key prefix / index name (the appended short hash keeps distinct namespaces collision-free). - Document the KNN-query injection-safety invariant (fields are constants) and why StoresGet uses a single aggregate DoMulti deadline for reads. - Unit tests for the Find nil/empty-key guard, fail-fast HNSW parsing, and VALKEY_DB parsing/validation; docs + .env updated for the new vars. Assisted-by: Kiro:claude-opus-4.8 golangci-lint Signed-off-by: Daria Korenieva <daric2612@gmail.com> * Address review feedback: configure valkey-store via model config richiejp asked that the valkey-store backend take its configuration from a model config rather than process-wide VALKEY_* environment variables, so multiple stores can each have their own Valkey config within one LocalAI process. This removes every env access from the backend and routes config through the model-config seam every other backend uses. - config.go: loadConfig(opts *pb.ModelOptions) now parses the model config `options:` list (key:value strings, split on the first ':') instead of os.Getenv. Option keys mirror the old VALKEY_* names without the prefix (addr, index_algo, distance_metric, ...). Defaults, fail-fast validation and the mandatory client name are unchanged. - store.go: Load threads opts into loadConfig; TLS comments/errors renamed off the VALKEY_* names. - core/backend/stores.go: StoreBackend and NewVectorStore take a *config.ModelConfigLoader, resolve the per-store ModelConfig by store name, and pass its Options (and Backend when unset) to the backend via WithLoadGRPCLoadModelOpts. No config -> default backend + built-in defaults, preserving the zero-config experience. - Endpoints/routes/application: thread the config loader to StoreBackend. - Unit + integration tests: configure via options; the integration test passes addr through the model-config path (VALKEY_ADDR is now only the test harness locating the server). - docs + .env: document the model-config options, drop the env var table. Assisted-by: Kiro:claude-opus-4.8 Signed-off-by: Daria Korenieva <daric2612@gmail.com> * Remove valkey-store informational comment from .env The backend is configured via model config, not env vars — the comment was unnecessary noise in .env. The configuration is already documented in docs/content/features/stores.md. Signed-off-by: Daria Korenieva <daric2612@gmail.com> * feat(valkey-store): gate Load on NamespacePrefix to refuse autoload probing Mirror local-store's pattern: reject model names without store.NamespacePrefix so the model loader's greedy autoload probe cannot bind an arbitrary model name to the vector store backend (the #9287 failure mode). Also adds unit tests for the gate covering: prefixed namespace, prefix alone, unprefixed model name, empty model, and nil opts. Signed-off-by: Daria Korenieva <daric2612@gmail.com> * feat(valkey-store): add username_env/password_env credential indirection Add support for resolving Valkey credentials from environment variables named in the model config, mirroring cloud-proxy's api_key_env pattern. This keeps secrets out of model YAML files and lets distinct store configs each reference their own credentials. Options: username_env / password_env name the env var holding the value. The direct username / password options still work and take precedence when both are set (backward compatible). Includes 5 unit tests and updated stores.md documentation. Signed-off-by: Daria Korenieva <daric2612@gmail.com> * fix: correct rebase artifacts in backend-matrix.yml and Makefile Fix two issues introduced by the conflict-resolution script during the rebase onto master: 1. .github/backend-matrix.yml: valkey-store entries were merged INTO the cloud-proxy entries (duplicate keys in same YAML map items) instead of being separate list items. This broke cloud-proxy Linux builds and the cloud-proxy darwin entry lost its build-type/lang. Fixed by making them standalone entries and restoring cloud-proxy exactly as on master. 2. Makefile: duplicated .NOTPARALLEL and docker-build-backends lines. Collapsed to single lines that are master's current content plus the valkey-store additions. Also adds the three optional pickups from #10801: - /valkey-store in .gitignore (the built binary) - valkey-store row in docs/content/reference/compatibility-table.md - valkey-store line in backend/README.md Signed-off-by: Daria Korenieva <daric2612@gmail.com> --------- Signed-off-by: Daria Korenieva <daric2612@gmail.com> Co-authored-by: Daria Korenieva <daric2612@gmail.com> |
||
|
walcz-de
|
2f33d6dee0 |
docs(gpu): add ROCm 7.x and RDNA 3.5 / Strix Halo (gfx1151) to GPU acceleration guide (#9229)
* docs(gpu): add gfx1151 / ROCm 7.x and fix ROCm section - Fix typo: "deditated" → "dedicated", "ROCm6" → "ROCm" - Add ROCm 7.x to requirements (alongside ROCm 6.x) - Add Ubuntu 24.04 to tested OS list - Add AMD Strix Halo / gfx1151 section with kernel params, required env vars (HSA_OVERRIDE_GFX_VERSION, ROCBLAS_USE_HIPBLASLT), and Docker Compose example - Add gfx1151 to the list of compiled GPU targets - Add ROCm version column to verified devices table - Add gfx1151 / Radeon 8060S (ROCm 7.11.0) as verified device * fix(docs/gpu): correct gfx1151 section — env vars, image tag, safety warning - Add all 4 required env vars (HSA_OVERRIDE_GFX_VERSION, ROCBLAS_USE_HIPBLASLT, HSA_XNACK=1, HSA_ENABLE_SDMA=0) with descriptions in a table - Fix Docker Compose example to use the ROCm 7.x image tag (-gpu-hipblas-rocm7), not the ROCm 6.x image - Add explicit warning: GGML_CUDA_ENABLE_UNIFIED_MEMORY must NOT be set (even =0 activates hipMallocManaged due to getenv != nullptr check) - Add --force-recreate note (docker restart does not update container env) - Add tested hardware note (Geekom A9 Mega / Ryzen AI MAX+ 395) * docs(gpu): single ROCm image — drop -rocm7 tag suffix Per maintainer feedback on PR #9229: there is only one ROCm/hipblas main image, and it ships with ROCm 7.x by default — no separate -rocm7 tag. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
localai-org-maint-bot
|
ecdb32193d |
docs(proxy): cover long inference timeouts (#11065)
Document the reverse-proxy settings needed for long-running and multimodal requests, and distinguish edge-generated 504 responses from the optional LocalAI busy watchdog. Assisted-by: Codex:gpt-5 [Codex] Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Richard Palethorpe
|
9058a2bb46 |
feat: Add 3d generation UI/API and trellis2cpp backend (#10979)
* feat(3d): add Generate3D RPC, FLAG_3D capability, and /v1/3d/generations endpoint Adds the plumbing for image-conditioned 3D asset generation (binary glTF / GLB output), modeled on the video generation path: - backend.proto: Generate3D RPC + Generate3DRequest (staged image src, glb dst, seed/step/cfg_scale/texture_steps, quality and background enums, params map for backend-specific extras) - pkg/grpc: thread Generate3D through client, server, embed, base and the backend interfaces; connection-evicting and distributed-node wrappers (in-flight tracking + file staging) included - core/config: FLAG_3D usecase (guessed only for the trellis2cpp backend), '3d' canonical usecase string mapped to the Generate3D method, and a '3d' output modality - REST: POST /v1/3d/generations (+ unversioned alias) returning OpenAIResponse with a /generated-3d URL or b64_json; conditioning image accepted as URL, base64, or data URI; quality/background validated at the edge; .glb served as model/gltf-binary - auth: '3d' route feature (default ON); /api/instructions entry Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(trellis2cpp): add the trellis2.cpp image-to-3D backend Wraps localai-org/trellis2cpp (C++/GGML port of Microsoft TRELLIS.2, pbr-textures branch) as a Go+purego backend, following the stablediffusion-ggml pattern: - backend/go/trellis2cpp: purego bindings to the flat C ABI (v9, asserted at startup), eager pipeline load with model-set validation (refuses non-trellis GGUFs; degrades coarse/geometry-only/textured exactly like the upstream demo), Generate3D via t2_generate + t2_bake_glb writing a binary glTF to dst. Weight-free unit tests cover resolution/validation/param mapping — CI never downloads the multi-GB GGUF set or runs inference. - CPU SIMD variants build into per-variant directories (the shared libggml sonames collide across variants, unlike sd-ggml's flat renamed-.so scheme); run.sh picks one via /proc/cpuinfo. - CI wiring: backend-matrix entries (cpu, cuda12/13, vulkan amd64+arm64, l4t, l4t-cuda13, darwin metal), index.yaml meta + latest/master image entries, bump_deps tracking of the pbr-textures branch, changed-backends.js mapping, top-level Makefile targets. - Importer: auto-detects trellis GGUF repos/URIs (registered before llama-cpp so the .gguf match isn't stolen) and expands any trellis URI to the full 10-file component set spanning the three LocalAI-io HF repos. - Gallery: trellis2-4b (full PBR + 1024 cascade) and trellis2-4b-geometry (512 untextured) with verified sha256s. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(ui): 3D generation page with native GLB viewer and IndexedDB history Adds a Studio tab + /app/3d page for the new image-to-3D endpoint: - GlbViewer ports the trellis2cpp demo's dependency-free WebGL2 renderer (quaternion trackball, metallic-roughness PBR, ACES, hidden-line wireframe with a bounded index budget) and pairs it with a minimal GLB parser for the two forms t2_bake_glb emits — dense vertex-PBR (linear COLOR_0 + _METALLIC_ROUGHNESS, uploaded as normalized integers) and the opt-in UV-atlas textured form. Parsing happens before any GL so stats and errors render without WebGL2. - use3DHistory stores past generations (params, input thumbnail, and the GLB blob itself) in IndexedDB with keep-newest-20 eviction — GLBs are multi-MB binaries localStorage can't hold — and the page offers a download button for the active GLB. - Wiring: CAP_3D capability constant (FLAG_3D — the exact string /api/models/capabilities serves), threeDApi, router entries, Studio tab, vite dev proxy, en locale keys. - e2e: render-smoke entry plus a focused spec that feeds a real one-triangle vertex-PBR GLB through the parser/viewer and exercises IndexedDB persistence, selection, deletion, and API errors. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(3d): address API correctness and UX issues Keep 3D generation on the LocalAI-specific /3d/generations route and ensure authentication and permissions cover it. Propagate distributed transfer failures, publish a portable ARM64 backend image, honor importer overrides, and align discovery, upload validation, and touch controls. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(3d): add previewable print remeshing Add a single-detail CGAL Alpha Wrap workflow for existing Trellis GLBs, including PBR reprojection, API documentation, tracing, and an in-browser preview before download. Allow the remesh route to enforce its 512 MiB upload cap independently of the smaller global default so generated high-resolution meshes can be processed. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * build(trellis2cpp): centralize remesh dependency pins Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(kokoros): implement Generate3D stub for new proto RPC The Generate3D RPC added to backend.proto for the trellis2cpp backend made tonic's generated Backend trait require generate3_d, breaking the kokoros-grpc build. Return unimplemented like the other unsupported modalities. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com> |
||
|
mudler's LocalAI [bot]
|
8089b2bf09 |
fix(ci): only rebuild the full backend matrix on breaking backend.proto edits (#11192)
backend/backend.proto is consumed by every language, so its SHARED_BUILD_INPUTS rule could only ever be always/always: 417 Linux plus 56 Darwin builds. It fires on ~1.3% of commits (10 of 767 over six months), which made it the single largest CI cost driver in the repo. On 2026-07-29 the queue reached 2178 jobs against 8 concurrent runners. Four runs totalling 935 of those jobs were triggered by nothing but a proto edit. The largest, 378 jobs on master, came from PR #11158, whose entire proto diff was six lines adding `bool cache_prompt = 8;` to one message. No backend that does not read that field behaves any differently for it. Make the rule content-aware. changed-backends.js resolves backend.proto at the base revision (the contents-API pattern already used for backend-matrix.yml) and hands both texts to protoChangeIsAdditive(), which compares them structurally so a comment reflow, reindent or field reorder does not read as a change. An additive-only edit (new field with an unused number, new message, new enum value, new RPC) suppresses the rule and rebuilds nothing; a removed, renumbered, retyped or renamed field, a dropped RPC or a changed option still rebuilds everything, as does an unresolvable base revision. Every other matched rule is untouched, so a PR that edits the proto and scripts/build/ is still a full rebuild, and the weekly full-matrix cron remains the backstop for stale wheels. Verified against all ten proto commits of the preceding six months: the nine with a resolvable parent all classify as additive, and controls covering a retyped-and-renumbered field, a deleted RPC, identical revisions and a reindent-plus-comment-reflow all classify correctly. Assisted-by: Claude:opus-5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-bot
|
89ee62b2af |
gallery: add KAT-Coder V2.5 Dev GGUF variants (#11186)
* gallery: add KAT-Coder V2.5 Dev GGUF variants Add Q4_K_M and Q8_0 builds of the newly released KAT-Coder-V2.5-Dev agentic coding model. Assisted-by: Codex:gpt-5 [Hugging Face API] * gallery: add KAT-Coder APEX variants Assisted-by: Codex:gpt-5 [web] * gallery: add KAT-Coder APEX checksums Assisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Richard Palethorpe
|
49ef40a187 |
feat(classifier/VAD): support voice control on low power devices (#10804)
* feat(llama-cpp): route Score through the slot loop Score previously bypassed the slot loop with a direct llama_decode: a conflict guard aborted the whole process if scoring raced generation, the config validator had to reject score alongside chat/completion/embeddings, and every candidate re-decoded the full shared prompt. Add SERVER_TASK_TYPE_SCORE to the (patched) upstream server so score tasks are scheduled like any other slot work: generation and scoring serialize naturally, the shared prompt is decoded once per call, and the slot's prompt cache carries the conversation prefix across calls. Context checkpoints at the score boundary and at the cache-divergence point keep SWA/hybrid/recurrent models (e.g. LFM2.5) from re-prefilling the whole prompt per candidate: warm-turn scoring on a 6-option set drops from ~8s to ~0.5s on a desktop CPU. The conflict guard and the validation split are removed; declaring score with generation usecases on one config is now supported and shares the slot cache. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): classifier wire types and pipeline config Wire types and YAML config for realtime classifier mode: sessions carry a localai_classifier extension (options with canned replies/tool calls, softmax threshold, normalization, history trimming, fallback modes, and a deterministic wake-word address gate), mirrored by pipeline.classifier in the model YAML and surfaced in the config-meta registry. The localai.classifier.result server event reports the full score distribution per turn. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): classifier response flow Classifier-mode responses: instead of autoregressive generation, each user turn is prefill-scored against the option list (router.ScoreClassifier prompt/candidate shapes over the Score primitive) and the winning option's canned reply and tool call are emitted through the existing response machinery. Below-threshold turns take the configured fallback (none / canned reply / generate); empty transcripts and unaddressed turns (wake word not mentioned) skip scoring entirely. The scoring probe defaults to the latest user message only — small scorers echo canned replies from prior turns back as the top option otherwise. Built for hardware that can afford prompt processing but not decode: with slot-based Score the option list stays KV-cached across turns, so a turn costs roughly one forward pass over the new words. session_update_error events now carry the validation cause instead of a generic message. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): bound the VAD tick's scan window and buffer retention The VAD tick loop re-scanned the entire input buffer every 300ms and only trimmed it on zero-segment ticks or commits. Audio that keeps producing segments without a committing pause (steady noise a mic pipeline lets through, music, continuous speech) grew the buffer toward the 100MB cap with each tick rescanning all of it — O(n^2), measured at ~3.3ms of silero per buffered second: past ~90s retained, ticks run back to back and pin ~4 cores until the stream stops. Silero's recurrent state only carries a few hundred ms of context, so rescanning old audio buys nothing. Clip the slice handed to the VAD to the largest silence the commit test can need to measure (server_vad silence window or the semantic eagerness fallback) plus a warm-up margin, and rebase the returned segment times so every downstream consumer keeps whole-buffer coordinates. An open turn whose clipped window is all silence now commits (the silence outran the window) instead of being discarded as no-speech. Independently, retain at most 90s of raw buffer, rebasing the live-feed and EOU cursors on trim — this also bounds the previously unbounded VAD-error path. Turn boundaries are otherwise unchanged: no forced commits, no new coordinator states. pipeline.turn_detection.vad_window_sec can widen the scan window; values below the automatic floor are ignored. The tick body is extracted into vadTick so specs can drive turn detection synchronously (same shape as classifySoundWindow); the babble reproduction that pinned 4 cores now plateaus under 10% of one core. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(backend): let per-model threads override the global default ModelOptions overrode a set per-model threads value with the app-level --threads whenever the latter was non-zero — and WithThreads defaults it to the physical core count, so it always was. The YAML threads: knob has been dead config: a tiny VAD model could never opt down from the global pool size. SetDefaults already fills an unset per-model value from the app config, which is the intended precedence; resolve threads through a helper that honors it (explicit threads: 0 still means unset). Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * chore(gallery): single-thread the silero VAD Silero is a ~2MB recurrent model with no exploitable graph parallelism: measured per-call latency is identical at 1 and 10 ORT threads, while every extra pool thread just spin-waits between the realtime loop's frequent tiny inferences. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * docs(realtime): classifier mode, VAD scan window, threads precedence Document the realtime classifier mode (options, threshold guidance, wake-word address gate, empty-transcript handling), the VAD scan window and 90s buffer retention (pipeline.turn_detection.vad_window_sec), the per-model threads precedence, and the M3 classifier note in the realtime state-machine design doc. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * perf(llama-cpp): score all candidates in one batched decode One scoring call is now a single SERVER_TASK_TYPE_SCORE task: the slot decodes the shared prefix (prompt + longest common candidate token prefix) once, then forks one sequence per candidate off it (metadata-only for the unified KV cache, copy-on-write for recurrent state) and decodes every candidate's unique tail in one llama_decode. Previously each candidate was its own task that restored the boundary checkpoint and re-decoded its full tail sequentially, paying per-candidate task and decode overhead. The context reserves SERVER_SCORE_FORK_SEQS extra sequence ids (and recurrent-state cells) beyond the parallel slots via the new common_params::n_seq_score_forks. Forking requires the unified KV cache (already this backend's default) since per-sequence streams would shrink n_ctx_seq; an explicit kv_unified:false disables forking and Score calls that need it fail cleanly. Candidates beyond the fork/output budget decode in successive chunks. Wire contract and scores are unchanged: per-token logprobs are stitched from the shared region and the forked tails. Verified bitwise deterministic call-to-call and independent of candidate order (no cross-fork leakage via equal-length candidate swap); ranking matches the per-candidate implementation on the drone battery (winner softmax 0.99996 vs 0.99997), and >16-candidate chunking, prefix-of-another and empty candidates all pass. Measured on a desktop CPU: warm /api/score calls 0.52s -> 0.23s; warm realtime classifier turns 196-303ms. The 9-candidate drone turn decodes ~17 unique tail tokens in one batch instead of nine sequential ~220ms checkpoint-restore tasks. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): gate scoring capacity by model usecase Reserve llama.cpp scoring slots only for models that explicitly declare the score usecase, while allowing score to coexist with chat and completion. Reject incompatible unified-KV settings and classifier activation on models without scoring capacity. Propagate application defaults when resolving realtime and preload pipeline stages so unset thread counts are resolved consistently without overriding explicit model settings. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): honor APT mirrors in the prebuilt llama-cpp compile step The builder-prebuilt path installs gcc-14 with apt directly and ignored the APT_MIRROR/APT_PORTS_MIRROR build args the from-source path already honors, so an ubuntu mirror outage broke every arm64 backend build. Pass the args into the stage and run apt-mirror.sh (already in the build context via COPY . /LocalAI) before the apt step. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): classifier argument slots via constrained completion Hybrid classify-then-complete: a classifier option's canned tool call can declare typed argument slots (number | enum | string, with defaults and prompt hints) referenced as "{{name}}" in the arguments template. When the option wins, the slots are filled by a short grammar-constrained completion that continues the exact scoring prompt — rendered by the same cached ScoreClassifier, so the llama.cpp prompt cache is already warm — with the chosen route JSON re-opened at the first slot field. A GBNF grammar pins the field skeleton and frees only the values; temperature 0, a couple dozen tokens at most (~300ms on a desktop CPU for two slots). Slot declarations and hints ride the option descriptions in the shared system prompt, informing scoring and the fill alike at no per-turn token cost. The localai.classifier.result event carries the final arguments and a fill_latency_ms. On inference failure the slots' defaults apply; a slot without a default fails the response (or falls through with fallback.mode: generate). Slot filling requires completion alongside score in the scoring model's known_usecases. Verified end-to-end on the Pi drone demo: "fly forward three meters" in distance mode classifies forward and infers {"distance": 3, "units": "meters"} in ~310ms, and the drone flies exactly 3 units. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): splice filled slot values into classifier replies A classifier option's spoken reply can now reference its tool's argument slots ("Going forward {{distance}} {{units}}."): the values inferred by the slot-fill completion — or the recovery defaults — are spliced into the reply as plain text before it is emitted, so what the assistant says confirms what it actually inferred. Placeholders without a value stay literal, and options without slots are untouched. FillToolArguments now returns the raw slot values alongside the spliced arguments JSON to make the reply templating possible. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): harden classifier slot completion Reserve context for constrained slot filling, size completions from their encoded output, and encode enum grammar literals as valid JSON. Reject empty enum values and cover the failure modes with regression tests. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): prewarm the classifier scoring prompt on registration Swapping a session's classifier option list (a voice-switched command mode, for instance) made the next turns pay a full re-prefill of the new option-list prompt — measured 2.4s vs 0.3s warm on a desktop CPU, and worse: on hybrid-memory models like LFM2.5, whose state cannot be partially rewound (llama.cpp can only restore checkpoints), *every* probe change re-prefilled from scratch whenever the last checkpoint missed the probe boundary, so even same-list turns intermittently cost full prefills. Registering an option list (pipeline seed or session.update) now fires a best-effort background prewarm: two throwaway scores with distinct probes. The first prefills the new option-list prompt; the second, diverging exactly where per-turn probe text starts, plants the backend's rewind point (KV checkpoint) at the stable-prefix boundary that every real turn reuses. The prewarm hides behind the canned mode-switch reply — by the time it finishes speaking, the cache is warm. Idempotent per option set, detached from the registering request's lifetime. Measured on the drone demo (LFM2.5-1.2B, desktop CPU): first turn after a mode switch 2374ms -> 340ms; intermittent same-list full prefills (1.3-2.1s) all -> under 0.5s. For clients that swap lists frequently, options: [parallel:2] on the scoring model additionally keeps one slot per list via prefix-similarity routing (+26MB RSS, unified KV). Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * perf(llama-cpp): checkpoint scoring at the caller-declared stable prefix Hybrid-memory models (LFM2.5 shortconv, Qwen3.5 deltanet — where new small models are headed) cannot rewind their state, so any prompt-cache reuse that needs a rewind falls back to a full re-prefill. For classifier scoring that meant every probe change re-processed the whole option-list prompt: the server's checkpoints were placed reactively (at wherever the previous task happened to diverge), so a checkpoint past the next divergence was erased rather than restored — measured as intermittent 2-10s turns on prompts with a 95%+ common prefix. The classifier now computes the probe-invariant prompt prefix once (the byte-wise common prefix of two synthetic probe renders) and declares its length with every Score request; the server maps it to a token boundary and forces a KV checkpoint exactly there on each score prefill. That checkpoint sits at or before every future divergence under the same option list, so it always survives and always restores — repeat scoring costs probe+candidates regardless of how the probe changes. Also: - prewarm reruns on every option-list registration instead of memoizing per list: with boundary checkpoints a redundant rewarm costs two probe-sized decodes, while skipping one after a slot eviction (three lists sharing fewer slots evict in LRU cascades) silently moves a full re-prefill onto the user's next turn - new llama.cpp backend option rs_seq:N exposes bounded recurrent-state rollback outside speculative decoding; measured impractical for deltanet-scale states (65GB for 64 snapshots on Qwen3.5-4B) but cheap insurance for small-state models - docs: the multi-list recipe (parallel:N + sps:0.5 — the default slot similarity threshold funnels distinct lists onto one slot) Measured on the drone demo (LFM2.5-1.2B scorer, desktop CPU), steady state: every turn 285-421ms including mode switches, vs 2.4s post-switch and intermittent 1.3-2.9s re-prefills before. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): align classifier cache guidance Document the single-score prewarm behavior and clean the vendored score patch formatting. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(llama-cpp): guard score task for fork backends TurboQuant and Bonsai reuse the primary gRPC server against llama.cpp forks that do not carry LocalAI's slot-based Score patches. Compile the Score integration only for the patched primary backend and return UNIMPLEMENTED from fork builds instead of referencing absent task types and common_params fields. Assisted-by: Codex:gpt-5 [gh] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(dev): generate gRPC code before commit lint The coverage phase regenerates ignored protobuf bindings, but lint runs first and can fail against missing or stale output. Generate the pinned bindings before lint so the gate always type-checks the current schema. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
localai-org-maint-bot
|
bc21f832aa |
gallery: add Laguna S 2.1 GGUF variants (#11188)
Add the official Q4_K_M and Q8_0 builds plus the DFlash speculative-decoding pairing for llama.cpp. Assisted-by: Codex:gpt-5 [Hugging Face API] Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]
|
967becb365 |
chore: ⬆️ Update ikawrakow/ik_llama.cpp to b054a8b983827c01aec59d4dc273a27c492c51c4 (#11175)
⬆️ Update ikawrakow/ik_llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com> |
||
|
mudler's LocalAI [bot]
|
0569bb30a2 |
chore: ⬆️ Update ggml-org/whisper.cpp to 97c56f1dc1d1100a9d859c865a20c82d22f823ed (#11182)
⬆️ Update ggml-org/whisper.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com> |
||
|
mudler's LocalAI [bot]
|
550545c03a |
chore: ⬆️ Update mudler/parakeet.cpp to e747acdaee69b916cef62263ae5f718bda9ff3f3 (#11181)
⬆️ Update mudler/parakeet.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com> |
||
|
mudler's LocalAI [bot]
|
c5e5141010 |
fix(ci): unbreak the sglang and darwin nemo backend builds (#11168)
* fix(sglang): keep nvidia-modelopt on a stable release Every cublas sglang image currently fails to build: Failed to build `nvidia-modelopt==0.46.0rc0` Call to `wheel_stub.buildapi.build_wheel` failed ModuleNotFoundError: No module named 'wheel_stub' sglang[all] pulls nvidia-modelopt in through its `diffusion` extra with no version bound of its own, and install.sh adds a GLOBAL --prerelease=allow so that flash-attn-4, which only ships 4.0.0b* wheels, can resolve. Unbounded plus prereleases-allowed picks 0.46.0rc0, whose build backend imports wheel_stub without declaring it in build-system.requires. EXTRA_PIP_INSTALL_FLAGS also starts with --no-build-isolation, so nothing installs wheel_stub and the build dies. Latest stable is 0.45.0 and resolves cleanly. Bounding this one package rather than dropping the global flag, because the flag is load-bearing for flash-attn-4 and this is the narrower change with the smaller blast radius. Raise the bound when 0.46.0 final ships. This is invisible on master because the backend build is path-filtered: sglang is only rebuilt when sglang changes. It surfaces on any PR touching a shared build input such as backend/backend.proto, which rebuilds the whole matrix. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(nemo): build the darwin venv on Python 3.12 The darwin nemo image fails to build: ModuleNotFoundError: No module named 'maturin' nemo_toolkit pulls in text2num, a Rust extension built with maturin, whose macOS arm64 wheels start at cp311: 3.0.2 publishes cp311, cp312, cp313 and cp314 and no cp310. libbackend.sh defaults PYTHON_VERSION to 3.10, so pip finds no wheel, falls back to the sdist, and dies in the PEP 517 hook because EXTRA_PIP_INSTALL_FLAGS carries --no-build-isolation and nothing installs the build backend. Taking the prebuilt wheel avoids the source build entirely, so the runner needs no Rust toolchain. Darwin only, deliberately: the Linux profiles resolve a cp310 manylinux wheel for the same package and have no reason to move. The override is set after libbackend.sh is sourced and before installRequirements, the same shape sglang's install.sh already uses for its l4t13 profile. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(nemo): pin the darwin portable-Python patch level too The 3.12 bump alone traded one failure for another: curl: (56) The requested URL returned error: 404 make[1]: *** [nemo-asr] Error 56 libbackend builds the portable-Python URL from cpython-${PYTHON_VERSION}.${PYTHON_PATCH}+${PY_STANDALONE_TAG}, and PYTHON_PATCH defaults to 18 because the default interpreter is 3.10.18. Setting only PYTHON_VERSION asked for a 3.12.18 that was never released. Patch 11, not the 12 that sglang/install.sh pairs with 3.12 for l4t13: at the 20250818 tag python-build-standalone published 3.12.12 for linux aarch64 but not for aarch64-apple-darwin, where 3.12.11 is the newest. Both URLs were checked against the release assets rather than assumed to match. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com> |
||
|
mudler's LocalAI [bot]
|
47c48e9409 |
chore: ⬆️ Update antirez/ds4 to 54b36ed9ba42da31b24f2d1a5feb075c2475dbb1 (#11178)
⬆️ Update antirez/ds4 Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com> |
||
|
mudler's LocalAI [bot]
|
6aaf7db5f0 |
chore: ⬆️ Update mudler/depth-anything.cpp to 2028b47ac75a8659c6a9aa617baf09be193eb55f (#11179)
⬆️ Update mudler/depth-anything.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com> |
||
|
mudler's LocalAI [bot]
|
193d49001b |
chore: ⬆️ Update CrispStrobe/CrispASR to 754b67289cf1137e3ed722885705f94132fc614f (#11180)
⬆️ Update CrispStrobe/CrispASR Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com> |
||
|
localai-org-maint-bot
|
8f9184fbb2 |
feat(cli): support systemd socket activation (#11169)
* feat(cli): support systemd socket activation Serve the API from a single stream listener inherited through the systemd activation protocol while retaining the existing address bind path when no listener is provided. Validate activation metadata, preserve the public-bind safety check, and document an on-demand systemd setup. Assisted-by: Codex:gpt-5 * fix(cli): satisfy listener cleanup lint Make the best-effort close explicit so errcheck accepts the deferred systemd listener cleanup. Assisted-by: Codex:gpt-5 [golangci-lint] --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]
|
84972cb745 |
chore(model-gallery): ⬆️ update checksum (#11176)
⬆️ Checksum updates in gallery/index.yaml Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
localai-org-maint-bot
|
034df6ceb1 |
fix(worker): report RAM alongside GPU memory (#11167)
* fix(worker): report RAM alongside GPU memory Assisted-by: Codex:gpt-5 * feat(ui): show worker RAM on node views Assisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
localai-org-maint-bot
|
996bdcecdc |
fix(mlx-vlm): install torch dependencies on Metal (#11164)
Some Transformers processors used by MLX-VLM, including Qwen vision models, import both PyTorch and Torchvision. Include them in the Metal backend environment so model loading does not fail with missing-library errors. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
walcz-de
|
4b4faa4ac7 |
feat(cloud-proxy): optional Anthropic prompt-cache breakpoints in translate mode (#11158)
The Anthropic translate provider builds the upstream request from scratch and
never emitted cache_control, so prompt caching was impossible for OpenAI-format
clients routed through cloud-proxy — even though the entire system prompt + tools
prefix is re-sent on every agentic turn.
Add an opt-in cache_prompt flag (ProxyOptions.cache_prompt; model YAML
proxy.cache_prompt: true). On a translate+anthropic model, buildAnthropicRequest
injects cache_control:{type:ephemeral} on the stable prefix — the system block,
the last tool, and the last message block (at most 3 of Anthropic's 4 allowed
breakpoints). Anthropic then serves the repeated prefix at the cache-read rate
(0.1x input) on subsequent calls, cutting cost on multi-turn/agentic workloads.
No effect in passthrough mode, for non-Anthropic providers, or when unset.
System is widened to any so it can carry the block form required to attach
cache_control, while still marshalling as a bare string when caching is off.
Adds a unit test asserting exactly three breakpoints when on and none when off,
and documents the option in docs/content/operations/cloud-proxy.md.
Assisted-by: Claude:opus-4.8
Signed-off-by: stefanwalcz <stefan.walcz@walcz.de>
|
||
|
mudler's LocalAI [bot]
|
0f7186f214 |
feat(ui): replace the stacked operations bar with a one-line strip and an Activity page (#11163)
* feat(ui): record finished gallery operations in a bounded history ring The operations panel drops an operation the moment it succeeds, so a user who steps away cannot tell whether an install finished, failed or was never started. OpCache now keeps the last 50 terminal operations, recorded from the point where an op leaves the cache. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): pin the history ring's dedupe, outcome order and start stamp Review of the history ring found four gaps. The dedupe guard and the bounded seen set were unreachable through the exported API and so had no coverage; an in-package spec file now drives opHistory directly. The outcome switch claimed an ordering was load bearing that nothing pinned, so an errored op that never reached Processed now has a spec. Two behaviour fixes come with it. StartedAt was the zero time for ops recovered from the store or replicated from a peer, since neither path stamps a start time, which would have rendered as a two-millennia duration; it now falls back to the finish time. Reusing a cache key with a fresh job ID orphaned the previous stamp, so Set and SetBackend now drop it. The comment on the outcome switch described a state the code cannot be in: CancelOperation sets Cancelled and Processed synchronously before the handler removes the entry, so status.Cancelled already covers the cancel endpoint. The !Processed clause stays for the dismiss endpoint firing on an in-flight op, and the comments now say so. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): record operations that end on a peer replica The NATS end event is the only signal a replica gets for an install another replica ran. Record from applyEnd too, deduped by job ID so the originating replica does not record its own broadcast twice. Three start-stamp defects in the same path go with it. applyEnd now drops the stamp unconditionally, since recordTerminal only cleans up on the path where it found a cache key and an end event can overtake the local Set. applyStart drops the stamp of the job whose cache key it replaces, which a peer-driven retry previously stranded. And recordTerminal reads the stamp once instead of testing Exists and then reading, so a concurrent record for the same job can no longer delete the stamp between the two and let the zero time overwrite the finish-time fallback, which the Activity page would render as a two-millennia run. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): do not guess the outcome of a peer operation with no local status A replica that restarts mid-operation hydrates its OpCache keys from PostgreSQL, but gallery statuses are in-memory only and come back empty. The end broadcast then landed on recordTerminal's nil-status branch, which reads a missing status as queued-and-removed and filed a successful install as cancelled. That reading is right locally and wrong on the peer path, where a missing status means the outcome was never held here. recordTerminal now takes the source of the terminal event and records nothing when the peer path finds no status, restoring what the replica did before the end event started recording. The local path is unchanged. Also move the ApplyEndForTest seam to the conventional export_test.go, and stop the dedupe spec from claiming to guard the ring's seen set: the local delete removes the status keys, so the broadcast that follows returns before reaching it. An in-package spec that calls recordTerminal twice does the pinning. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(api): add GET and DELETE /api/operations/history Admin gated like the rest of the operations API. The live /api/operations payload is unchanged so the one second poll stays small. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): expose operation history through OperationsContext Fetched on demand and when the live list shrinks, never on the one second poll interval. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): detect operation departure by identity and ignore committing ops in the ETA gate Refetching history on a shrinking live count missed a completion that coincided with a start, which is the common case during a batch install. Track the live job IDs instead, so any departure triggers the refetch regardless of how the count moved. An operation that has finished downloading stays live at currentBytes == totalBytes for the whole commit and install phase and can never produce an estimate, so counting it in the all-or-nothing gate blanked every other operation's time remaining for as long as it lasted. Only operations still moving bytes get a vote. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): let only downloading operations gate the time remaining estimate Verifying pins an operation flat below its total for the whole sha256 pass: the AfterDownload hook reports completedBytes plus the finished file against a total summed over every file, then hashes synchronously without emitting progress. Files download sequentially, so a 15 shard model enters that window 14 times, and a byte comparison cannot see it because the counter is genuinely below the total throughout. Gating on phase closes resolving, verifying, committing and persisting in one predicate, so a quiet neighbour no longer blanks every other operation's estimate for minutes at a time. The byte clauses stay: a producer can report downloading with bytes already at the total. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): collapse the operations bar to a single line Four concurrent installs used to take four rows above every page. The strip now shows one operation, failure first, with a counter linking to Activity. The close button hides the strip and no longer cancels an install. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): keep the operations strip from widening the page and from muting a failure A long install error made the strip report a 1600px minimum width, which sized main-content to fit and gave every page under it a horizontal scrollbar. Inline-size containment plus shrinkable detail and bytes cells keep it inside the viewport. Hiding is no longer able to swallow the hidden job's own failure, a completed removal or staging says so instead of claiming an install, a cancelling operation renders as cancelling, and the live region no longer covers the per-second percentage. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): shrink main-content instead of containing the strip, and expose progress min-width on .main-content is what actually lets a long install error shrink, and unlike inline-size containment it has no browser support floor and no latent collapse if the strip ever lands in a shrink-to-fit context. It matches what .app-layout-chat .main-content already does, and it clears pre-existing horizontal overflow on narrow viewports as a side effect. The progress track is now a labelled progressbar, so assistive tech can read the value on demand rather than losing it to the aria-hidden that stopped the live region re-announcing every poll. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): add the live operation card for the Activity page Carries the detail the one-line strip has to drop: phase, bytes, the per-node breakdown for cluster installs, and a labelled Cancel button. Cancelling is destructive, so it gets a labelled button rather than a glyph. A cancelling operation drops its progress bar and its time estimate, the same call the strip makes: a percentage still climbing under "Cancelling" reads as the cancel not having taken. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): give the operation card a verb, live node disclosure and its per-node detail The card carried no verb, so an install, a removal and a staging op rendered as spinner plus name plus kind tag and were indistinguishable. It now runs the same verb and icon chain as the one-line strip, which is what stops the page that is meant to carry more detail from carrying less. The auto-expand default was evaluated once at mount. An operation is listed as soon as it is admitted but its nodes are filled in only when the fan-out starts reporting, so a card mounted at creation latched on the empty list and stayed collapsed. The default is a live expression now, and state holds only an explicit choice. Also: an optional onRetry gates a Retry button, so the page can own the install reconstruction without the card ever showing a control with nothing behind it; the disclosure moved above the region it controls and gained aria-controls; the toggle is gated at more than one node so the count is never "1 nodes"; an unmapped node status is passed through instead of being relabelled "Queued"; error text is clamped with the full string in the title; and file_name plus the per-node progress bar are rendered again, reviving three CSS rules that had gone dead along with the detail they styled. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): add the Activity page Live operations, unacknowledged failures and the record of what finished, at /app/activity in the Operate console. Cancelling an install now lives here behind a labelled button rather than on the strip, and a failed install can be retried: the retry dismisses the failure first so it still reaches the record, then reissues the model, backend or node-scoped backend install. The sidebar Operate entry carries the operation count. The console rail is only rendered on an Operate route and can be collapsed, so a badge there could vanish while operations were still running. Two follow-ups from review fold in here: a failed removal or staging job no longer reports a failed install on either the card or the strip, and the card's error text can shrink so one unbroken token cannot widen the card. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): dismiss operations by job, and stop the Activity page contradicting itself Dismissing resolved the job by display id, but /api/operations strips the "node:<nodeID>:" prefix before emitting, so a local install and a node-scoped install of one backend arrive as two jobs sharing one id. Dismissing by id retired whichever came first. That defeated the guarantee retry was built around: with the wrong job dismissed, the reinstall overwrote the acted-on failure's opcache entry in place, bypassing recordTerminal, while an unrelated failure vanished from Needs attention. dismissFailedOp, the card's dismiss control and the strip now all pass the jobID, which is what the endpoint takes. A filter matching nothing rendered the "nothing has ever run" empty state while the header counted the records the filter had hidden. The empty state is now gated on the All chip and a narrowed view gets its own message plus a way back; the header counts the instance rather than the chip, so selecting Backends no longer reports "Nothing running" over running model installs. Also: the summary drops a zero clause instead of rendering "0 needs attention" on the happy path and pluralises both counts; a record duration is floored at "< 1s" and rejected above a day, so a zero-value start stamp cannot render a span of millennia and a zero span cannot render "installed in" with nothing after it; a deletion cancelled mid-flight reports the cancellation rather than claiming it was removed; and the retry variant comment names the fix instead of calling the gap closed. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: document the Activity page and the operations history endpoints Adds an Activity page under Operations covering the one-line operations strip, the /app/activity sections and filters, per-operation cancel, retry and dismiss, the in-memory 50-entry record, and the sidebar count. Documents GET and DELETE /api/operations/history, and fills the gap in the admin-only endpoint list, which also omitted the pre-existing POST /api/operations/:jobID/dismiss. Corrects the distributed-mode install-watching section: the per-node breakdown now lives on the Activity page rather than on the strip, which rolls a fan-out up into a single phrase. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: correct nine details in the Activity page documentation The operations strip never renders a file name: its detail line is the error, the node roll-up, the target node, the phase or the queued note. Drops the stale clause in the distributed-mode section, where the per-node bullet is now the only place a file name is described. Scopes the phase vocabulary to artifact-backed gallery models, since a plain GGUF install emits no phase. Corrects the per-node list: the toggle exists for any fan-out of two or more workers and the four-node threshold only governs whether it starts open, while the N nodes tag needs more than one node. Notes that a cancelled operation can sit in the live section reading Cancelling, that cluster staging never reaches the record, and that Clear history appears only when the record has something in it. Names the operations response envelope, with a JSON example, so callers do not index a bare array, and stops describing the icon-only dismiss control as a labelled button. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: drop the unreachable Cancelling state and scope the byte claims An operation can only report isCancelled while it is unprocessed, but every writer of Cancelled sets Processed in the same breath, on the peer path as much as the local one, and the cache evicts cancelled entries before the handler sees them. The state cannot reach the page, so the live section is described again as running or queued operations. Byte counts come from the artifact bridge alone, the same producer as the phase, so a plain GGUF install, a removal and a backend install report none. Scopes both to artifact-backed gallery models and leaves the verb, the name and the percentage as what every operation shows. A worker backend install reports its bytes through fields the operations payload does not carry, so the distributed section now describes the percentage and the node roll-up, with per-file counts pointed at the per-node detail. Also: staging jobs carry no error, so they never reach Needs attention and Retry never had a staging case to exclude; an install that involves workers is no longer called node-scoped, which this page uses for node-targeted installs; and the record timestamps carry nanoseconds. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: state only the verb and the name as unconditional on the strip The percentage is as conditional as the bytes were: it renders only for a running operation that has reported progress, so a queued operation, a failed one and a removal never carry it. A removal in particular sits at progress zero for its whole visible life, since the delete path reports none and its completion is filtered out. Both the strip and the card paragraphs now lead with what always shows and list the rest as conditions. The Cluster chip matches on a node list that finished operations do not carry, so a fan-out install leaves the chip once it reaches the record. Scoped that claim to the live sections. Two more of the same shape, found by re-reading each clause alone: the strip also appears for a failure, which is not running, and the four-second hold only applies when nothing replaces the operation that just finished. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): stop reporting a cancelled install as installed, and make queued real Three defects that all trace to one root cause: `isCancelled: true` is unreachable from /api/operations. Every writer of Cancelled=true also sets Processed=true, the handler skips Processed && Cancelled, and OpCache.GetStatus evicts a cancelled op before the handler iterates it. Cancelling the last running operation put a green "Installed model X" on the strip for four seconds: the completion hold was guarded by `!previous.isCancelled`, which is dead. A cancellation deletes the operation server side, so the strip sees exactly what it sees on a completion, and nothing in the payload separates the two. The signal now comes from the side that issued the cancel: the operations context remembers the job IDs it cancelled (pruned after a minute) and the strip asks before it holds anything. A cancelled operation goes as soon as it stops; the record already reports it as cancelled. isQueued was set only when the gallery status was missing, but markQueued publishes a "queued" status at admission, so a queued op has a status for its whole queued life and the state was unreachable outside a microsecond window. Every operation waiting behind a running install rendered as "Installing model X" with a spinner. The queued phase is now the signal, via an exported PhaseQueued and a nil-safe OpStatus.IsQueued() next to the writer. With those two fixed, the Cancelling state has no way to be entered: cancelling is instantaneous from the API's point of view. Its branches, CSS, locale key and the isCancelled field itself are removed rather than left for a future reader to assume they work. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): keep a removal a removal, and say what an install is doing OpStatus.Deletion was set once, at admission, and lost on the next status write: UpdateStatus replaces the whole status and only carried Nodes forward. Every later writer (the worker's first write, the progress ticks, the failure path) leaves the field at its zero value, so the flag survived only the queued window, and both surfaces test isQueued first. The reachable consequence is that a failed removal reported itself as a failed install, which is exactly the shape the Activity page offers Retry for, and Retry installs: pressing it on a removal that failed re-downloaded the model. A running delete also rendered as "Installing model X" with a spinner, and a successful one as "Installed model X". Carry Deletion forward the way Nodes already is. A job is a delete or an install for its whole life; an unset flag means "no new information", not "this is an install". Pinned by Go specs on both the service and /api/operations: the existing Playwright specs were green only because they stubbed a payload the server could not emit. Also restore the operation's own status message on the Activity card. Phases and byte counters exist only on the managed-artifact path, so a legacy files: gallery model and every backend install rendered a sub-row with nothing in it but the verb. The strip stays terse on purpose. And give the strip's name a min-width floor: overflow: hidden zeroes its automatic minimum, so a long error squeezed the name down to "mod…" and the identity of the thing that broke was the first thing lost. primaryOperation is made module-private: its comment claimed the Activity page selected the same operation, but that page shows all of them, partitioned into failed and running, and never imported it. Assisted-by: Claude Code:Opus 5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(activity): read the operations record from PostgreSQL The Activity page's record of finished installs and removals was a 50-entry in-memory ring per frontend replica. In distributed mode that is the wrong place for it: each replica keeps its own copy, a replica added by a scale-out or a rolling deploy starts empty and never backfills, and "Clear history" clears only the replica that served the request, so the record reappears on the next poll routed elsewhere. The data is already in gallery_operations. Read it from there. GalleryStore gains ListTerminal and ClearTerminal, sharing a lifted terminalStatuses set with CleanOld so there is one definition of "finished". ListTerminal orders by updated_at, when the operation reached its terminal status, because the record reports what finished and when. OpCache.History and ClearHistory dispatch on whether a store is wired, so the HTTP handlers and the OpRecord JSON shape are unchanged and the page needed no change. A failed store read falls back to the local ring rather than blanking the page, and ClearHistory empties the ring as well so a database blip cannot resurrect a record the admin just cleared. The name derivation in recordTerminal is lifted into operationDisplayName and used by both paths, so the ring and the store cannot name the same operation differently. Also fixes a pre-existing bug the store path made visible: the backend channel hardcoded op_type "backend_install" even for a removal, while the model channel derives model_install/model_delete from op.Delete. Both channels carry the same ManagementOp, whose Delete field the backend handler already branches on, so the backend channel now derives backend_delete the same way. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(activity): keep a cancelled operation cancelled, and report a failed clear Review follow-up on the store-backed Activity record. A cancelled install was recorded as a failure. The cancel handler persists "cancelled" synchronously, then the handler goroutine unwinds with the context error and Start hands that to updateError unconditionally, which overwrote the row with "failed: context canceled". The page rendered a cancelled install as a red failure card offering Retry, with a raw context error as the reason. Fixed in GalleryStore rather than in Start, because an operation finishes once and the paths that retire one are not mutually exclusive: UpdateStatus now refuses to rewrite a row that already reached a terminal status. That also pins updated_at to when the operation really finished, which is the key the record is ordered by, and Create's upsert now freezes the same columns so a worker dequeuing an operation the admin cancelled while it was queued cannot reopen it as pending. ClearHistory returned nothing, so a failed delete logged a warning while the handler still answered 200. The admin watched the record clear and come back on the next fetch with nothing said about why. It now returns the error, the DELETE handler answers 500, and the store is cleared before the local ring so a failure leaves the fallback record intact rather than faking an empty one. Hydrate is the only reader that decides from op_type whether an operation is a removal, and it tested for "model_delete" exactly, so the backend_delete added in the previous commit hydrated as an install: a replica restarting during a backend removal rendered "Installing backend X". Both discriminations now go through IsDeleteOpType/IsBackendOpType so a fifth op_type cannot silently read as an install in whichever consumer was missed. Also: the backend channel now persists Cancellable as !op.Delete, matching the model channel; IsBackend falls back to the op_type prefix, since is_backend_op is only written by UpsertCacheKey and the rows needing the name fallback were reporting backend operations as models; and an unrecognized terminal status is logged rather than quietly filed as a success, which is what the comment already claimed. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(activity): keep a reaped operation correctable by its real outcome The terminal-status freeze added in the previous commit was too wide. It froze "failed" alongside "completed" and "cancelled", and the stale reaper writes "failed" onto operations that are still going to run. The gallery worker is a single goroutine consuming both channels serially, so an operation queued behind a large download sits in "pending" with nothing bumping updated_at, and ReapStaleOperations gives up on it after 30 minutes. That used to be self-healing: the worker dequeued it, Create reset the row to "pending", and the operation reported its real outcome. With the freeze the row stayed "failed" forever while the install ran and succeeded underneath it: a red failure card offering Retry for a model that is installed, omitted from ListActive so no replica hydrates it, and no longer deduped cluster-wide by FindDuplicate. Freeze on ("completed", "cancelled") instead. That is all the cancelled-install fix ever needed, and it leaves a failure correctable by what actually happened. The set is separate from terminalStatuses, which ListTerminal, ClearTerminal and CleanOld all still want in full, because the two mean different things: a failure can be superseded by a real outcome, a completion or a cancellation is the real outcome. UpdateStatus now writes the error column unconditionally, so a corrected outcome drops the previous attempt's reason rather than being recorded as completed while still carrying "stale operation reaped" as its error. Also adds the route-level spec for the 500 branch of DELETE /api/operations/history, and trims a comment that credited the persisted cancellable column with more than it survives long enough to do. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(activity): offer Cancel in the phase that can honour it The cancellable flag was set at both ends of an operation's life and was wrong at both, in opposite directions. A queued operation is cancellable whatever it is. EnqueueModelOp and EnqueueBackendOp select on the operation context, so cancelling one that is still waiting releases the delivery goroutine and abandonQueued retires it: the worker never sees it, nothing is downloaded, nothing is deleted. markQueued nevertheless wrote Cancellable: !deletion, so a queued removal reported cancellable: false and the UI hid the Cancel button in the one window where pressing it both works and leaves no trace. A removal queued behind a large install was stuck there until the install finished. A running removal is not cancellable at all. DeleteModel and DeleteBackend take no context, and modelHandler only checks the operation context after the call returns, so a "cancelled" verdict would land after the model was already gone. Both handlers nevertheless wrote Cancellable: true unconditionally at entry, ahead of the op.Delete branch, offering a Cancel button the server cannot honour. So the queued phase is more cancellable than the running phase, which is the reverse of the usual shape. markQueued now reports true unconditionally, and the handler-entry writes report !op.Delete. Both sites carry a comment saying why, because reading either one alone suggests the other is a bug. GalleryStore.Create keeps !op.Delete: it runs at dequeue, so its value already describes the running phase. Its comment now says so. Specs cover queued removal, queued install, running removal and running install through the handlers, plus the queued-removal case through /api/operations where the flag is consumed, plus the behaviour the whole asymmetry rests on: a removal cancelled while queued never reaches the worker and deletes nothing. No existing spec asserted the old values. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Write] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(activity): clamp the installer message, and add a real-binary e2e spec Running the page against a real local-ai showed the legacy installer message wrapping to three lines and dominating the card: it embeds an absolute file path, so it is both long and a single unbreakable token. One line, ellipsised, full text in the title, matching what the error string already does. The spec that found it runs with no route stubbing at all. Every other spec here stubs /api/operations, which is how a payload the server cannot emit (isDeletion true on a live operation) stayed green through a full review while the UI rendered a removal as an install. It is skipped unless LOCALAI_REAL_BINARY is set, so CI is unaffected. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-bot
|
823fc25bb7 |
fix(kokoro): add CPU backend fallback (#11161)
Publish the existing Kokoro CPU profile for amd64 and arm64 and use it as the default gallery capability so Vulkan-only and CPU hosts can install the backend. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]
|
366db11c59 |
chore: ⬆️ Update mudler/parakeet.cpp to 3e1ddd8455ceb9bfae564f84db24ba068b00c56e (#11150)
⬆️ Update mudler/parakeet.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]
|
54012002fd |
chore: ⬆️ Update ikawrakow/ik_llama.cpp to 5f063b7bbae8f9a34dfc5c704aa77939e76494a9 (#11153)
⬆️ Update ikawrakow/ik_llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]
|
176000190e |
chore: ⬆️ Update ggml-org/llama.cpp to 1cbfd1988311775425d36c0ce066590f7d3049cf (#11155)
⬆️ Update ggml-org/llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]
|
c4d0c060ed |
chore: ⬆️ Update CrispStrobe/CrispASR to 7bb8be77a8c1677e32bba58514bb2d42f29a7a48 (#11156)
⬆️ Update CrispStrobe/CrispASR Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]
|
cff31bbac0 |
chore: ⬆️ Update leejet/stable-diffusion.cpp to 22516991cbdf725e69b0b4a87e52ca16cce07c2d (#11157)
⬆️ Update leejet/stable-diffusion.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]
|
e218c7f56a |
chore(model-gallery): ⬆️ update checksum (#11152)
⬆️ Checksum updates in gallery/index.yaml Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]
|
12f2e1b99c |
feat(swagger): update swagger (#11149)
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]
|
2ecf893c6c |
fix(sherpa-onnx): install cuDNN in the CUDA builder so the package can bundle it (#11145)
sherpa-onnx links onnxruntime's CUDA execution provider, and libonnxruntime_providers_cuda.so carries cuDNN as a hard DT_NEEDED. The onnxruntime GPU tarball ships no cuDNN of its own, and Dockerfile.golang only installs libcudnn9 on the arm64 + CUDA 13 branch, so the amd64 CUDA builders have none at all. Since #10946 added the packaging guard, that combination is fatal rather than silent: package-gpu-libs.sh reports 'cuDNN: venv=absent system=absent -> bundle=detect', correctly detects the reference, finds nothing to copy and refuses to emit the package. Both -gpu-nvidia-cuda-12-sherpa-onnx and -gpu-nvidia-cuda-13-sherpa-onnx have failed to build since 2026-07-19, so neither image has been published. Before the guard existed they shipped without cuDNN and failed at load time instead. Install the runtime package for this backend only. The auto-detection bundles solely what a package references, so no other backend would grow, but every Go CUDA builder would pay ~1.1 GB of layer and registry cache for a library ggml never calls. Assisted-by: Claude Code:claude-opus-5 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]
|
05ff401de8 |
fix(test): stop the backend-trace specs racing the lossy trace channel (#11146)
RecordBackendTrace does a non-blocking send onto a 100-slot channel and
drops when it is full, so tracing never stalls inference. The payload
bounding specs pushed all 200 traces in one tight loop, which overruns
that channel on a loaded machine: entries are dropped for good and the
Eventually waiting for 200 can never be satisfied, no matter the timeout.
CI hit this on master at
|
||
|
Tai An
|
a7fa678d83 |
fix(tts): forward the OpenAI speed field to the backend (#11097) (#11120)
* fix(tts): forward the OpenAI speed field to the backend (#11097) /v1/audio/speech accepted the documented OpenAI `speed` field and then dropped it: schema.TTSRequest had no Speed member, so the value never reached proto.TTSRequest and the request returned 200 with an unchanged playback rate. Accept speed and normalise it into the existing per-request params map, which core/backend forwards verbatim to the backend. An explicit params["speed"] still wins, and a value outside the documented 0.25-4.0 range is now rejected with 400 instead of being silently ignored. Signed-off-by: Anai-Guo <antai12232931@outlook.com> * fix(tts): distinguish explicit speed=0 from an omitted field Make TTSRequest.Speed a *float32 so an explicit `"speed": 0` (invalid, below the documented 0.25 minimum) is rejected with 400 instead of being treated as unset and silently defaulted. An omitted field stays nil and leaves the backend default untouched. Add a request-boundary regression that distinguishes an omitted speed from an explicit zero, addressing review feedback. Signed-off-by: Anai-Guo <antai12232931@outlook.com> * docs: drop the speed field from the TTS docs Per review: no backend consumes params.speed today, so documenting it would be misleading. The API-level plumbing and validation stay. Signed-off-by: Anai-Guo <antai12232931@outlook.com> --------- Signed-off-by: Anai-Guo <antai12232931@outlook.com> |
||
|
Anupam Mediratta
|
3698361510 |
fix: upgrade hono to 4.12.25 (CVE-2026-54290) (#11023)
* fix: CVE-2026-54290 security vulnerability Automated dependency upgrade by OrbisAI Security Signed-off-by: orbisai0security <mediratta@gmail.com> Signed-off-by: Anupam Mediratta <mediratta@gmail.com> * fix(deps): override hono transitive dep to eliminate CVE-2026-54290 Add package.json `overrides` field to force hono@4.12.25 across the entire dependency graph, including the transitive copy pulled in by @modelcontextprotocol/sdk. Previously bun.lock retained a scoped `@modelcontextprotocol/sdk/hono` entry resolved to the vulnerable hono@4.12.8; the override removes that entry so only the patched version ships. Assisted-by: Claude Code:claude-sonnet-4-6 Signed-off-by: Anupam Mediratta <mediratta@gmail.com> --------- Signed-off-by: orbisai0security <mediratta@gmail.com> Signed-off-by: Anupam Mediratta <mediratta@gmail.com> |
||
|
mudler's LocalAI [bot]
|
856b0ea951 |
fix(ci): build the CUDA 13 image on Ubuntu 24.04 (#11143)
* fix(ci): build the CUDA 13 image on Ubuntu 24.04 The amd64 `-gpu-nvidia-cuda-13` image is the only runtime image still built FROM ubuntu:22.04. The Ubuntu 24.04 migration (#7769) bumped its `ubuntu-version` to 2404 but left `base-image` on jammy, so the image ships glibc 2.35 while adding the noble CUDA apt repository, and every backend it unpacks is built on noble. Backends therefore cannot dlopen the libraries they bundle. The vLLM backend dies at import time with: OSError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by /backends/cuda13-vllm/lib/libnuma.so.1) and torchcodec finds no usable libavutil because jammy ships ffmpeg 4.x (libavutil.so.56) while torchcodec looks for .so.57 through .so.60. Add a spec over the build matrices that fails when a base image and the `ubuntu-version`/`ubuntu-codename` it is paired with disagree, or when the runtime images are split across Ubuntu releases. Entries whose base image does not name a release (JetPack) are left alone. The `base-grpc-cuda-13-amd64` builder base stays on jammy: it only compiles backends, and a lower glibc floor in a builder is safe. Fixes #11059 Assisted-by: Claude:claude-opus-5 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ci): drop the build matrix invariant spec Per review, the CI matrix guard does not belong in the tree. Only the base image bump remains. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]
|
878a0d00a1 |
fix(distributed): reaper reaps live backends, ghost model stubs, in_flight leak, sidecar staging runaway (#11142)
* fix(distributed): stop the probe reaper from orphaning busy backends
The reconciler's liveness probe is a 1s gRPC HealthCheck, and a single
failed probe deleted the model's node_models row. A backend that is
merely busy cannot answer it: single-threaded Python backends (video and
avatar generation) block for minutes inside one request, so the reaper
was deleting registry rows for backends that were alive and mid-request.
The model then vanished from the nodes page while it was still
generating, and because the row was gone the in-flight decrement had
nothing to decrement ("DecrementInFlight: no matching row or already
zero"). Every subsequent request re-routed and re-staged the full model
from scratch.
Two guards:
- Replicas with in-flight requests are excluded in SQL. A row that is
actively serving is proof of life, and the running request is
exactly what stops the backend from answering the probe.
- Idle replicas must miss three CONSECUTIVE probes before removal, so
a transient blip cannot orphan a live replica. A successful probe
resets the streak.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(distributed): drop the local model stub when its last replica goes
In distributed mode every routed model leaves an in-process stub in the
frontend's ModelLoader, and DistributedModelStore.Range reports local
stubs UNION the registry rows. Every registry removal path deletes only
the DB row, so the stub outlived the replica and the model was reported
as loaded forever.
That is the "loaded on the home page, absent from every node" ghost:
/system reads the union and still sees the stub, while /api/nodes/models
reads the registry and correctly sees nothing. It never self-healed,
and both frontend replicas showed it independently.
The replica-removed chokepoint could not fix this as it stood, because
it held a SINGLE hook that the prefix cache already owned, and it was
registered only when the prefix cache was enabled. Registering a second
listener would have silently displaced the first.
- Turn replicaRemovedHook into a list (AddReplicaRemovedHook), so
independent subsystems can each register without displacing others.
- Add NewLocalStubInvalidator, which drops the local stub once no
healthy replica of the model remains anywhere in the cluster, and
wire it unconditionally in startup.
The stub is kept while another node still serves the model: the
frontend is right to consider it loaded, and each request re-routes
through SmartRouter to pick a live replica anyway.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(distributed): stop staging checksum sidecars back to workers
The file transfer server writes a "<file>.sha256" sidecar next to every
file it accepts. The sender walked the model directory with no filter,
so it staged those sidecars too, and the receiver duly wrote a sidecar
for each sidecar. Every staging pass multiplied the tree:
config.json -> config.json.sha256 -> config.json.sha256.sha256 -> ...
One LongCat snapshot had grown to 498 files, 466 of them chained, up to
29 levels deep, and the staged file count climbed on every pass. This
inflates each transfer and grows disk without bound on both ends.
Skip hash sidecars in stageDirectory, and mirror the skip in
countStageableFiles so the progress bar still reaches 100%. The check is
"a sidecar sitting next to a real file" rather than a blanket suffix
ban, so a model that genuinely ships a .sha256 payload with no
corresponding base file is still transferred.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(distributed): classify the liveness probe instead of gating on in_flight
The previous commit excluded replicas with in-flight requests from the probe
reaper. That was the wrong guard, and could invert the bug it fixed.
in_flight has no decrement guarantee: track() balances its increment with a
defer, but a frontend killed mid-request never runs it, and the load-time
reservation is released only when the first inference completes. Nothing
resets a leaked counter. Gating the reaper on it therefore meant a leaked
counter would shield a genuinely dead replica from ever being reaped.
Nor was patience alone a fix: three misses at the default interval is ~90s of
silence, while the generation that triggered this blocks for 15+ minutes.
The real conflation was in the probe itself. A gRPC HealthCheck against the
backend's serving port measures "is it idle enough to answer", not "does the
process exist", and probeLoadedModels discarded the error that tells them
apart. Because the gRPC client is lazy, the status code is decisive:
- DeadlineExceeded: transport fine, nothing serviced the RPC. Busy.
- Unavailable: nothing is listening. Gone.
ModelProber now returns a ProbeOutcome, and only ProbeUnreachable counts
toward the reap threshold. ProbeBusy clears the streak: it is evidence of
life. A blackholed network reads as busy too, deliberately, since whole-node
failure is the health monitor's job.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(distributed): reconcile replicas against worker-reported processes
Probing a backend's own serving port cannot distinguish "busy" from "gone"
without inferring it from an error code. The worker can answer directly: it
spawned the process, holds the handle, and its reply is not blocked by
whatever that backend is doing.
Adds a models.running request-reply subject. The worker answers out of its
in-memory process table, reporting each live process as (modelID,
replicaIndex, address) — the supervisor's process keys are `modelID#replica`,
which is isomorphic to a NodeModel row, so the reconciler can diff the two
directly.
reconcileNodeProcesses runs before the port probe and reaps rows for models
the worker is not running. Models the worker vouches for get updated_at
bumped, which takes them out of the port prober's stale set entirely: that is
what keeps a backend deep in a long generation away from the probe in the
first place, rather than relying on classifying its silence after the fact.
A worker that does not answer is skipped, not assumed empty. A messaging
failure says nothing about the processes, and assuming the worst would delete
a node's rows on a transient NATS blip; the port probe stays as the fallback
for those nodes. Rows younger than probeStaleAfter are ignored so a freshly
created row is never judged against a process table that has not caught up.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(distributed): stop in_flight leaking and pin replicas against eviction
A leaked in_flight counter is not cosmetic. FindLRUModel,
FindGlobalLRUModelWithZeroInFlight and the router's eviction query all require
in_flight = 0, so a replica whose counter never came back is pinned and its
VRAM is unreclaimable for the lifetime of the process.
Two halves.
The source: routing reserves in_flight = 1 at load time so a freshly loaded
replica is not evicted out from under the request that caused the load. That
reservation was released ONLY by the first inference completing, so a route
torn down before any inference ran (client disconnect, handler error, failure
between load and the backend call) stranded it. newRouteResult now wires the
reservation to a sync.Once fired by whichever comes first, the first inference
or route teardown, and replaces three copies of the old wiring.
The backstop: a sweeper for counters leaked by paths that cannot run a defer
at all, such as a frontend killed mid-request.
Identifying a leak by elapsed time alone is unsafe. IncrementInFlight stamps
last_used at request START and nothing moves it while the request runs, so a
long generation is indistinguishable from a leak by age, and resetting there
would expose a serving model to eviction. The probe supplies the missing bit:
a backend that answers a health check promptly is not inside a request,
because that is precisely what a busy one cannot do. Requiring the row to also
be idle for 30 minutes covers backends that serve in parallel and can answer
while working, since those keep last_used fresh through each new increment.
Two existing tests asserted the old behaviour ("No decrement on Release").
That assertion was the leak, so both now pin the release instead.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
mudler's LocalAI [bot]
|
e9c2754fc2 |
chore(model-gallery): propose variant groupings for review (#11139)
chore(model-gallery): propose variant groupings Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]
|
0a8a7fbbb4 |
chore(llama-cpp): bump llama.cpp and adapt to the load-mode refactor (#11140)
Bump LLAMA_VERSION to 0d47ea7427463093e69128bf2c2f9cd06b3ee5b3 (73 commits touching common/, src/ and tools/server/). Two upstream changes break the backend: * ggml-org/llama.cpp#20834 folded common_params::use_mmap / use_mlock / use_direct_io into a single `load_mode` enum. LocalAI still exposes the three as independent settings (`mmap`, `mmlock`, and the `direct_io` option), so params_parse folds them once all three have been read, keeping the precedence the separate booleans had: direct I/O bypasses the page cache, mlock implies mmap, everything off is a plain buffered read. turboquant and bonsai compile this same grpc-server.cpp against forks that predate the refactor, so prepare.sh probes the checkout for LLAMA_LOAD_MODE_MMAP and generates llama_compat.h with LOCALAI_LEGACY_LOAD_MODE set accordingly. Probing beats a per-fork build flag here because the fork flavor targets disagree on whether they forward CMAKE_ARGS or EXTRA_CMAKE_ARGS, and it heals itself once a fork rebases past the refactor. * The MiniMax M3 patch no longer applies. Upstream merged the model half of llama.cpp#24523 (LLM_ARCH_MINIMAX_M3, src/models/minimax-m3.cpp, the gguf-py constants and conversion/minimax.py) but not the chat half, so the patch is re-cut to carry only the common/chat.cpp template detection and PEG parser, rebased onto the new pin and onto the thinking_end_tag -> thinking_end_tags rename. Dropping it wholesale (as #11008 did, reverted in #11136) would have silently regressed MiniMax M3 tool calling and thinking. Verified with a CPU docker build of the backend plus LoadModel and Predict against a real GGUF over gRPC in all four load modes. Assisted-by: Claude:claude-opus-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
d9f3007876 |
Revert "chore: ⬆️ Update ggml-org/llama.cpp to d2a818231effb12b7b20b80b3b8c7756a9a33a04" (#11136)
Revert "chore: ⬆️ Update ggml-org/llama.cpp to `d2a818231effb12b7b20b…"
This reverts commit
|
||
|
mudler's LocalAI [bot]
|
6e69dbd617 |
chore: ⬆️ Update ggml-org/llama.cpp to d2a818231effb12b7b20b80b3b8c7756a9a33a04 (#11008)
* ⬆️ Update ggml-org/llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix(llama-cpp): drop upstreamed MiniMax M3 patch The pinned llama.cpp revision already contains MiniMax M3 support, so the downstream patch rejects during backend preparation on every platform. Assisted-by: Codex:gpt-5 --------- Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]
|
19ffc33011 |
chore: ⬆️ Update leejet/stable-diffusion.cpp to 2d0385ba85af358f7115dda608a63eafd9de7ffd (#11132)
⬆️ Update leejet/stable-diffusion.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]
|
76ce4f59b6 |
chore: ⬆️ Update vllm-metal (darwin) to v0.3.0.dev20260726174827 (#11133)
⬆️ Update vllm-project/vllm-metal (darwin) Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
dependabot[bot]
|
4f883c86a9 |
chore(deps): bump postcss from 8.5.15 to 8.5.23 in /core/http/react-ui in the npm_and_yarn group across 1 directory (#11106)
chore(deps): bump postcss Bumps the npm_and_yarn group with 1 update in the /core/http/react-ui directory: [postcss](https://github.com/postcss/postcss). Updates `postcss` from 8.5.15 to 8.5.23 - [Release notes](https://github.com/postcss/postcss/releases) - [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md) - [Commits](https://github.com/postcss/postcss/compare/8.5.15...8.5.23) --- updated-dependencies: - dependency-name: postcss dependency-version: 8.5.23 dependency-type: indirect dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]
|
c56373d772 |
chore(model-gallery): ⬆️ update checksum (#11134)
⬆️ Checksum updates in gallery/index.yaml Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
Tai An
|
53006bb8e1 |
fix(realtime): accept legacy 'modalities' alias for output_modalities (fixes #11103) (#11104)
* fix(realtime): accept legacy 'modalities' alias for output_modalities OpenAI's Realtime *beta* used the field name `modalities`; the GA field is `output_modalities`. LocalAI only binds `output_modalities`, so a client sending the still-common beta field `modalities: ["text"]` has it silently dropped by encoding/json and the session falls back to audio: TTS runs and the client receives large response.output_audio.* frames even though it asked for text-only. Accept `modalities` as an alias on both session.update (RealtimeSession) and response.create (ResponseCreateParams). The GA `output_modalities` wins when both are present, so GA clients are unaffected. Applied at the two existing resolution points via a small modalitiesWithAlias helper. Fixes #11103 Signed-off-by: Anai-Guo <antai12232931@anaiguo.com> * test(realtime): add JSON-boundary regression for modalities alias Decode representative session.update and response.create payloads that carry only the legacy beta `modalities` key and assert the effective output modality resolves to text (not audio), reproducing the exact expressions used in updateSession and triggerResponseAtTurn. This guards against a wrong JSON tag or a missed call site letting encoding/json drop the alias silently. Also document output_modalities (and the accepted legacy modalities alias) for text-only sessions in the realtime feature docs. Signed-off-by: Tai An <antai12232931@outlook.com> --------- Signed-off-by: Anai-Guo <antai12232931@anaiguo.com> Signed-off-by: Tai An <antai12232931@outlook.com> Co-authored-by: Anai-Guo <antai12232931@anaiguo.com> |
||
|
mudler's LocalAI [bot]
|
62e3b8304e |
chore: ⬆️ Update CrispStrobe/CrispASR to 306faee45fab641d54f9f941f075de1e9c0d3278 (#11131)
⬆️ Update CrispStrobe/CrispASR Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]
|
2476509321 |
chore: ⬆️ Update ikawrakow/ik_llama.cpp to 0a4e10c7fb65d2dd5a4afb78339c7d373a8cdfaa (#11128)
⬆️ Update ikawrakow/ik_llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
dependabot[bot]
|
ac9352ef54 |
chore(deps): bump actions/setup-node from 4 to 7 (#11080)
Bumps [actions/setup-node](https://github.com/actions/setup-node) from 4 to 7. - [Release notes](https://github.com/actions/setup-node/releases) - [Commits](https://github.com/actions/setup-node/compare/v4...v7) --- updated-dependencies: - dependency-name: actions/setup-node dependency-version: '7' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |