One agent turn runs several internal LLM generations (tool selection,
reasoning, final answer) that all emit stream_event deltas over the same
per-agent SSE channel. The chat page accumulated every 'content' delta
into a single live bubble and ignored the 'done' boundary events, so the
internal generations' text (e.g. the English tool-selection rationale)
merged with — and visually corrupted — the streamed final answer.
Reset the accumulated content/reasoning on 'done': each generation gets
a clean live bubble, and the authoritative full answer still arrives via
the final json_message event as before.
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
chore(moss-transcribe-cpp): bump pin to CUDA K-quant embed fix
Bumps the moss-transcribe.cpp pin to 190a569c, which merges the
host-side embed-lookup fallback for K-quant token_embd tensors
(localai-org/moss-transcribe.cpp#2).
Before this, running a q5_K/q4_K/q6_K moss-transcribe GGUF on CUDA
(or any non-CPU backend) aborted in getrows.cu with
"unsupported src0 type: q5_K" because ggml's GET_ROWS op has no
K-quant implementation on GPU, killing the backend process on the
first request. The engine now dequantizes the needed rows on the
host when the backend cannot run GET_ROWS for the tensor type,
producing bit-identical results.
Fixes#10862
Assisted-by: Claude:claude-opus-4-8 [Bash] [Edit]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Add the q8_0 (default) and f16 gallery entries for the moss-tts-cpp backend, each
pulling the MOSS-TTS-Local v1.5 GGUF plus the MOSS-Audio-Tokenizer-v2 codec and
the text tokenizer from mudler/MOSS-TTS-Local-Transformer-v1.5-GGUF. The backend
auto-discovers the codec and tokenizer siblings; output is 48 kHz stereo with
reference-audio voice cloning.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(backend): add moss-tts-cpp text-to-speech backend
Add a Go + purego backend wrapping the moss-tts.cpp ggml port of the OpenMOSS
MOSS-TTS-Local v1.5 text-to-speech model (GPT-J local transformer decoded through
MOSS-Audio-Tokenizer-v2), producing 48 kHz stereo audio with optional
reference-audio voice cloning. Mirrors the qwen3-tts-cpp backend: dlopen the
static-ggml shared library, bind the moss-tts.cpp C-API via purego, and serve
the gRPC TTS method. A thin C shim holds the pipeline handle and copies engine
PCM into a Go-freeable buffer.
Wires the CI registration: backend-matrix.yml (CPU, CUDA 12/13, Intel SYCL
f16/f32, Vulkan, ROCm, NVIDIA L4T, plus Darwin metal), backend/index.yaml metas
and image entries pointing at mudler/MOSS-TTS-Local-Transformer-v1.5-GGUF, the
root Makefile build targets, and the changed-backends.js path mapping.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs: list the moss-tts-cpp backend among the LocalAI-maintained engines
Add moss-tts.cpp to the README "Backends built by us" table, the
Text-to-Speech compatibility table, and the reference-audio voice-cloning
backend list, so the new backend is documented alongside its peers.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(moss-tts-cpp): pin moss-tts.cpp to the squashed single-commit release
moss-tts.cpp history was collapsed to a single commit; repoint MOSSTTS_CPP_VERSION
to ee722b8e9205ee9b1b1c398a4e87e4e393e9be41.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* backend(moss-tts-cpp): add the moss-tts-cpp-development gallery meta
The gallery had the -development image entries but no matching -development
meta anchor (as locate-anything-cpp and depth-anything-cpp have), so the master
build was not installable as a gallery backend. Add moss-tts-cpp-development
mirroring the production meta with the -development capability image names.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Google shipped the Gemma 4 MTP drafter heads and llama.cpp merged native
support in ggml-org/llama.cpp#23398. LocalAI's pinned llama.cpp already
carries it, and the config plumbing (draft_model + core/config/mtp.go)
was built for exactly this path, but no official Gemma 4 gallery entry
wired it up.
Add llama.cpp draft-mtp speculative-decoding variants for the dense
sizes, sourced from the unsloth QAT GGUF repos (target UD-Q4_K_XL +
mtp-*.gguf drafter + BF16 mmproj):
- gemma-4-e2b-it-qat-mtp
- gemma-4-e4b-it-qat-mtp
- gemma-4-12b-it-qat-mtp
- gemma-4-31b-it-qat-mtp
These replace the previously commented-out attempts, which were disabled
because the Janvitos/boxwrench drafter GGUFs declared the architecture as
`gemma4_assistant` (underscore) and failed to load on stock llama.cpp.
The unsloth drafters use the upstream `gemma4-assistant` (hyphen) spelling
that mtp.go's isDraftOnlyAssistantArch expects, so they load without any
backend patch. The 26B-A4B MoE is intentionally omitted (the upstream PR
reports no meaningful MTP speedup for it).
Also fix gemmable-4-12b-mtp: it loaded the draft-only `-mtp` GGUF as the
main model with no draft_model set, which cannot run standalone. It now
loads the target as the model, wires the drafter via draft_model, enables
spec_type:draft-mtp, and downloads both files.
All sha256 pins were taken from the HuggingFace API lfs.oid (reliable
content hash even for Xet-backed repos).
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
StopP2P() read and wrote a.p2pCtx/a.p2pCancel without holding
a.p2pMutex, and StartP2P() reassigned both fields with no lock at
all -- including when RestartP2P() calls it from a background
goroutine after releasing the mutex. Both paths are reachable from
POST /api/settings (empty p2p_token -> StopP2P, non-empty ->
RestartP2P), so concurrent requests race on the same fields.
Take a.p2pMutex in StopP2P and around the field publication in
StartP2P, factor the shared teardown into stopP2PLocked() so
RestartP2P reuses it, and route the goroutine error path through
StopP2P instead of touching a.p2pCancel unlocked.
Signed-off-by: Anai-Guo <antai12232931@anaiguo.com>
Co-authored-by: Anai-Guo <antai12232931@anaiguo.com>
* ⬆️ Update CrispStrobe/CrispASR
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(crispasr): rewrite c2pa-audio submodule path for subproject builds
CrispASR a38cb89 adds a crispasr_c2pa_native static library whose sources
live in the new third_party/c2pa-audio git submodule, located via
CMAKE_SOURCE_DIR in src/CMakeLists.txt. That variable assumes CrispASR is
the top-level CMake project; LocalAI embeds it via add_subdirectory, so
the path resolved to backend/go/crispasr/third_party/c2pa-audio and every
build variant failed at CMake generate with 'Cannot find source file:
c2pa_native.cpp'.
Extend the existing talk-llama sed workaround to also rewrite the
c2pa-audio reference to PROJECT_SOURCE_DIR, which is correct both
standalone and as a subproject. The submodule itself is already checked
out by the recursive submodule init. Verified locally: the exact CI error
reproduces with CMAKE_SOURCE_DIR, and with the rewrite CMake configure,
crispasr_c2pa_native, and crispasr-lib all build cleanly on a CPU-only
fallback configuration.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-4-8
---------
Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The turboquant and bonsai backends copy backend/cpp/llama-cpp/ wholesale
into their build directories and reuse its Makefile/prepare.sh against
their own llama.cpp forks. When PR #10837 added
backend/cpp/llama-cpp/patches/0001-add-minimax-m3-support.patch, the
copied patches/ directory was mis-applied to the fork checkouts: the
fork trees diverge from upstream, hunks rejected, and because the
patch-apply loop in prepare.sh ran before set -e took effect the build
kept going and died much later with a confusing compile error
("'LLM_ARCH_MINIMAX_M3' was not declared in this scope"). This broke
tests-turboquant-grpc on that PR.
Two hardening changes:
- turboquant/bonsai Makefiles: delete the copied patches/ directory
right after the cp -rf of backend/cpp/llama-cpp/. Patches vendored
for upstream llama.cpp must never be applied to the forks; each fork
carries its own patch series under backend/cpp/<backend>/patches/,
applied by its apply-patches.sh.
- llama-cpp prepare.sh: run the patch-apply loop under set -e so a
rejecting patch fails fast and loudly at apply time instead of
surfacing as a downstream compile error. A missing or empty patches/
directory remains a no-op success, so all existing callers (the
llama-cpp Makefile targets and the turboquant/bonsai copies) are
unaffected when no patches ship.
Exposed by PR #10837.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(settings): add declarative runtime-settings field registry
One fieldSpec row per RuntimeSettings field, with a reflection
completeness spec so a field added without a registry row is a red
test instead of a silently-dropped setting (the #10845 bug class).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* refactor(settings): drive ToRuntimeSettings/ApplyRuntimeSettings from the field registry
Behavior-preserving: ~350 hand-written per-field lines become two loops
over runtimeSettingsFields, gated by a To->Apply->To round-trip spec.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* feat(settings): baseline-driven startup merge for persisted runtime settings
ApplyRuntimeSettingsAtStartup compares the live config against
DefaultRuntimeBaseline (option-less-run defaults incl. kong-injected
flag defaults) instead of per-field == 0 guards. Fixes persisted
lru_eviction_max_retries, tracing_max_items, agent_job_retention_days,
memory_reclaimer_threshold, galleries and autoload flags being
silently ignored at boot.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* fix(settings): registry-driven startup merge, applied before consumers
loadRuntimeSettingsFromFile becomes a thin wrapper over
ApplyRuntimeSettingsAtStartup and runs at the top of New(), before
model configs capture app-level defaults. WithThreads stops eagerly
resolving 0 so a persisted thread count survives restart while
LOCALAI_THREADS still wins (#10845); the physical-core fallback moves
after the merge.
Also: run.go now injects the memory-reclaimer threshold unconditionally
so the option-less boot matches DefaultRuntimeBaseline and a UI-saved
threshold survives restart.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* refactor(settings): file watcher delegates to the registry merge; shared API-key merge
Manual edits to runtime_settings.json now behave like a boot-time load
(env still wins) instead of the inverted diverged-from-startup guard
that ignored most manual edits. MergeAPIKeys dedups env keys in one
place for the endpoint and the watcher.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* docs(settings): document unified runtime-settings precedence
Document the single env/CLI > runtime_settings.json > defaults rule,
applied identically at boot, on POST /api/settings, and on manual file
edits, plus the two known limitations (default-valued env vars are
indistinguishable from unset; API-changed fields hot-apply on the next
restart only). Also add a completion debug log when the watcher applies
runtime_settings.json.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
* test(settings): reset the global VRAM cap leaked by the round-trip spec
The round-trip spec applies vram_budget=12GiB, whose post-loop hook
installs a process-global default cap; without a reset every spec
ordered after it runs under that phantom budget. Also drop a stale
enumeration in the ApplyRuntimeSettings doc comment.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-fable-5
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The gRPC configuration table only listed the two fields with a one-line
description each, without defaults, without explaining what the total
load window looks like, and without hinting when a user should adjust
them. In practice the default 20 attempts x 2 s = 40 s window is way
too tight for large NVFP4 / FP8 models on slow storage or first-run
CUDA-graph capture, and the resulting kill (exitCode=120, 'context
canceled') looks like a backend crash even though the backend is still
making legitimate forward progress.
Extend the section with:
- Defaults column (20 and 2) added to the table
- Prose explaining that these govern the readiness handshake between
LocalAI and a freshly spawned backend (Health polling loop)
- Total-load-window formula
- Concrete failure signature so users can recognize a timeout-kill
vs. a real backend crash
- Example configuration for a ~10 min cold-load window (grpc.attempts
140, attempts_sleep_time 5), with a note that inference-timeouts and
the watchdog are unaffected.
The sglang Python backend inherits the default Status RPC from
backend_pb2_grpc.BackendServicer, which raises NotImplementedError.
LocalAI's backend-monitor polls /backend.Backend/Status periodically on
every registered backend; when the call fails, /backend/monitor returns
HTTP 500 and downstream inference requests to the sglang backend are
blocked even though the model is loaded and answering directly via the
gRPC endpoint.
Add a minimal Status shim that mirrors the existing Health method and
returns StatusResponse{state=READY} unconditionally. This unblocks the
monitor path; a state-aware follow-up (UNINITIALIZED during load, BUSY
under active inference) is left for a subsequent change.
Reproduced on DGX Spark (GB10, arm64-l4t-cuda-13 image) with the sglang
v0.5.15 backend and Qwen3-Coder-Next-NVFP4-GB10; verified locally that
patching the shim in place immediately restores /backend/monitor and
inference across the sglang slot.
* fix(audio-transform): serialize WebSocket writes to avoid concurrent-write panic
AudioTransformStreamEndpoint writes to the same Gorilla WebSocket connection
from two goroutines: the backend-forwarding goroutine emits binary PCM frames
(and can call sendWSError on a backend recv error), while the read loop calls
sendWSError for malformed mid-stream JSON or a backend send failure. Gorilla
WebSocket permits only one concurrent writer, so these writers race and can
panic with "concurrent write to websocket connection", resetting the client
session; a -race build reports the data race directly.
Wrap the connection in a lockedConn that serializes WriteMessage behind a
mutex, mirroring the existing lockedConn used by the openresponses WebSocket
endpoint. Reads stay on the single read loop, so only writes need the lock.
Fixes#10844
Signed-off-by: Tai An <antai12232931@outlook.com>
* chore: empty commit to re-trigger checks
Signed-off-by: Anai-Guo <antai12232931@anaiguo.com>
---------
Signed-off-by: Tai An <antai12232931@outlook.com>
Signed-off-by: Anai-Guo <antai12232931@anaiguo.com>
Co-authored-by: Anai-Guo <antai12232931@anaiguo.com>
When the watchdog's busy-killer decides a backend has been busy past the
busy timeout, it shuts it down via ModelLoader.ShutdownModel -> deleteProcess,
which grabs ml.mu and then waits for IsBusy() to clear BEFORE stopping the
process. But a backend that exceeds the busy timeout is, by definition,
stuck on an in-flight gRPC call, so the graceful wait never returns, ml.mu
is held forever, and every other ml.Load blocks — including the shared
opus backend load at the start of every realtime (WebRTC) session. New
realtime connections then hang at "Connected, waiting for session..."
whenever the watchdog is enabled, while logs repeatedly print the
watchdog's busy / "active connection" line.
Fix: add a force shutdown path (ShutdownModelForce / deleteProcess(s,
force=true)) that stops the process FIRST — dropping the stuck call's
gRPC connection and unblocking it — instead of waiting on it. Route the
watchdog's busy-killer and busy LRU / group / memory evictions through
the force path; keep the graceful wait for idle and user-initulated
unloads. Graceful/unforced kills are unchanged.
Regression test: the watchdog busy-killer uses ShutdownModelForce.
Fixes#10391
Assisted-by: opencode:glm-5.2 [opencode]
Signed-off-by: Nandana Dileep <110280757+nandanadileep@users.noreply.github.com>
feat(bonsai): add PrismML llama.cpp fork backend + Bonsai gallery models
Adds a new `bonsai` backend that runs the PrismML fork of llama.cpp
(github.com/PrismML-Eng/llama.cpp, `prism` branch), which ships the Q1_0
(1-bit) and Q2_0 (ternary / 1.58-bit) weight-quantization kernels used by the
Bonsai and Ternary-Bonsai models. Stock llama.cpp cannot decode these quants.
Modeled on the turboquant backend: reuses backend/cpp/llama-cpp/grpc-server.cpp
against the fork's libllama via a thin wrapper Makefile, so the sub-2-bit models
are served with the same OpenAI-compatible API. No grpc-server allow-list patch
is needed (bonsai adds weight quants, transparent to the server, not KV-cache
types), and the reused server compiles cleanly against the fork with no skew
patches (validated locally via a CPU docker build; patches/ is present but empty
for any future re-pin skew).
Backend wiring: backend/cpp/bonsai/, .docker/bonsai-compile.sh,
backend/Dockerfile.bonsai, top-level Makefile targets, backend-matrix.yml build
rows (CPU, CUDA 12/13, L4T, SYCL f32/f16, Vulkan, ROCm/hipblas), backend/index.yaml
meta-backend + per-platform images, and a nightly bump_deps entry tracking the
`prism` branch.
Gallery: 8 entries across 4 families - bonsai-8b-1bit, ternary-bonsai-8b (+g64,
+pq2), bonsai-27b-1bit (vision), ternary-bonsai-27b (+pq2, +g64, vision). The 27B
models wire the mmproj vision tower; the DSpark speculative drafter GGUFs are not
wired (custom semi-autoregressive drafter, not a standard llama.cpp draft model).
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(watchdog): guard StopWatchdog with watchdogMutex to prevent double close
StopWatchdog checked, closed and cleared a.watchdogStop without holding
a.watchdogMutex, while startWatchdog and RestartWatchdog reassign and close the
same channel under that lock.
POST /api/settings dispatches to StopWatchdog or RestartWatchdog depending on
ApplicationConfig.WatchdogShouldRun(), so both are reachable concurrently. Two
callers can observe a non-nil watchdogStop and both close it, which panics with
'close of closed channel' and takes the server down.
Take the mutex, matching the other two writers. StopWatchdog is only called from
the settings handler, which holds no lock, so this cannot deadlock.
Fixes#10841
Co-authored-by: Anai Guo <antai12232931@anaiguo.com>
Finetune() compiled every model cutstrings/extract_regex entry via regexp.Compile
and called xlog.Fatal on failure, which terminates the entire local-ai process.
A single model config with an invalid regex (e.g. cutstrings: ["("]) turns one
/v1/chat/completions request into a process-level denial of service.
Log the compile error and skip the offending pattern instead. The mutex is
released before continuing, and skipping avoids dereferencing the nil regexp
that removing the fatal would otherwise leave behind.
Fixes#10843
Signed-off-by: Tai An <antai12232931@outlook.com>
ApplyRuntimeSettings persists the performance settings (threads,
context_size, f16) on the live /api/settings path, but the startup
loader loadRuntimeSettingsFromFile never read them back, so a value
saved via the Middleware UI was silently ignored on the next restart:
the model booted with the CLI/physical-core default and GET /api/settings
echoed that default instead of the saved value (#10845).
Threads needs special handling: unlike context_size/f16, WithThreads
eagerly resolves an unset (0) value to xsysinfo.CPUPhysicalCores() at
option-apply time, so options.Threads is never 0 in the loader and the
usual "== default" heuristic cannot tell an env/CLI value from the
physical-core fallback. Detect LOCALAI_THREADS/THREADS explicitly so the
env still wins over the persisted file value.
Signed-off-by: Anai-Guo <Anai-Guo@users.noreply.github.com>
Co-authored-by: Anai-Guo <Anai-Guo@users.noreply.github.com>
Add stable and development gallery variants for Linux and Darwin, and wire the backend build matrix so the referenced images are published.
Assisted-by: Codex:gpt-5 [yq]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
Pairs unsloth/Qwen3.5-4B-GGUF (Q4_K_M target) with the
AtomicChat/Qwen3.5-4B-DFlash-GGUF Q8_0 drafter (quantized from
z-lab/Qwen3.5-4B-DFlash, upstream GGUF arch `dflash`), same shape as
the existing DFlash entries.
Assisted-by: Claude Code:claude-fable-5 [Bash] [Read] [Edit]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): add vrambudget primitive for per-node VRAM caps
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): apply default VRAM budget in xsysinfo aggregate getters
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): wire LOCALAI_VRAM_BUDGET flag to xsysinfo default budget
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): persist VRAM budget via runtime settings with live apply
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(vram): reset process-global VRAM budget after runtime-settings spec
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): add VRAM budget field to Settings page
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): store and enforce per-node VRAM budget in the node registry
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): apply per-node VRAM budget in router hardware defaults
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): report worker VRAM budget in node registration
The distributed worker now reports its operator-set VRAM budget string
(LOCALAI_VRAM_BUDGET) to the server on registration. The worker keeps
reporting RAW total/available VRAM and never sets the xsysinfo
process-global budget (that stays standalone-only); the server resolves
and enforces the budget uniformly (Task 6).
Also closes a Task 6 gap: on re-registration, a struct Updates zero-skips
an empty budget, so a worker that dropped LOCALAI_VRAM_BUDGET left the
stale cap in place. For non-admin-override nodes the budget columns are
now force-written (map Updates) even when empty, so removing the env var
clears the cap; admin overrides are preserved unchanged.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* style(vram): drop em dash from worker-clear comment
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): add node VRAM budget admin endpoints
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): add node VRAM budget control to the node UI
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(vram): expose set_node_vram_budget MCP admin tool
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(vram): document LOCALAI_VRAM_BUDGET and node VRAM budget UI
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vram): avoid double-applying VRAM budget in GetResourceAggregateInfo
The GPU-branch aggregate returned by GetResourceInfo is sourced from
GetGPUAggregateInfo, which already caps total/free/used against the
process-wide VRAM budget. GetResourceAggregateInfo then applied the
budget a second time. For an absolute budget this is idempotent, but for
a percentage budget b.Apply resolves the ceiling as a fraction of its
input total, so a second pass yields P*(P*T) instead of P*T and distorts
UsagePercent (read by the memory reclaimer in pkg/model/watchdog.go).
Remove the redundant second application so the budget is applied exactly
once, against the raw physical totals, upstream in GetGPUAggregateInfo.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(vram): implement SetNodeVRAMBudget on mcp assistant test stub
The LocalAIClient interface gained SetNodeVRAMBudget; the stubClient in
core/http/endpoints/mcp used by the assistant tests is a separate
implementer and needs the method too (broke golangci-lint typecheck and
both test jobs).
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
* test(core/http): make the suite's HTTP port overridable
app_test.go and openresponses_test.go hardcoded 127.0.0.1:9090. When
another service already listens on 9090 the suite does not fail fast:
the server goroutine logs the bind error and the specs then poll
whatever is squatting the port until Eventually times out. On machines
where 9090 is permanently taken this makes the pre-commit coverage gate
impossible to pass.
Introduce testHTTPAddr, defaulting to 127.0.0.1:9090 (what CI has
always used) and overridable via LOCALAI_TEST_HTTP_PORT for local runs.
Assisted-by: Claude:claude-fable-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(distributed): make per-node backend upgrade actually upgrade
The node detail page's Upgrade button reused the node-scoped install
path (POST /api/nodes/:id/backends/install). That fires NATS
backend.install with force=false, and the worker's install handler is
deliberately "ensure installed": when the backend binary already exists
on disk it short-circuits without touching the gallery. Since only an
installed backend can be upgraded, the whole chain was a guaranteed
successful no-op - the UI then toasted "backend upgraded" without even
waiting for the async job.
Route upgrades through the real force-reinstall path instead:
- BackendManager.UpgradeBackend now receives the ManagementOp (like
InstallBackend already did) so implementations can honor
op.TargetNodeID.
- DistributedBackendManager.UpgradeBackend scopes the backend.upgrade
fan-out to op.TargetNodeID when set, and errors when the target node
does not report the backend as installed.
- New POST /api/nodes/:id/backends/upgrade endpoint enqueues an
Upgrade=true node-scoped op (async 202 + jobID, mirroring install).
- NodeDetail UI calls the new endpoint and reports the dispatch
("Upgrading ... on this node...") instead of claiming success; the
Operations panel tracks the actual job.
Verified against a live local cluster (NATS + Postgres + two workers):
the target worker stops the running process, force-reinstalls from the
gallery and re-downloads the OCI image; the second worker receives no
backend.upgrade event; upgrading a backend missing from the target node
fails the job with a clear error.
Assisted-by: Claude:claude-fable-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
PyTorch 2.13 XPU pulls oneAPI 2026 libraries that conflict with the oneAPI 2025.3 backend image. Pin torch and torchaudio to the matching 2.11 XPU pair so the build resolves a coherent 2025.3 runtime.
Assisted-by: Codex:GPT-5 [uv]
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
SetDefaults injected the llama.cpp server options cache_reuse
(ApplyServingDefaults) and parallel (ApplyHardwareDefaults, re-applied
per selected node by the distributed router) onto every model config
regardless of backend. Every other backend ignores options it does not
understand, so this was harmless until longcat-video, which strictly
validates its options and fails LoadModel with
"unknown model option(s): cache_reuse, parallel".
Gate both injections behind a new UsesLlamaCppServingOptions allow-list
(llama-cpp plus the empty/auto-detect case that resolves to llama.cpp
from a GGUF file, mirroring how llamaCppDefaults is registered). This
follows the existing UsesLlamaSamplerDefaults precedent for llama-only
defaults. The typed NBatch field is deliberately left alone: it is a
proto field every backend simply ignores, which is why batch never
triggered the error.
Also harden the longcat-video backend to warn-and-ignore unknown model
options and request params through a testable select_known_options
helper, matching the other LocalAI Python backends, so a future
server-injected option cannot break loading again.
Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
handleRegenerate rebuilt the outbound message from the display-only
message.files metadata ({name, type: 'file'|'image'|..., content}),
which doesn't carry the base64/textContent payload sendMessage's
file-building loop expects. As a result, regenerating any answer whose
own question had an attachment silently dropped that attachment from
the resent message. This wasn't fork-specific, but forking a chat and
then regenerating an earlier (now non-last) answer is the natural way
to hit it.
Fix by reusing the original message's already-assembled `content`
verbatim (it already has the file text / image_url / audio_url /
video_url parts embedded from the first send) instead of trying to
reconstruct it from lossy display metadata.
Fixes#10806
Assisted-by: Claude:claude-sonnet-5
Signed-off-by: ajuijas <189517297+ajuijas@users.noreply.github.com>
Co-authored-by: ajuijas <189517297+ajuijas@users.noreply.github.com>
