mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-29 09:35:02 -04:00
4151ceda858ffded16de2906cfeccd7eb0d75760
353
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 localai-org-maint-bot
|
4151ceda85 |
chore: merge master into distributed transport PR
Keep the version-reporting import from master and omit the unused sanitize import after the transport changes. Assisted-by: Codex:GPT-6 |
||
|
Stefan Walcz
|
bc1d9924de |
fix(llama-cpp): keep llama.cpp's default cache_ram instead of no limit (#12297)
grpc-server.cpp forced params.cache_ram_mib = -1 (no limit) since #7009. Since v4.3 kv_unified and cache_idle_slots are on by default, so every distinct prompt now leaves its slot KV state in the host-side prompt cache, and without a limit the backend grows until the host runs out of memory. Measured on gfx1151 (Strix Halo, 128 GB), llama-cpp backend, one request at a time, 100 distinct prompts of ~2000 characters plus a fixed system prompt, max_tokens 200: model cache_ram RSS loaded -> after 100 gemma-4-26B-A4B (q8_0 KV) -1 (default) 1.4 GB -> 25.3 GB Qwen3.6-35B-A3B (q8_0 KV) -1 (default) 1.1 GB -> 19.5 GB gemma-4-26B-A4B -1, same prompt 100x 1.4 GB -> 1.6 GB gemma-4-26B-A4B 4096 1.4 GB -> 5.4 GB (flat from request 20 on, same latency) Qwen3.6-35B-A3B 4096 1.1 GB -> 5.1 GB (flat) The memory is not released when idle. In production a document classification pass pushed the daily chat model to 34 GB RSS overnight. Drop the override so llama.cpp's own default (8192 MiB) applies; the cache_ram option still accepts -1 for users who want no limit. Update both docs tables (the option reference and the prompt-cache table) and note what -1 does. Assisted-by: Claude:claude-opus-5-5 Assisted-by: Codex:GPT-6 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> |
||
|
mudler-agentandEttore Di Giacinto
|
97ad8f1d70 |
fix(distributed): stage the files a model install declares (#12309)
* fix(distributed): stage every shard of a split GGUF A split GGUF is configured by its first shard only. llama.cpp opens the other "-0000N-of-0000M.gguf" files from the same directory by name. The router staged only the configured path, so the worker received shard 1 and the load failed with "failed to load GGUF split". The router now stages the remaining shards next to the first one. A missing shard fails the load and names the file. The file count for progress and the payload size also include all shards. The payload size feeds the load deadline and the disk-headroom check. For a 111 GB model whose first shard is 10 MB, both were sized for less than 1 GB. Assisted-by: Claude:claude-opus-5-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): stage the files a model install declares Replace the split GGUF file name matching with the model's own file list. A gallery install or an import records every file of the model in ._gallery_<name>.yaml (files:), and a config can list more under download_files:. The router now stages all of these files, not only the files that the config's path fields name. This includes the other shards of a split GGUF, which llama.cpp opens by name. The application gives the router a resolver that reads the two lists. The resolver looks up the files by model name when it stages them, so a replica that the reconciler loads from saved load options gets the same files. backend.proto does not change. A declared file that is missing on the frontend is skipped with a warning. The load deadline and the disk headroom check include the declared files. Assisted-by: Claude:claude-opus-5-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Pratik Gandhi
|
ea10f26f57 |
docs: fix 11 dead links in the documentation (#12320)
- compatibility table: voxtral.c, OuteTTS and VoxCPM repos live under antirez, edwko and OpenBMB - distributed inferencing: llama.cpp RPC README moved to tools/rpc under ggml-org - customize-model: the phi-2 example config moved to LocalAI-examples, and embedded/models was replaced by the gallery - model-gallery: malformed URL; link the gallery index - GPU acceleration: ROCm install guide moved - integrations: Wave Terminal docs page moved to ai-presets Assisted-by: Claude:claude-fable-5-1 Signed-off-by: Pratik Gandhi <travpreneur@gmail.com> |
||
|
|
61f4f67b75 |
sglang backend: pass through thinking_budget + require_reasoning (#12193)
* sglang backend: pass through thinking_budget + require_reasoning sglang's raw Engine.async_generate() API (which this backend calls directly, bypassing sglang's own OpenAI server) supports a precise, tokenizer-derived reasoning-length budget via sampling_params["custom_params"]["thinking_budget"] plus require_reasoning=True, gated behind --enable-strict-thinking. Neither was reachable through LocalAI: this backend built sampling_params only from a fixed field mapping (temperature, top_p, ...) with no custom_params key, and never passed require_reasoning to async_generate at all. - LoadModel now reads a model-level "thinking_budget" option (same mechanism as the existing tool_parser/reasoning_parser options), and _build_sampling_params adds it as custom_params.thinking_budget on every request when configured. - _new_reasoning_parser already derives, from the rendered prompt, whether the model's chat template pre-opened a reasoning block (Qwen3-style templates append <think> to the prompt instead of letting the model emit it) -- the same signal sglang's own OpenAI server computes from per-template config to decide require_reasoning. This backend has no template manager, so it now returns that signal too and _predict forwards it to async_generate(require_reasoning=...). Verified against production (NVFP4, sm_121, Qwen3.6-35B-A3B) via a raw Engine.async_generate() call bypassing this backend: 301 reasoning tokens against a 300-token budget, clean completion, ~27s. Not yet verified through this backend's own gRPC path end-to-end (no local CUDA/sglang environment available here) -- existing + new unit tests in test.py cover the pure-Python merge/passthrough logic only. Scope note: require_reasoning is derived only from the existing prompt-suffix heuristic, not sglang's full per-template _get_reasoning_from_request decision tree (minimax-m3/hunyuan special cases etc.) -- this backend has no template manager to evaluate that tree against, and the prompt-suffix check is the one heuristic already validated in this file (test_reasoning_parser_forced_when_template_prefills_think_tag). Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com> * sglang backend: honour a model-level reasoning_default A model YAML can already carry "parameters: reasoning_effort:", but that value only reaches this backend when a *caller* sets it per request (the Go side turns it into Metadata["enable_thinking"]). As a model-level default it is silently dropped: a config reading "reasoning_effort: none" still produces full reasoning on every request, so the config says one thing and the model does another. That gap is expensive in practice. On a self-hosted Qwen3.6-35B-A3B the reasoning phase consumed the entire max_tokens budget before any content was produced - 90% of code completions came back empty at max_tokens=768, and the server log filled with "backend produced only reasoning, retrying". The config looked like reasoning was off the whole time. This adds "reasoning_default:off" (or ":on") on the same model-level options: mechanism as thinking_budget. A per-request value always wins; the default only fills in when the request is silent. Measured on the stack above (sglang 0.5.20, NVFP4, GB10/sm_121) after applying it: default (nothing set) -> 0 chars reasoning, 27 tokens "reasoning_effort": "none" -> 0 chars reasoning, 27 tokens metadata enable_thinking=true -> capped at the 512-token thinking_budget, 541 tokens total, finish_reason stop Tests: three cases added to backend/python/sglang/test.py covering the default, per-request override in both directions, and the unconfigured case (which must leave the template untouched). Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com> * sglang backend: validate thinking_budget instead of crashing LoadModel Addresses the review on this PR: - `int(thinking_budget)` raised on values like "5000.0" or "abc" and took LoadModel down. The option is now parsed by _parse_thinking_budget(): integral numbers in any spelling are accepted, anything else is ignored with a warning on stderr. - Zero and negative budgets are ignored with a warning instead of being passed to sglang, where they have no defined meaning. Turning reasoning off is what reasoning_default:off is for. - A load-time warning when thinking_budget is set but enable_strict_thinking is not in engine_args, since sglang then ignores the budget silently. - Tests for integral spellings, unset, zero, negative, non-integer and the strict-thinking warning. Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com> * docs(sglang): explain reasoning options Document the reasoning budget, strict-thinking requirement, and precedence of request metadata over the model-level default. Also note that the budget has to stay well below max_tokens (otherwise it never triggers and the reply can end up empty), and that POST /models/reload or a backend-only restart does not pick up changed options; LocalAI itself has to be restarted. Assisted-by: Codex:GPT-6 Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com> * docs(sglang): clarify configuration reloads Distinguish rereading model configuration from updating a running backend. Keep the full LocalAI restart recommendation for changed reasoning options. Assisted-by: Codex:GPT-6 * sglang backend: only pass require_reasoning when sglang supports it Engine.async_generate() gained the require_reasoning keyword in sglang 0.5.13 and takes no **kwargs. The CPU profile builds v0.5.11 from source and the other profiles only set a >=0.5.11 floor, so passing the keyword unconditionally made every request fail with TypeError. Detect support once at import time, as the file already does for sampling_seed. enable_strict_thinking first appears in sglang 0.5.12; fix the comment. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5-5 [Claude Code] --------- Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com> Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: localai-org-maint-bot <localai-org-maint-bot@users.noreply.github.com> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-bot
|
733eeda123 |
chore: merge master into distributed transport PR
Keep the newer SQLite dependency from master to resolve the conflict. Assisted-by: Codex:gpt-6 |
||
|
Ettore Di Giacinto
|
0565fc06af |
Merge PR #12302: chore(deps): bump LocalAGI to 7e0947d (no-RAG-DB crash fix, tool filters, per-collection models)
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
f82efdb43b |
fix(models): fallback to application config default context size in /v1/models/capabilities (#12202) (#12216)
* fix(models): fallback to application config default context size (#12202) Honor appConfig.ContextSize in /v1/models/capabilities when model context_size is unset. * docs(models): explain context size fallback Describe the application default used by capability discovery and preserve the distinction between total context and per-request limits. Assisted-by: Codex:GPT-6 * fix(models): apply the default context size only when context_size is unset The request path applies the application default context size only when a model leaves context_size unset. An explicit 0 or -1 falls through to the backend fallback. The capabilities endpoint now does the same, so it reports the value the backend uses. Assisted-by: Claude:claude-opus-5-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
84e2fc5eac |
feat(agents): support tool lists and required tool in distributed mode
LocalAGI 7e0947d added allowed_tools/excluded_tools and the required_tool_before_finish gate. Single-node agents get them through LocalAGI's runtime, but the distributed executor drives cogito directly and its static config meta did not list the fields, so the agent form hid them and the worker ignored them. The distributed config now parses the tool lists from a JSON array or a comma/newline separated string, and the meta entries match LocalAGI's. The executor filters the knowledge base, skill and MCP tools (MCP via cogito.WithMCPToolFilter) before the model sees them, and re-prompts the model when it answers before the required tool returned "ok": true, up to the configured number of reminders. LocalAGI keeps its filter and gate helpers unexported, so a minimal copy lives in core/services/agents/toolpolicy.go. A spec compares the meta entries with LocalAGI's to catch drift. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5-5 [Claude Code] |
||
|
Ettore Di Giacinto
|
b7155a9d97 |
chore(deps): bump LocalAGI to 7e0947d
Pick up the LocalAGI PRs merged after f2a2af4: - per-collection embedding and reranker models, locked per collection so one agent's upload or rerank no longer stalls the others (#499) - required_tool_before_finish: a tool the agent must call successfully before it may answer (#495) - allowed_tools / excluded_tools per agent, applied to MCP tools too (#480) Document the new agent settings. They show up in the single-node agent form, which reads LocalAGI's config metadata; distributed mode keeps its own field list and does not offer them yet. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5-5 [Claude Code] |
||
|
Ettore Di Giacinto
|
7f821ab7ef |
Merge PR #12287: chore(gallery): add Sharp-Spark 4B variants
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> # Conflicts: # docs/content/features/model-gallery.md |
||
|
Ettore Di Giacinto
|
4693ccf737 |
Merge PR #12293: chore(gallery): add Swift 1.5 GSQ-RCO variants
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> # Conflicts: # docs/content/features/model-gallery.md |
||
|
Ettore Di Giacinto
|
aed7b7823a |
Merge PR #12295: chore(gallery): add ThinkingCap Qwen3.8 variants
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> # Conflicts: # docs/content/features/model-gallery.md |
||
|
Ettore Di Giacinto
|
a45dd81e22 |
Merge PR #12296: chore(gallery): add Agention Qwen3.8 variants
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> # Conflicts: # docs/content/features/model-gallery.md |
||
|
Ettore Di Giacinto
|
ae6ccb5f52 |
Merge PR #12298: chore(gallery): add Qwopus Flash V2 variants
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
1819c33f5f |
Merge PR #12300: chore(gallery): add Cyber-Tiel-Coder variants
Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
490b952d06 |
feat(gallery): publish signed OCI fallbacks (#12182)
* feat(gallery): publish signed OCI fallbacks Publish both official gallery indexes with their local base configs so an outage of the HTTP and GitHub sources can fall back to Quay. Keep artifact signing policies separate from backend image policies, and expose each moving gallery tag only after its digest is signed. Assisted-by: Codex:gpt-6 * fix(gallery): confine packaged files to selected roots Use directory-scoped file access to reject symlink escapes during gallery packaging. Create private bundle files for the publishing runner. Assisted-by: Codex:GPT-6 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
5794495a37 |
fix(responses): preserve streamed output items (#12048)
Keep each message and reasoning item at its announced output index. Include the answer in completed responses with reasoning or fallback function calls, and retain reasoning supplied through backend deltas. Add regression coverage for stream indices, final output, plain text, and automatic tool parsing. Assisted-by: Codex:GPT-6 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
0e52bb657e |
fix(responses): wait for complete JSON tool calls (#12001)
Partial JSON parsing heals a name-only chunk into a tool call. The stream emits that call with empty arguments and skips later chunks. Require complete JSON before emitting terminal tool-call events. Preserve complete calls before an unfinished trailing call, and count only actual tool calls. Add split-chunk regression tests and docs. Refs #11635. The non-streaming report remains unconfirmed. Assisted-by: Codex:GPT-6 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
1b1bd0f069 |
fix(compose): request NVIDIA compute capability (#11990)
The legacy NVIDIA device reservation requests utility without compute. Docker derives driver capabilities from that list, leaving CUDA libraries unavailable even when monitoring works. Include compute in the legacy example and clarify the matching docs. Assisted-by: Codex:GPT-6 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
localai-org-maint-bot
|
065f9691fa |
chore(gallery): add Cyber-Tiel-Coder variants
Add Q4 and Q8 MTP builds with a shared vision projector and installation docs. Assisted-by: Codex:gpt-6 |
||
|
localai-org-maint-bot
|
6043e5e0cb |
chore(gallery): add Qwopus Flash V2 variants
Add Q4_K_M and Q8_0 builds with vision and MTP decoding. Pin the weights and projector to a verified Hugging Face revision. Assisted-by: Codex:gpt-6 |
||
|
localai-org-maint-bot
|
dcddb641f0 |
chore(gallery): add Agention Qwen3.8 variants
Add IQ4_XS and Q4_K_M GGUF builds with a BF16 vision projector. Pin verified artifacts and document installation and variant selection. Assisted-by: Codex:gpt-6 |
||
|
localai-org-maint-bot
|
7460312d23 |
chore(gallery): add ThinkingCap Qwen3.8 variants
Add Q4_K_M and Q8_0 builds with the F16 vision projector and install docs. Pin artifact revisions and verify SHA256 against HF LFS metadata and HTTP headers. Assisted-by: Codex:gpt-6 |
||
|
Ettore Di Giacinto
|
0cb7b7e70d |
fix(distributed): prove agent worker credentials
Keep pending agent workers alive with authenticated heartbeats while preserving approval as the tunnel and execution boundary. Exercise the real binary credential handoff through an authenticated inference and verify that credential remains non-admin. Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] |
||
|
localai-org-maint-botandEttore Di Giacinto
|
f041bee1cb |
fix(ui): complete node operation journeys (#12070)
Worker setup now uses a focused drawer without interrupting fleet monitoring. Both generated worker commands retain their NATS settings. Log views preserve their launch context and show one-based replica labels. Bulk controls disclose selections outside the current view. Assisted-by: Codex:gpt-5 Playwright ESLint Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-bot
|
b2be52e6f7 |
fix(worker): resolve staging directory symlinks
Resolve allowed directories before comparing them with resolved files. Otherwise staging rejects valid files under macOS temporary paths. Cover aliased roots, sibling paths, and symlinks escaping the root. Assisted-by: Codex:gpt-6 |
||
|
Ettore Di Giacinto
|
df30b1a0c4 |
fix(distributed): back a failed claim off instead of respinning it
The claim queue's attempts counter grew without bound and nothing read it. At the default two-second poll a permanently undispatchable row cost about 43000 UPDATEs a day, and it cost more than writes: rows are claimed oldest first, so the oldest stuck row was re-claimed ahead of every newer one on every tick and held a dispatch slot while it failed. One poison row starved the queue behind it. No dead letter, and that is the decision rather than the omission. Read settleClaim: the only outcome that releases a claim is one where NOTHING was learned about the work. No agent worker was connected, the tunnel broke, a peer could not be reached, the stream was refused before the request body left this replica. Not one of those is a worker saying it ran the job and it failed, and an attempt ceiling would turn "the fleet was away long enough" into a job failure nobody reported, which is the collapse this whole design exists to prevent pointed at work instead of at nodes. The one verdict available here, that no build of any worker serves this kind, is already settled as an answer. So the retry stays unbounded and the RATE does not. Each release stamps the row with the earliest it may be claimed again, doubling from two seconds to a cap of sixty, computed in the release statement from the row's own attempts count and stamped on the DATABASE clock, because that is the clock competing replicas order the queue on. Queued work becomes claimable again within one cap of the fleet returning, and a stuck row no longer holds the head of the queue. A claim released by the reap carries no delay at all: that work was never handed to anyone, so there is nothing to back off from. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
4aa0288c6e |
fix(worker): check a gRPC port is free before handing it out
The backend port allocator allocated from its own bookkeeping alone. That bookkeeping records what this worker did with a port, and the collision it cannot see is with something this worker never did: the default base port is 50051, inside Linux's default ephemeral range of 32768 to 60999, so the kernel hands ports in this range to outbound connections and to anything that binds port 0. A backend handed one of those dies on bind, and the frontend sees a backend that will not start. Every candidate is now probed by binding the exact address the backend will listen on, in all four allocation branches: the key's own port, the free pool, a grown port and a stolen one. Probing the free pool matters as much as probing a grown port, because a port this worker released is exactly as available to the kernel as one it never used. A candidate that fails the probe is quarantined rather than blacklisted, since whatever holds it is usually an ephemeral connection that gives it back, and its affinity claim is dropped so an unbindable port does not stay reserved for the key that last held it. Exhaustion now says how many candidates were skipped, which is what tells an operator "something else is in my range" from "my range is too narrow". This does not remove the race and cannot: between the probe and the child's bind the kernel can still give the port away. It removes the far larger window in which the allocator hands out a port the kernel gave away minutes ago, which was the whole of the observed one-in-three harness flake. The e2e harness comment that recorded the missing check is corrected, and the docs say how to move the range out of the ephemeral one entirely. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
07208b9180 |
fix(distributed): say that skills and collections are replica-local
Skills and RAG collections had no cross-replica invalidation, and the two builders that would have published one were deleted earlier in this branch because nothing called them. Wiring one now would be wrong, not merely late. Both features are derived entirely from the frontend's own state directory. A skills.Service indexes <state dir>/skills, a collections backend enumerates <state dir>/collections and holds one handle per collection it found there, and no replica reads or writes another replica's copy of either. In distributed mode PostgreSQL carries a skill's NAME and description in skills_metadata, and nothing else: Get, Search, Export and the resource verbs all read local files. So a peer told to drop a cache entry would rebuild it from a directory that does not hold the change. For a postgres-engine collection it would be worse than a no-op, since re-deriving one on a replica with no local index file yields a collection that answers with an empty file list against a populated vector store. What is missing is shared storage, not a broadcast. Recorded rather than left silent: the two cache fields say why nothing invalidates them, a distributed frontend logs the limitation once at startup, and the docs name the two deployments that avoid it. The new spec pins the premise, so a change that moved either directory onto storage every replica mounts reddens and the decision gets taken again. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
d72b591a78 |
feat(cluster): make a peer prove which replica it is
GET /api/cluster/peer authenticated with the deployment's shared registration token and took the dialling replica's id from ?id= on trust. Every worker holds that token, so anything holding it could open a peer link as any replica: relay through it to every worker tunnel that replica owns, displace a real replica's inbound link by declaring its id, and point the roughly 31 GiB per-session receive window at one replica. Validating the id against the instances table does not fix this, because the attack declares a real replica's id. So the route now checks two credentials and needs both. The shared token still says the dialler belongs to this deployment; a new per-replica credential says which replica it is. The credential follows the per-node worker credential rather than inventing a second mechanism: crypto/rand.Text, stored only as a hex SHA-256, compared in constant time, with no fallback to the shared token. It differs in the stronger direction. A worker's credential is minted by the frontend and handed over once; a replica writes its own instances row, so it mints its own secret, publishes only the hash in the same statement that publishes its address, and never sends the plaintext anywhere but the peer dial. A peer that presents no credential is refused, not waved through. An old replica and an attacker holding the shared token send the same request, so accepting the first accepts the second; there is no safe downgrade here, only a quiet one. The refusal is made loud instead, on both sides, naming the upgrade rather than the network. On the documented frontend-first order a new replica still dials an old one; an old replica cannot dial a new one, which costs relayed requests that land on a not-yet-restarted replica and surfaces as no route, never as absence. A rejected peer gets its own sentinel, ErrPeerRejected, whose unwrap chain carries ErrPeerUnreachable as well and no absence sentinel at all. Keeping the older sentinel means no existing consumer changes behaviour; the cause stays out of the chain, so absence cannot escape through it and nothing can read an authorization failure as a worker that went away. One consequence beyond the fix: a replica with no advertised address has no instances row, so it now cannot dial out either. It was already unreachable inward. The startup error and the docs say so. Registry.Register, NewMembership, NewPeerPool, PeerHandler and RegisterClusterRoutes all gained required arguments, so the identity cannot be dropped without a compile failure. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
eb1656ed9c |
chore(distributed): take the nats-io modules out of the build
Distributed mode has not dialled a message broker since the control plane moved onto the workers' own outward tunnels and every fan-out family moved onto PostgreSQL LISTEN/NOTIFY. What was left was the dependency itself, and the code that existed only to feed it. Dropped from go.mod: nats-io/jwt/v2, nats-io/nats.go, nats-io/nkeys, nats-io/nuid and testcontainers-go/modules/nats, along with the fourteen indirect requires that only the NATS testcontainer pulled in. go.sum carries no nats line either, so the removal is not the partial kind where the require goes and the checksum stays. Deleted with them: pkg/natsauth in full, the broker client's remaining options and TLS files, the per-node JWT minting on both the register and the approve path, and the natsauth.Config parameter threaded through the node routes. The credential manager is renamed and stripped rather than deleted, because it still holds the tunnel token that every re-registration rotates. The bus flags stay accepted and ignored, and are now hidden, on every command that had them, so an existing unit file, compose file or Helm values file still starts on the day of the upgrade. What is not kept is the validation that REQUIRED one: a distributed frontend started with no bus URL is no longer fatal. The TLS paths lose type:"existingfile" deliberately, so a certificate deleted along with the broker cannot fail a startup. One operator-visible behaviour change: --nats-require-auth no longer makes an agent worker wait through admin approval. Ask for that wait with --distributed-require-auth, which already implied it. It is documented in the migration section and pinned from both sides. A deployment now needs PostgreSQL and the frontends' own HTTP listener, and nothing else. coverage-baseline.txt moves from 54.2 to 62.0. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
e4e5dbd227 |
chore(distributed): stop telling an operator to run a NATS cluster
Every carrier had already moved and no process opened a bus connection, but the surface an operator reads still described a deployment with a broker in it: a compose service, a 220-line credential-generation script, two CI steps pulling a container nothing started, two flag tables offering --nats-url, an architecture diagram with a NATS box wired to the workers, a join-command generator in the Nodes page that emitted --nats-url for agent workers, and a test suite that stood a NATS server up for specs that no longer used it. That is the one way this programme could still fail invisibly. Every test passes, every binary works, and every production deployment goes on running and paying for infrastructure that carries nothing. Nothing in this repository starts a NATS server any more. The compose file is four services, the docs say to shut the broker down and what to keep, and the e2e suite runs on one PostgreSQL container. The three LOCALAI_NATS_*_TIMEOUT env vars are KEPT, and are now documented twice as being kept. They were never broker settings: each names a control-RPC budget the frontend applies to a worker, still read and still enforced. They carry the prefix only because they arrived with the bus, and renaming them would break every existing deployment for cosmetics. The agent worker's join command was the last surface still emitting the flag, two tasks after the agent worker stopped dialling. The Playwright spec that covered it asserted the opposite of what is now true, so it is inverted rather than deleted, and it reads the rendered command string rather than the component's variables: the variables are what the fix removes, so a spec reading them would have stopped compiling instead of failing, and a compile error is not evidence about what an operator is shown. nats_jwt_test.go and its helpers are deleted. They pinned a real server ENFORCING the minted permissions. The CONTENT of those allow lists is still pinned, untouched, by pkg/natsauth's own suites, including the spec that refuses to let the agent lists go empty, since an empty allow list in NATS means unrestricted. The enforcement half is retired rather than moved: enforcement is a property of a connection, and nothing opens one. The suite's own NATS container goes with them, which the brief left for the next task. Removing the pre-pull while BeforeSuite still ran the image would have defeated the step rather than cleaned it up, and this change removes the last reader of TestInfra.NC. agent_native_executor_test.go and mcp_ci_job_test.go are moved onto infra.Bus() instead of deleted: they were the last two specs building a bridge and a dispatcher on a client nobody uses, which is exactly the drift TestInfra.Bus's own comment warns about. cluster.Options.NatsURL is now fed a deliberately dead address rather than a live container's. Frontends and agent workers still receive LOCALAI_NATS_URL, because that is the coverage for the promise that an existing command line still starts; sourcing it from a running server would have let a regression that actually dialled it pass. The control in cluster_control_test.go keeps its assertion and loses its explanation, which claimed the deployment had a bus and no longer could. One latent spec race surfaced and is fixed: the background-run spec waited for a COUNT of events and then read a snapshot for the terminal status, which is the last event of a run and therefore always arrives after the count is met. Its immediate twin had already been fixed this way. Nothing in production changed. pkg/natsauth keeps its files. It is reachable from production only through the natsauth.Config parameter thread, and that thread is the next task's. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
793c4e211f |
fix(distributed): read a departed agent tunnel as the routing fact it is
Task 4 gave agent workers tunnels and deliberately left the NodeType skip in HealthMonitor.tunnelDeparted, with a spec asserting that an agent node whose presence reader answers PresenceGone is NOT marked unhealthy. That spec was scaffolding. It was true while an agent worker took its jobs and its verbs over the message bus: a departure row for one said nothing about whether it could work, and an early bug in the new tunnel client could otherwise have demoted a fleet of healthy agent workers. There is no bus. An agent worker is reachable through its tunnel and through nothing else, so a departed agent tunnel means exactly what a departed backend tunnel means: no live replica holds it, the departure has outlived the reconnect grace, and that is a routing fact the scheduler and a reaper may act on. The skip would now hide the only symptom an unreachable agent worker has. This is the deliberate removal Task 4's M6 predicted, and task-4-report.md is where that mutation already stands recorded red against the spec this commit deletes. The skip existed at ONE site. router_liveness.go has none: its candidates come from queries that already filter node_type = 'backend'. The two skips in managers_distributed.go stay, because an agent worker still runs no backend processes, so it has no backend to list and no backend op to apply. Two node types can depart now, which is why the second half exists. Before this, one type could depart and every per-node cache a departure left stale was dropped from wherever its owner happened to notice, so a reader could not tell which caches a demotion invalidated by reading the demotion path. Departure gets ONE notification point. DepartureNotifier is edge triggered, because the monitor runs on a ticker and a departed node stays departed; its subscribers are NAMED, because what has to be caught is a forgotten cache and a count can say only that one of four is missing; and NewHealthMonitor takes it as a required positional argument, so a caller that does not pass one fails to compile. Four caches subscribe: prefix-cache affinity in every model, probe freshness at every address, in-flight staging operations, and the per-node breakdown of every open gallery operation. The prefix-cache one is registered only when prefix-cache routing is enabled, so --distributed-prefix-cache=false stays a true no-op. The notification carries the node's name as well as its id, because the staging tracker keys on the name and the other two key on the id, and a subscriber should not have to read the registry from inside an eviction hook. A departure notification is an act on absence, so it fires only on the routing fact. A tunnel lost inside the grace, a worker that never dialled, a presence query that failed and a stale heartbeat all announce nothing, asserted per node type. The stale-heartbeat branch is excluded on purpose: it already marks the node offline, which deletes its rows and runs the registry's replica-removed hooks, so firing there too would double-evict and make the notification mean two different things at its subscribers. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
da8cf9c9fd |
feat(distributed): carry an agent cancel on the worker's own tunnel
agent.<name>.cancel was the last family on a message bus, and the only reason an agent worker dialled one. Its subscriber is the worker running the execution, and a worker has no database, so the family could not move to the PostgreSQL fan-out carrier: a cancel published there would reach no worker while reporting that it had been sent. It is a control verb now. An agent worker mounts workerctl.PathAgentCancel on the loopback control plane behind its tunnel and applies the cancel to the same registry the executor registers a run on. The frontend issues it through nodes.AgentControlClient.CancelAgentRun. That call is a FAN-OUT and not a pick, because nothing records which worker holds a given execution: the claim row names the claiming replica, and it is deleted when the run ends. Every agent worker a live replica can reach is asked over its own tunnel, relayed by the peer mesh when a peer holds it, and each worker answers only for itself. The answers stay apart, which is why this family was held back. A cancel a worker made is nil. A cancel some worker could not be asked is ErrAgentCancelUndelivered, which is neither a refusal nor a missing run. A cancel every reachable worker declined to own is ErrAgentRunNotOnAnyWorker. A deployment with no agent worker is ErrNoAgentWorker. Neither new sentinel wraps ErrWorkerUnroutable and neither is a worker answer, so nothing is reaped, demoted or evicted because of a cancel. A worker in the ABSENT CONNECTION condition, one whose tunnel was lost inside the reconnect grace, counts as undelivered. It is not retried in the call and not queued: a retry would spend a budget the caller did not choose, and a queue would need durable state whose only consumer is a run whose control stream went with the tunnel. A worker whose departure has outlived the grace is the one routing fact a caller may act on and is excluded, or a single retired agent node would make every cancel undelivered for ever. The fan-out reads a different node set from the pick. A draining worker takes no new work but is still finishing what it holds, so it is offered the cancel; a pending one is refused by the tunnel route on every dial and is not. With that, nothing in LocalAI connects to NATS. The agent worker's dial, its credential ladder and its refresh loop are gone, and so is the frontend's cancel carrier. LOCALAI_NATS_URL is accepted and ignored everywhere, and distributed mode no longer requires it. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
eb7ed9b910 |
refactor(distributed): delete MessagingClient and shrink the NATS client to fan-out
Nothing in the tree publishes, subscribes, queue-subscribes or requests through
the MessagingClient interface any more, so it is deleted rather than shrunk to
Broadcaster: two exported names for one method set in one package is an
invitation for the next author to pick whichever the surrounding file already
imported.
$ grep -rn 'messaging\.MessagingClient' --include='*.go' .
core/services/syncstate/syncstate.go:54: // It is messaging.Broadcaster rather than messaging.MessagingClient because
(one hit, a comment; no live referent. The naive grep in the plan also matches
prose and the local test type names fakeMessagingClient and
countingMessagingClient, so it can never be empty.)
*messaging.Client is shrunk to exactly Broadcaster plus its own lifecycle.
QueueSubscribe, QueueSubscribeReply, SubscribeReply, Request, Conn and the
package helpers QueueSubscribeJSON and RequestJSON go with it; none had a
production caller. Deleting the methods rather than only the call sites is what
makes putting a family back on this carrier a build error instead of a line that
compiles, publishes successfully, and is delivered onto a carrier the deployment
is being taken off. Conn is in that list because while it existed every other
name was one c.Conn().X() away; the flush-and-verdict that its real consumers
needed is now ConfirmRoundTrip, which keeps the NATS JWT permission specs armed.
The client, its options and its TLS plumbing are NOT deleted, and both processes
stay on the bus. agent.<name>.cancel is the one fan-out family that could not
move: its only subscriber is the agent worker, which has no database and cannot
join the PostgreSQL carrier at all, so a cancel published there would reach no
worker and be reported as sent. The frontend passes the client to
newFanoutBridges as its cancelCarrier and the worker subscribes on it, so
--nats-url stays required on agent-worker. Both go with the tunnel cancel verb.
The struct field is renamed Nats -> CancelCarrier to say what it is for, and
agentpool loses the messaging.Publisher it held only to be non-nil: it never
published on it, and it was gating whether a frontend runs agents distributed or
in an in-process pool. Retiring the bus would have flipped every replica back to
the in-process pool silently. The gate now reads the agent store, which is the
dependency the mode actually requires.
Also deletes four subject builders with no production publisher
(SubjectFineTuneProgress, SubjectFineTuneCancel, SubjectCacheInvalidateSkills,
SubjectCacheInvalidateCollection), the queue and request/reply halves of the
shared test double, and the e2e specs that were their only callers. Every
surviving subject is now pinned to its exact literal, because a subject is a
cross-version wire format and a rename that looks internal stops half a fleet
hearing the other half.
Docs: distributed-mode.md and cli-reference.md no longer claim NATS carries the
agent-worker job subjects, the frontend's cross-replica events, or an agent
worker's real work.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Ettore Di Giacinto
|
978f7922bd |
feat(distributed): move the last nine fan-out families onto PostgreSQL
Gallery progress and cancel, the operation cache's start and end, the model
and backend cache invalidations, staging progress, and the prefix cache's
observations and invalidations now travel on the LISTEN/NOTIFY carrier. No
subject is published or subscribed on messaging.Client anywhere in the tree,
which is what makes retiring that package a deletion rather than a migration:
$ grep -rn 'natsClient\.Publish\|nats\.Publish\|\.Nats\.Publish\|QueueSubscribe\|SubscribeReply\|\.Request(' \
--include='*.go' core/ pkg/ | grep -v _test \
| grep -v 'c\.Request()\|ctx\.Request()\|Request()\.Context' \
| grep -v 'core/services/testutil/fakebus.go'
core/services/messaging/client.go:168,170,172,227,234,236,250,252,254,268,269,287
core/services/messaging/interfaces.go:21,22,23
Every remaining hit is inside core/services/messaging itself. The production
reads of the NATS client are now three, all of them the documented agent-worker
exception: Close on shutdown, the agent pool's publisher, and the agent-cancel
carrier passed to newFanoutBridges.
Prefix-cache observations publish like every other family rather than through a
method that refuses a message too large for a notification. The plan proposed
such a refusal on the reasoning that a long prompt makes a chain of thousands of
entries; ExtractChain caps a chain at Config.MaxDepth blocks, MaxDepth is a
constant with no operator knob, and the chain reaching Sync.Observe has one
source, the router's own extraction hook. A worst-case observation is a few
kilobytes against an 8000-byte cap, so the hot-path spill the refusal was
designed to avoid cannot occur, and shipping it would have added the programme's
only deliberate message drop to guard a condition that cannot arise. pgbus gains
FitsInline instead, a predicate that shares one size decision with Publish and
decides nothing, and core/application refuses at startup to wire a prefix cache
whose configured depth would put every observation over the cap.
The carrier choice is no longer stated at four sites. StagingTracker.SetPublisher
and SubscribeBroadcasts become one SetBroadcaster, so a tracker that publishes
where its peers are not listening cannot be spelled; prefixcache.Sync gains
SubscribeBroadcasts, which reads the carrier it publishes on; and the gallery
service and the operation cache are wired by methods on DistributedServices that
name no carrier at all, so the NATS client beside it cannot be handed over.
OpCache.SetMessagingClient and GalleryService.SetNATSClient are renamed to
SetBroadcaster so a missed call site fails to compile.
Two pre-existing defects that the two-real-carrier specs surfaced are fixed. A
progress tick published before a cancel and delivered after it cleared Cancelled
and left the operation reading as still running on that replica; mergeStatus now
drops a stale tick rather than merging it. GetStatus and GetAllStatus handed out
the stored OpStatus pointer while the broadcast subscribers mutated it in place,
so an /api/operations response could be marshalled mid-write; both now copy.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Ettore Di Giacinto
|
6e2be0fd5a |
refactor(distributed): move job and agent fan-out onto the PostgreSQL carrier
Five of the six families whose subscriber is an open HTTP response rather than a process-lifetime cache now travel on pgbus: jobs.<id>.progress, jobs.<id>.result, jobs.<id>.cancel, agent.<name>.events.<user> and responses.<id>.cancel. Both ends of each move together, so there is no state where a publisher is on one carrier and its subscriber on the other. agent.<name>.cancel does NOT move, and the plan was wrong about why. Its only subscriber in the tree is the agent worker, which has no database and so cannot join the PostgreSQL carrier at all. Publishing that cancel on pgbus would have lost every cancel of a worker-run agent while returning nil, which reports a cancel that reached nobody as a cancel that was sent. EventBridge now names its cancel carrier separately, a frontend replica sets it to the carrier the worker reads, and it stays there until a cancel rides the worker's tunnel like every other verb addressed to a worker. The carrier drops at 256 rather than blocking, which is not safe on its own for a result: a lost result has no successor message. It is not the only path. The claiming replica persists the terminal line before it releases the claim, and an open progress stream re-reads the job row once after subscribing and then periodically, so a dropped terminal broadcast costs promptness and never the answer. Both per-request subscriptions close in a defer instead of on one return path, and pgbus grows Subscribers() so the leak they would otherwise cause can be asserted. It has no other symptom: only the first subscriber of a channel issues a LISTEN, so a leaked filter just adds one closure per notification for every stream the replica has ever served. Subscribe now issues its LISTEN before it registers, which makes that count a readiness signal rather than a figure to compare against itself. Two rules that were stated at several sites and pinned at none are now one each. The re-broadcaster is built beside the dispatcher and the bridge and handed to the dispatch loop, so no line is left that can point it at a carrier nobody subscribes to while every spec stays green. The set of statuses a job never leaves is one exported set that the SSE bridge and the store both read. The last hand-written subject filter in production code became messaging.SubjectAgentEventsWildcard. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
affad9c06d |
feat(distributed): carry the state.*.delta families on PostgreSQL
syncstate.Config held one carrier field typed as the NATS client, so a pgbus.Bus could not be handed to a SyncedMap at all: it satisfies messaging.Broadcaster and not MessagingClient. The durable re-hydration path built for the responses map therefore had a NATS-only consumer and nothing in the build said so. The field becomes Bus messaging.Broadcaster, SubscribeJSON moves to its own file and relaxes its parameter to Broadcaster, and the four adopters fan out over PostgreSQL LISTEN/NOTIFY: fine-tune jobs, quantization jobs, agent tasks with their per-tenant children, and Open Responses metadata. A new spec proves it on a real database, over two Bus instances on two pinned listener connections: a Set and a Delete carry, a payload past the 8000-byte notification cap comes back byte identical through the spill row, two families sharing one LISTEN channel stay separate, and a terminated listener re-hydrates a row written while it was gone. The five sites that each chose a carrier for an adopter are collapsed into one DistributedServices.Broadcast() accessor. Five field reads were five chances to leave one family on NATS with nothing failing, because messaging.Client satisfies Broadcaster too. The accessor also refuses to hand out a nil pgbus.Bus wrapped in a non-nil interface, which every adopter would read as "broadcast" and dereference on the first Set. SetTaskSyncNATS and SetJobSyncNATS are renamed to SetTaskSyncBus and SetJobSyncBus so a missed wiring site fails to compile. The response metadata table gains a retention of its own, defaulting to 24 hours. It inherited the Open Responses store TTL, which defaults to 0 meaning no expiration. Zero is defensible for a map that dies with the process and is not for a table: the table grew for the life of the deployment and a restarting replica re-hydrated every response the cluster had ever created. A row that names its own expiry is still judged on that column alone, and "this row is dead" now has one SQL spelling that PurgeExpired deletes by and ListUnexpired is the negation of, so a hydrate cannot resurrect what a sweep has already retired. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
1bf2b3f3d9 |
fix(distributed): give one tenant's agent tasks a subject of their own
Every AgentJobService built its tasks SyncedMap with the name "agent.tasks", and there is one service per user. So every tenant published on and subscribed to the same subject, state.agent-tasks.delta, and SyncedMap.apply scopes nothing: a task tenant A created was written into tenant B's in-memory map on every replica, and ListTasks reads that map. Nothing repaired it short of a process restart. The subject now carries the tenant in a token of its own, state.<name>.<tenant>.delta. Four tokens where the unscoped builder makes three, deliberately: SubjectMatches compares token count before anything else, so a tenant's subject and the cluster-wide one cannot cross-match, and neither can two tenants. Putting the tenant inside the name token would not do that, because the sanitizer folds '.' to '-' and the only filter that could then span tenants is state.*.delta, which spans every other family too. The rule is stated once. subscribeFilters calls publishSubject rather than restating the subject, so a map cannot end up publishing scoped and subscribing unscoped, which would leak exactly as before while every publish assertion passed. The one case that decides on its own is the cluster-wide administrative view: it hydrates from every tenant's rows, so it also takes the per-tenant wildcard, or it would be stale the moment any tenant wrote. A tenant hydrates from its own rows and applies only its own deltas. PerTenant defaults to false, so finetune, quantization and the responses store keep the subject they have. The second half of the same defect was the delete. taskStoreAdapter.Delete called DeleteTask(id) and JobStore deleted by primary key with no user predicate, reachable from DELETE /api/agent/tasks/:id, which takes the id off the URL. A tenant who learned another tenant's task id destroyed that tenant's row. The user id now travels with the id and lands as a user_id predicate. Empty stays the administrative any-owner scope, the same thing an empty id already means for ListTasks and ListJobs. A foreign delete removes nothing and returns no error: not yours and not there are the same answer to the caller, and neither is a store failure. SetUserID rebuilds the tasks map for the same reason SetTaskSyncNATS does. GetJobs happens to set the user id first, nothing enforced it, and with the order reversed the map would be built with an empty tenant and put that user's tasks back on the cluster-wide subject. Both halves predate this programme; they are surfaced here rather than caused. Neither is fully closed for a deployment with the agent pool off, where the task routes are still served by one cluster-wide service that every authenticated caller shares; that is a separate gap and it is documented. testutil.FakeBus grew a real defect this was the first change to trip: Unsubscribe matched on the filter string, so with two subscribers on one filter, closing one deafened the other. Subscriptions now carry an id. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
4b15161ffc |
feat(distributed): give responses.metadata something to re-hydrate from
The responses.metadata SyncedMap had no durable Store, so its reconnect re-hydrate replaced nothing. That was survivable while responses converged through deltas on a broker that mostly stayed up. It is not survivable on a carrier whose listener is one pinned PostgreSQL session: every response created while the subscription was down stays invisible on that replica forever, and the symptom is a 404 from one replica and a 200 from another for the same response_id. State that must survive a gap now lives in a response_metadata table, and the notification only says it changed. The map writes through on a Set and reads the table on hydrate, on reconnect and on reconcile, so the gap closes instead of becoming permanent. The row carries the whole projection as JSON rather than one column per field. A column-per-field schema would be a second definition of what a peer may act on, and the two would drift the first time syncedResponse gained a field: the map would broadcast the new field and hydrate without it, so a replica that had reconnected would serve a different response body from one that had not, with nothing failing anywhere. Only PayloadJSON is ever decoded; owner_replica and owner are indexed copies for an operator reading the table by hand. A missing row and an unreachable database are different facts. Every store and adapter method returns a driver failure as an error and never as an empty result, and syncstate replaces nothing when its source errors, so an outage leaves the map holding what it had rather than blanking it into a cluster-wide 404. Liveness is the database's clock, spelled expires_at IS NULL OR expires_at > now(), because every replica hydrating from this table must agree on which rows are live and a Go-side cutoff makes that a property of whichever process asked. The test container shares the host clock, so no behavioural spec can tell the two apart; the statement shape is pinned instead. The constructor refuses a non-PostgreSQL handle, because an unguarded now() on the single-binary path reads as a missing migration. A ticker sweeps expired rows every five minutes on each replica, and Close waits for it rather than racing it. Note that the sweep removes nothing while LOCALAI_OPEN_RESPONSES_STORE_TTL is 0, which is the default: with no TTL nothing ever expires and the table grows for the life of the deployment. The docs say so plainly. EnableDistributed takes the store positionally and last, so a call site that forgets it fails to compile rather than silently restoring the deltas-only map this change exists to replace. A nil store there is refused by name: it is reached only from the distributed branch of route registration, so it is a wiring bug and not a deployment shape. What still never leaves the owning replica is unchanged: the resume buffer and the CancelFunc. The write-through is one row per response state change, not one per generated token. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
a4c9247a68 |
feat(distributed): dispatch queued work as a claim queue
The three NATS queue groups jobs.new, jobs.mcp-ci.new and agent.execute are gone. Dispatching work is now a row in a work_claims table, taken by one frontend replica with SELECT ... FOR UPDATE SKIP LOCKED and driven on an agent worker as a streaming control RPC over that worker's tunnel. Exactly-one delivery among competing consumers is a database problem, not a broker feature. An agent worker has no database, so it never claims; it executes what the claiming replica hands it. A claim must not outlive the replica that took it. The reap releases a claim whose owner is no longer a live replica in the instances table, on the database clock, and never asks how long the claim has been held. A job that legitimately runs for an hour on a heartbeating replica is left alone, while a claim whose owner stopped heartbeating becomes claimable again within one liveness window. A replica with no advertised address has no instances row at all, so it refuses to claim rather than have its work reaped out from under it mid-run. The settle rule is stated once, in settleClaim, and every exit path calls it. A transport failure releases the claim and never completes or discards it; only a decoded reply line completes it. That line is deliberately not cluster.IsWorkerAnswer, which accepts the stream refusals a worker's tunnel writes before any request body reaches its control server: completing on those would discard work that never ran. The terminal line is persisted before the claim is completed, so a store that refuses leaves the claim standing rather than leaving the job running for ever. That is the dropped-result defect fixed structurally rather than by retry. This also surfaces a pre-existing gap rather than causing one: no worker has ever served plain task jobs, and publishing them into an empty queue group left them running with no trace. Such a claim is now failed with a reason. Removes QueueWorkers, --agent-subject and --agent-queue, and narrows an agent worker's minted JWT by agent.execute and jobs.mcp-ci.new. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
61e863e14a |
feat(distributed): stop a backend on one route, whatever the worker is
nodes.<id>.backend.stop was the last worker-facing NATS subject, and it existed only because ONE publisher had not moved. An agent worker already mounted workerctl.PathBackendStop on the tunnel it holds, and a backend worker already took its stop there, so RemoteUnloaderAdapter branched on NodeType to pick a carrier for a verb both kinds of worker served the same way. The branch is gone, and with it nodeTypeOf and its NodeTypeBackend default, which removes one of the ten NodeType branches left to sweep. The adapter loses its messaging.MessagingClient outright rather than keeping an unused field: it now holds no publisher, so re-routing any verb back onto the bus is a change to the struct and to every caller of the constructor, and does not compile until all of them agree. messaging.SubjectNodeBackendStop and subjectNodePrefix are deleted, the agent worker's subscription with them. pkg/natsauth drops the per-node backend.stop grant from the agent SUB list. That is a narrowing of eleven entries to ten, never to nothing: NATS reads an EMPTY allow list as unrestricted, so the coverage spec asserts both that the retired subject is no longer covered and that the queue subjects an agent worker lives on still are. The e2e half proves it against a real enforcing server: one spec subscribes successfully on an agent-minted JWT, the next is refused the retired subject on a JWT minted the same way. Both halves of the old split were pinned, so both pins are re-aimed rather than deleted, and the two node types are asserted separately rather than as one parameterised case, because only two cases can show that the two used to differ. Three assertions that the adapter published nothing are deleted instead: with no publisher to hold, no change could ever redden them. The CLI's handler set moves into agentWorkerControlHandlers so a spec can stand it up and post to it. That wiring was a bare literal no spec pinned, and deleting the subscription made it the ONLY carrier for backend.stop: a dropped field would have been a 404 the frontend reads as a worker too old to serve the verb, and nothing in the repo would have noticed. Mutations: the agent branch restored off the control route reddens two specs; the backend branch restored, separately, reddens five; PathBackendDelete in place of PathBackendStop reddens nine across both node types; dropping the CLI wiring line reddens the new wiring table; re-adding the allow-list entry reddens the unit spec and the JWT e2e spec; and restoring the publisher for real does not compile. Four comments this change falsified are fixed, in core/cli, pkg/model and the distributed-mode docs, which now say both kinds of worker serve POST /v1/control/backend/stop and what each does with it. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
e3040c7e3c |
feat(distributed): make MCP execution and discovery a selection
mcp.tools.execute and mcp.discovery were the only NATS subjects that combined a queue group with a reply, and no carrier in this design provides both. They never needed one: a queue group is a way of choosing a subscriber, and choosing is a query. The frontend now lists the approved, non-draining agent nodes, asks the node_connections table in one joined statement which of those tunnels a live replica holds, prefers one this replica holds so the call skips the relay hop, and issues an ordinary control RPC on the path task 4 already mounted. A peer-held tunnel is reached through the relay. That is a choice a broker's hidden balancing could not make. The selection reads presence and nothing else. It is filtered only on node type and on the two statuses an operator controls, never on a health verdict written on another clock, because refusing a worker that is connected and answering is the same defect as picking one that is gone. An empty fleet answers ErrNoAgentWorker, which is deliberately neither ErrWorkerUnroutable nor anything cluster.IsWorkerAnswer accepts: nothing was asked of any worker, so no reap guard may act on it. A reply carrying an Error is the worker's own answer and is returned unchanged; it is never offered to a second worker, which would turn "this MCP server rejected your arguments" into "the fleet is broken" and could run a tool twice. A call that never reached a worker is retried against a different pick, at most three times, and whatever error is finally returned is returned unwrapped so its identity survives the loop. MCP prompts and resources now answer 501 in distributed mode instead of an empty 200. They are served only from sessions the frontend holds, and in distributed mode it holds none. That gap predates the removal of the bus and is not closed by it; this only stops it being silent. Agent workers keep every other subject, including nodes.<id>.backend.stop. Their minted JWT loses the two MCP subjects and keeps a non-empty allow list, because NATS reads an empty one as no restriction at all. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
125e7080ce |
feat(distributed): give agent workers a tunnel of their own
Phase 2 gated agent nodes out of tunnel credentials at the mint site. That was right while nothing dialled into an agent worker: a credential would have replaced nothing, and the gate was structural rather than a second check that could drift. It is wrong now that the frontend needs to reach an agent worker by RPC. attachTunnelToken mints for backend and agent nodes and CLEARS for anything else, through one tunnelEligible predicate rather than two conditions that can be widened separately. ConnectHandler still never reads NodeType, so an empty hash is still what refuses an ineligible node. An agent worker now starts a loopback control server behind the same bearer check a backend worker uses, and holds one tunnel whose only stream tag is http: it runs no backend processes, so the grpc tag has nothing to route to and is not offered. Its MCP tool, MCP discovery and backend.stop verbs are served from ONE implementation reached by both the bus and the tunnel, so a frontend cannot get different bytes depending on which carrier delivered. The tunnel is an ADDITION. --nats-url is still required, and agent jobs, MCP execution, MCP CI jobs and nodes.<id>.backend.stop all still travel on the bus. Absence semantics are unchanged. An agent node now has a real node_connections row whose departure ages past the grace, so the node type check in HealthMonitor.tunnelDeparted stopped being an optimisation and became the rule; its comment says so, and the spec that pins it is shown red under a mutation that deletes the check. The scheduler needed no change: every placement query already filters node_type = backend, so an agent node never reaches nodeMayTakeWork. Shared rules moved to one site each. The request bounds, the POST-only check and the unknown-path 404 live in workerctl and are called by both worker packages; the bearer check that guards every extra route is one function in core/services/nodes used by both server constructors. workerctl.AllPaths splits into BackendPaths and AgentPaths, with AllPaths as their deduped union, because a backend worker does not mount the agent verbs and asserting otherwise would fail a correct worker. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
9f7ef0097b |
feat(distributed): keep the listener draining and let it come back
PostgreSQL holds undelivered notifications in a queue it shares with every session on the server, and it kills a listener that stops draining. Two failure modes follow, and both are silent: a carrier that blocked on a slow resolver would lose its connection and with it every later broadcast, and a carrier that reconnected without re-registering would be connected and deaf. The receive and dispatch halves were already separate. What was missing is everything around them. The listener path moves into listener.go and gains a carrier-level Dropped() so a replica that is behind can be seen; the queue depth and the spill retention become Config fields with exported defaults; the LISTEN session gets an application_name so an operator can count listeners in pg_stat_activity and a spec can drop exactly one of them; and OnReconnect fires after the re-LISTEN, on a goroutine of its own, because a callback re-hydrates from a database and must never run on the path whose only job is to drain. That callback is reached through an optional interface assertion, so deleting its invocation compiles and every adopter silently stops converging. The spec is the only guard, and it is named in a comment at the site. The slow consumer is proved through the transport rather than a seam: an ACCESS EXCLUSIVE lock on bus_messages stalls the resolver's spill SELECT for exactly as long as the spec holds it, and the listener is shown still draining and dropping while it does. The dropped connection is a pg_terminate_backend matched on the carrier's own application name. Neither Dropped nor IsConnected is on messaging.Broadcaster, and a spec asserts that over the interface type. Both are facts about a frontend; the conditions a scheduler acts on are facts about a worker, and no consumer holding the interface can read one as the other. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
17a1fa312e |
feat(distributed): carry fan-out on PostgreSQL LISTEN/NOTIFY
Distributed mode needs an operator to run a NATS cluster. This adds the carrier that replaces its fan-out half, so a deployment eventually needs PostgreSQL and its own HTTP listener and nothing else. pgbus holds one PostgreSQL session per replica, pinned for the life of the process because LISTEN registrations belong to one backend session and a pooled handle would lose them on the next checkout. Publishes go out on the pool with pg_notify. Subjects map onto a channel by their first token, from a closed set of roots. A subject outside the set is refused at publish AND at subscribe rather than mapped to a channel of its own: a channel name is capped at 63 bytes, and one LISTEN per job id would be unbounded. Refused rather than dropped, because a subject that goes nowhere and reports nothing is the class of defect this work exists to remove. PostgreSQL refuses a notify payload of 8000 bytes or more, and several subjects on this bus exceed that in normal operation: a job result carries a whole LLM output, a gallery progress event carries one entry per node. Those are written to a row and the notification carries the id. What is measured against the cap is the ENCODED notification, not the caller's payload, because the subject and the envelope travel too. The filter grammar is not respelled here. Subscribe asks messaging.ValidFilter and delivery asks messaging.SubjectMatches, which makes this the first production caller of a matcher that had only test doubles. New refuses a DSN that names a different database from the pool: that pairing publishes successfully, delivers nothing, on every replica, and reports no error anywhere. Nothing publishes on it and nothing subscribes yet. The construction is wired anyway, because the DSN has exactly one legitimate source and a setting that decides whether any broadcast is delivered should not be invented by whichever call site is migrated first. Delivery is at-most-once, like NATS core. Nothing downstream may read a message it did not receive as evidence about a node: a carrier that cannot deliver is not a worker that is gone. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
6de095b222 |
feat(distributed): take the backend worker off NATS entirely
A local-ai worker no longer opens a bus connection. connectNATS and its
spec are gone; Run registers once, starts its tunnel, arms /readyz on that
tunnel, and heartbeats. The worker's bus credential flags (--nats-jwt,
--nats-user-seed, --nats-require-auth, the three TLS flags) and
Config.NatsAuthRequired go with it. --nats-url stays, accepted and
ignored, so an existing worker command line still parses.
/readyz was the thing most likely to wedge a tunnel-only worker: it
required a live NATS link, so a worker with no bus would have reported
itself unready forever. nodes.NATSReadiness becomes nodes.TunnelReadiness
over a local interface{ Connected() bool }, and worker.Tunnel gains
Connected(), backed by a mutex-guarded session field the loop publishes
and clears. A closed-but-not-yet-cleared session reads as disconnected:
the loop waits for every in-flight stream before it clears the field, and
the probe must answer not-ready through that wait.
The heartbeat gate is DELETED rather than re-pointed at the tunnel. The
heartbeat is the worker's own answer that its process is alive; whether
the frontend can reach it is a separate fact the frontend already holds
and ages against LOCALAI_WORKER_RECONNECT_GRACE. Withholding the
heartbeat would report an unreachable worker as an absent one on the one
path with no grace, where the health monitor marks it offline and its
pending backend ops are deleted behind it. heartbeatLoop is given no view
of the tunnel, so a gate cannot be added back without changing its
signature.
Removing the NATS credential manager from this path also removes a defect
it carried: its refresh loop re-registered on a timer to renew a JWT, and
Register CLEARS a node's NodeModel rows. Any backend worker running on
frontend-minted credentials had its replica rows deleted roughly every
18 hours.
Of core/cli/workerregistry, everything survives. The manager is still
used in full by core/cli/agent_worker.go, which still needs NATS: Acquire,
Provider, RefreshLoop, HasCredentials and TunnelToken are all untouched.
The backend worker simply calls RegisterFullWithRetry directly now.
WorkerPermissions is documented as serving agent nodes, and its non-agent
branch narrowed to _INBOX.> on both sides. It is NOT deleted: NATS reads
an empty allow list as no restriction, so returning nil would upgrade
every JWT the frontend still mints for a backend node from its own inbox
to the whole account.
Agent workers keep the bus everywhere: their CLI flags, their
subscriptions, the agent branch of WorkerPermissions, and the compose
service with its LOCALAI_NATS_URL and depends_on: nats.
Also corrected two flags the Nodes page advertised that do not exist
(--distributed-nats, --distributed-db), and a log line plus several
comments that still named a bus the code no longer touches.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Ettore Di Giacinto
|
19280c4508 |
fix(distributed): let absence be decided by something, at all three call sites
Removing "Presence: clusterRegistry" from the options literal in initDistributed left all seven suites and tests/e2e/distributed green. The predicate was right and its input was silently nil, which returns the deployment to absence being decided by nothing, with no log line and no failing request. That is the fourth finding of this exact shape in this phase. The two assignments move out of a twenty-field literal into distributedSchedulerOptions, a named function a unit spec can reach. Deleting either is now red. The health monitor takes its presence reader and grace as a required positional pair instead, so deleting those does not compile at all. requireAbsenceWiring then refuses to start a distributed frontend whose scheduler or health monitor has no source of absence, because refusing to boot is the only symptom either failure has. With a fresh heartbeat and a permanently gone tunnel there was no reaper at all. A heartbeat says the worker's supervisor is alive; it says nothing about whether anything here can reach that worker's backends, because those are reached over the tunnel. A proxy that stops upgrading WebSockets, a rotated registration credential or a reconnect loop longer than the grace left a node listed healthy forever while every request for a model already loaded on it failed "no route to that worker", and every reaper keyed on the heartbeat. The health monitor now reads presence from the same place and against the same window as the scheduler and demotes such a node. That also ends the 15s re-promotion: the demotion arm returns before the recovery arm, so the scheduler's demotion is no longer undone on the next tick, and recovery needs the tunnel back rather than just the heartbeat. The demotion is status-only. MarkOffline would DELETE the node's rows, and deleting rows on a presence read would give any future defect in that read the widest blast radius in the system for nothing the demotion does not already deliver. LRU eviction is the third path that commits work to a node, and it read only the stored status. A node full enough to be an eviction target is exactly the node the VRAM and idle selectors never offer, so pickReachableNode structurally cannot cover it. It now runs its chosen node through the same nodeMayTakeWork predicate, demotes it and evicts again rather than handing back an install that cannot land. Presence is read after the transaction and not inside it: reading it inside would hold a FOR UPDATE lock across a query needing a second pooled connection, which is how concurrent evictions deadlock a pool. Also: a router built with a presence reader and no grace now has its documented default pinned by a spec rather than only claimed by a comment; ageDeparture asserts RowsAffected, since an UPDATE matching nothing succeeds and the inside-the-grace spec returned the same verdict either way; the scheduler comment that still described the bus is corrected; the docs stop conflating heartbeat recovery with tunnel recovery and name the third reader; and an overlong rewrapped line in membership.go is folded. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
94b248666a |
feat(distributed): read worker absence from the database, not from a bus timeout
The scheduler decided whether a worker had gone away from nats.ErrNoResponders: one frontend's observation that nobody answered IT within a request budget. Two replicas asking in the same moment could disagree and demote each other's workers, and a worker re-homing its tunnel between replicas looked identical to one that had died. SmartRouter now reads cluster.Presence instead. Only PresenceGone -- no live replica holds the tunnel AND the departure has outlived the reconnect grace -- excludes a node from placement, and it is a fact every replica reads identically from the database. PresenceReconnecting, PresenceUnknown and a failed presence query are all non-verdicts and place work as normal: excluding on a database hiccup would cost the fleet its capacity for a reason that has nothing to do with any worker. nodeAnswersOnBus is deleted. It excluded on a sentinel no control RPC can produce, so it decided nothing while PingNode cost a relayed round trip per scheduling decision to feed it. PingNode goes with it, from the adapter and from NodeCommandSender. isRequestTimeout drops nats.ErrTimeout: every verb this adapter sends now travels over the worker's tunnel. The predicate is named nodeMayTakeWork rather than nodeHasRoute. "Route" is ErrWorkerUnroutable in this package, the condition nobody may act on; PresenceGone is the one a scheduler may. Spelling them the same way is the collapse this work exists to prevent. Also folds in ReapStale's return rename: it counts connection rows CLEARED, never rows deleted, and reading it as a delete count would make a worker that is re-dialling right now look forgotten. The spec pinning that a message merely quoting "nats: timeout" is not a timeout was scripting a SUCCESSFUL reply carrying the phrase, which comes back with a nil error and never reaches the classifier. Restoring the string match left it green. It now scripts a 5xx whose body carries the phrase, and asserts that the phrase reaches the classifier as a precondition. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |