* fix(models): fallback to application config default context size (#12202)
Honor appConfig.ContextSize in /v1/models/capabilities when model context_size is unset.
* docs(models): explain context size fallback
Describe the application default used by capability discovery and
preserve the distinction between total context and per-request limits.
Assisted-by: Codex:GPT-6
* fix(models): apply the default context size only when context_size is unset
The request path applies the application default context size only
when a model leaves context_size unset. An explicit 0 or -1 falls
through to the backend fallback. The capabilities endpoint now does
the same, so it reports the value the backend uses.
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
fix(router): re-seed the knn corpus index when the vector store comes back empty
The corpus manager records a store as synced by file fingerprint and
embedding fingerprint. The local-store backend behind it is an in-memory
gRPC process the model loader may evict (active-backend cap, memory
pressure) or the idle watchdog may kill, and relaunch on the next
request — empty. The file is unchanged, so EnsureLoaded returned early
and the router went blind: every probe fell back with similarity 0 while
corpus/stats kept reporting the full count.
Measured on a production router (LOCALAI_MAX_ACTIVE_BACKENDS=6, four
resident models + two router stores): loading any further backend
evicted a store, and the idle watchdog killed both after 15 minutes;
/stores/find returned 0 hits against a 100-line corpus file whose stored
vectors matched fresh embeddings with cosine 1.000.
Two parts, because the knn classifier is built once and cached
(GetOrBuildClassifier), so the sync at build time is otherwise the only
one for the process lifetime:
- corpus.Manager remembers one vector it inserted (probe) and, on the
synced path, asks the live index for it. A miss means the index was
relaunched — fall through and re-seed from the file (no re-embedding).
- The router middleware wraps the knn classifier's store so every
lookup runs EnsureLoaded first; the loader gets the raw store, so its
probe never re-enters the wrapper. A sync error fails the lookup
closed, like the build-time load.
Specs: corpus package (relaunched empty store is re-seeded under an
unchanged file), middleware (relaunched index behind the cached
classifier is re-seeded instead of falling back; the spec is red without
the wrapper). The test fake now answers Search for inserted vectors.
Folds in the maintainer's follow-up (router-corpus-reseed-after-store-relaunch): reviewed and accepted.
Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de>
* feat: return 429 when backends are saturated
When backends are at capacity (per-model max_concurrent or the
process-wide --max-concurrent-backend-requests ceiling), the response
was 503. The OpenAI SDK, litellm, and most agent harnesses key on 429
for rate-limit backoff and treat 503 as a hard error.
Both saturation paths now return 429 with the existing Retry-After
header and type: "rate_limit_error" in the JSON body. The per-model
admission middleware keeps admission_rejected as the code field so
existing alerts that match on it still fire.
Non-saturation 503s are unchanged: model cold-loading (with progress
body), model-load failure cooldown, PII detector fail-closed, and
classifier unavailable. These mean "not ready" rather than "busy".
Assisted-by: AGENT:regolo/glm5.2 [TOOL]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix: return 503 when scheduler has no available nodes
When the scheduler cannot find any healthy node to serve a model —
all nodes are full and eviction cannot free a slot, or a node_selector
excludes every candidate — the error fell through to 500. A 500 tells
clients something is broken when the condition is transient and
retryable.
The router now wraps these errors with a new ErrNoAvailableNodes
sentinel. The HTTP error handler maps it to 503 via applyNoAvailableNodes,
following the same pattern as applyBackendAdmission (429). Unrelated
scheduler errors (DB timeouts, registry lookups) still return 500.
Three return sites are wrapped:
- resolveSelectorCandidates: selector matches zero healthy nodes
- scheduleNewModel eviction-busy: all models have in-flight requests
- scheduleNewModel eviction-failed: eviction itself errored
The existing scheduleAndLoad wrapper ("no available nodes: %w") preserves
the sentinel through the chain via errors.Is, as does ModelRouterAdapter.
Assisted-by: AGENT:regolo/glm5.2 [TOOL]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(http): use Ginkgo for admission tests
Replace forbidden testing.T calls with Ginkgo and Gomega so the lint
check accepts the admission handler tests.
Assisted-by: Codex:GPT-6 forbidigo
* fix(middleware): show the recorded status for admission rejections
The admission audit row now records 429, but the Middleware page still
printed a hard-coded 503. Read the status from the event, and update
the two package comments that still said 503.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Hardcoding size/size_vram as 0 made Ollama clients treat loaded models as
free. Prefer ModelFileName+ModelPath Stat when available, and omit size_vram
(and size) when the value is unknown instead of emitting literal zeros.
Resolve each listed model by its stored ID so tagged variants use their
own weights.
Fixes#11969
Signed-off-by: lei_lei <imleilei123@gmail.com>
LocalAGI 7e0947d added allowed_tools/excluded_tools and the
required_tool_before_finish gate. Single-node agents get them through
LocalAGI's runtime, but the distributed executor drives cogito directly
and its static config meta did not list the fields, so the agent form
hid them and the worker ignored them.
The distributed config now parses the tool lists from a JSON array or a
comma/newline separated string, and the meta entries match LocalAGI's.
The executor filters the knowledge base, skill and MCP tools (MCP via
cogito.WithMCPToolFilter) before the model sees them, and re-prompts the
model when it answers before the required tool returned "ok": true, up
to the configured number of reminders.
LocalAGI keeps its filter and gate helpers unexported, so a minimal copy
lives in core/services/agents/toolpolicy.go. A spec compares the meta
entries with LocalAGI's to catch drift.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Pick up the LocalAGI PRs merged after f2a2af4:
- per-collection embedding and reranker models, locked per collection
so one agent's upload or rerank no longer stalls the others (#499)
- required_tool_before_finish: a tool the agent must call successfully
before it may answer (#495)
- allowed_tools / excluded_tools per agent, applied to MCP tools too
(#480)
Document the new agent settings. They show up in the single-node agent
form, which reads LocalAGI's config metadata; distributed mode keeps its
own field list and does not offer them yet.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
* feat(gallery): publish signed OCI fallbacks
Publish both official gallery indexes with their local base configs so
an outage of the HTTP and GitHub sources can fall back to Quay.
Keep artifact signing policies separate from backend image policies,
and expose each moving gallery tag only after its digest is signed.
Assisted-by: Codex:gpt-6
* fix(gallery): confine packaged files to selected roots
Use directory-scoped file access to reject symlink escapes during gallery packaging. Create private bundle files for the publishing runner.
Assisted-by: Codex:GPT-6
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
* feat(system): report per-model DRM VRAM
Expose optional resident device memory for local backend process trees.
Deduplicate DRM clients and omit unsupported or incomplete readings.
Document accounting limits and preserve a measured zero in JSON.
Closes#11970.
Assisted-by: Codex:gpt-6
* fix(system): document trusted procfs reads
Scope G304 annotations to paths built from the fixed procfs root,
integer process IDs, and kernel directory entries. These reads accept
no user-controlled path components.
Assisted-by: Codex:GPT-6 gosec
---------
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Keep each message and reasoning item at its announced output index.
Include the answer in completed responses with reasoning or fallback
function calls, and retain reasoning supplied through backend deltas.
Add regression coverage for stream indices, final output, plain text,
and automatic tool parsing.
Assisted-by: Codex:GPT-6
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Partial JSON parsing heals a name-only chunk into a tool call. The
stream emits that call with empty arguments and skips later chunks.
Require complete JSON before emitting terminal tool-call events.
Preserve complete calls before an unfinished trailing call, and count
only actual tool calls. Add split-chunk regression tests and docs.
Refs #11635. The non-streaming report remains unconfirmed.
Assisted-by: Codex:GPT-6
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The legacy NVIDIA device reservation requests utility without compute.
Docker derives driver capabilities from that list, leaving CUDA libraries
unavailable even when monitoring works.
Include compute in the legacy example and clarify the matching docs.
Assisted-by: Codex:GPT-6
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
Older Go linkers stamp pure-Go hosts with SDK metadata that disables
modern Metal APIs. Select Go 1.27 for Darwin builds and document the
backend rebuild requirement.
Assisted-by: Codex:gpt-6
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
The link text still said chatbot-ui, but it now pointed at the examples
repository root. Link the configurations directory, which holds the
example model config files, and describe it as such.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Add IQ4_XS and Q4_K_M GGUF builds with a BF16 vision projector.
Pin verified artifacts and document installation and variant selection.
Assisted-by: Codex:gpt-6
Add Q4_K_M and Q8_0 builds with the F16 vision projector and install docs.
Pin artifact revisions and verify SHA256 against HF LFS metadata and HTTP
headers.
Assisted-by: Codex:gpt-6
Register the new quantization, mock fixture, and protocol inventory tests through package-consistent Ginkgo suites. Keep table case names in Ginkgo diagnostics and label the protocol inventory for distributed CI selection.
Refresh the current-master fixture baseline using the final test structure.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
The cluster target identifies feature and machine-auth scenarios while
retaining one shared binary build. CI requires binaries, so neither
scenario can skip silently.
Record the current-master fixture baseline and measured local suite
budgets. Keep new conformance tests within the branch lint policy.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Keep pending agent workers alive with authenticated heartbeats while preserving approval as the tunnel and execution boundary. Exercise the real binary credential handoff through an authenticated inference and verify that credential remains non-admin.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Worker setup now uses a focused drawer without interrupting fleet
monitoring. Both generated worker commands retain their NATS settings.
Log views preserve their launch context and show one-based replica
labels. Bulk controls disclose selections outside the current view.
Assisted-by: Codex:gpt-5 Playwright ESLint
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
The binary conformance design now covers a WebUI-authenticated deployment. It distinguishes browser sessions, shared registration credentials, and per-node tunnel credentials, including rejection and approval paths.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
The existing process tests prove chat routing and model transfer, but file-bearing backend operations rely on in-process coverage. Define the binary deployment shape, feature inventory, fixture contract, master baseline, and CI repair boundary before implementation.
Assisted-by: Codex:GPT-5 [apply_patch] [exec_command]
Resolve allowed directories before comparing them with resolved files.
Otherwise staging rejects valid files under macOS temporary paths.
Cover aliased roots, sibling paths, and symlinks escaping the root.
Assisted-by: Codex:gpt-6
The claim queue's attempts counter grew without bound and nothing read it. At
the default two-second poll a permanently undispatchable row cost about 43000
UPDATEs a day, and it cost more than writes: rows are claimed oldest first, so
the oldest stuck row was re-claimed ahead of every newer one on every tick and
held a dispatch slot while it failed. One poison row starved the queue behind
it.
No dead letter, and that is the decision rather than the omission. Read
settleClaim: the only outcome that releases a claim is one where NOTHING was
learned about the work. No agent worker was connected, the tunnel broke, a peer
could not be reached, the stream was refused before the request body left this
replica. Not one of those is a worker saying it ran the job and it failed, and
an attempt ceiling would turn "the fleet was away long enough" into a job
failure nobody reported, which is the collapse this whole design exists to
prevent pointed at work instead of at nodes. The one verdict available here,
that no build of any worker serves this kind, is already settled as an answer.
So the retry stays unbounded and the RATE does not. Each release stamps the row
with the earliest it may be claimed again, doubling from two seconds to a cap
of sixty, computed in the release statement from the row's own attempts count
and stamped on the DATABASE clock, because that is the clock competing replicas
order the queue on. Queued work becomes claimable again within one cap of the
fleet returning, and a stuck row no longer holds the head of the queue. A claim
released by the reap carries no delay at all: that work was never handed to
anyone, so there is nothing to back off from.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The backend port allocator allocated from its own bookkeeping alone. That
bookkeeping records what this worker did with a port, and the collision it
cannot see is with something this worker never did: the default base port is
50051, inside Linux's default ephemeral range of 32768 to 60999, so the kernel
hands ports in this range to outbound connections and to anything that binds
port 0. A backend handed one of those dies on bind, and the frontend sees a
backend that will not start.
Every candidate is now probed by binding the exact address the backend will
listen on, in all four allocation branches: the key's own port, the free pool,
a grown port and a stolen one. Probing the free pool matters as much as
probing a grown port, because a port this worker released is exactly as
available to the kernel as one it never used.
A candidate that fails the probe is quarantined rather than blacklisted, since
whatever holds it is usually an ephemeral connection that gives it back, and
its affinity claim is dropped so an unbindable port does not stay reserved for
the key that last held it. Exhaustion now says how many candidates were
skipped, which is what tells an operator "something else is in my range" from
"my range is too narrow".
This does not remove the race and cannot: between the probe and the child's
bind the kernel can still give the port away. It removes the far larger window
in which the allocator hands out a port the kernel gave away minutes ago,
which was the whole of the observed one-in-three harness flake. The e2e
harness comment that recorded the missing check is corrected, and the docs say
how to move the range out of the ephemeral one entirely.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Skills and RAG collections had no cross-replica invalidation, and the two
builders that would have published one were deleted earlier in this branch
because nothing called them. Wiring one now would be wrong, not merely late.
Both features are derived entirely from the frontend's own state directory.
A skills.Service indexes <state dir>/skills, a collections backend enumerates
<state dir>/collections and holds one handle per collection it found there,
and no replica reads or writes another replica's copy of either. In
distributed mode PostgreSQL carries a skill's NAME and description in
skills_metadata, and nothing else: Get, Search, Export and the resource verbs
all read local files. So a peer told to drop a cache entry would rebuild it
from a directory that does not hold the change. For a postgres-engine
collection it would be worse than a no-op, since re-deriving one on a replica
with no local index file yields a collection that answers with an empty file
list against a populated vector store. What is missing is shared storage, not
a broadcast.
Recorded rather than left silent: the two cache fields say why nothing
invalidates them, a distributed frontend logs the limitation once at startup,
and the docs name the two deployments that avoid it. The new spec pins the
premise, so a change that moved either directory onto storage every replica
mounts reddens and the decision gets taken again.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
GET /api/cluster/peer authenticated with the deployment's shared
registration token and took the dialling replica's id from ?id= on trust.
Every worker holds that token, so anything holding it could open a peer
link as any replica: relay through it to every worker tunnel that replica
owns, displace a real replica's inbound link by declaring its id, and
point the roughly 31 GiB per-session receive window at one replica.
Validating the id against the instances table does not fix this, because
the attack declares a real replica's id. So the route now checks two
credentials and needs both. The shared token still says the dialler
belongs to this deployment; a new per-replica credential says which
replica it is.
The credential follows the per-node worker credential rather than
inventing a second mechanism: crypto/rand.Text, stored only as a hex
SHA-256, compared in constant time, with no fallback to the shared token.
It differs in the stronger direction. A worker's credential is minted by
the frontend and handed over once; a replica writes its own instances
row, so it mints its own secret, publishes only the hash in the same
statement that publishes its address, and never sends the plaintext
anywhere but the peer dial.
A peer that presents no credential is refused, not waved through. An old
replica and an attacker holding the shared token send the same request,
so accepting the first accepts the second; there is no safe downgrade
here, only a quiet one. The refusal is made loud instead, on both sides,
naming the upgrade rather than the network. On the documented
frontend-first order a new replica still dials an old one; an old replica
cannot dial a new one, which costs relayed requests that land on a
not-yet-restarted replica and surfaces as no route, never as absence.
A rejected peer gets its own sentinel, ErrPeerRejected, whose unwrap
chain carries ErrPeerUnreachable as well and no absence sentinel at all.
Keeping the older sentinel means no existing consumer changes behaviour;
the cause stays out of the chain, so absence cannot escape through it and
nothing can read an authorization failure as a worker that went away.
One consequence beyond the fix: a replica with no advertised address has
no instances row, so it now cannot dial out either. It was already
unreachable inward. The startup error and the docs say so.
Registry.Register, NewMembership, NewPeerPool, PeerHandler and
RegisterClusterRoutes all gained required arguments, so the identity
cannot be dropped without a compile failure.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Distributed mode has not dialled a message broker since the control plane
moved onto the workers' own outward tunnels and every fan-out family moved
onto PostgreSQL LISTEN/NOTIFY. What was left was the dependency itself, and
the code that existed only to feed it.
Dropped from go.mod: nats-io/jwt/v2, nats-io/nats.go, nats-io/nkeys,
nats-io/nuid and testcontainers-go/modules/nats, along with the fourteen
indirect requires that only the NATS testcontainer pulled in. go.sum carries
no nats line either, so the removal is not the partial kind where the require
goes and the checksum stays.
Deleted with them: pkg/natsauth in full, the broker client's remaining
options and TLS files, the per-node JWT minting on both the register and the
approve path, and the natsauth.Config parameter threaded through the node
routes. The credential manager is renamed and stripped rather than deleted,
because it still holds the tunnel token that every re-registration rotates.
The bus flags stay accepted and ignored, and are now hidden, on every command
that had them, so an existing unit file, compose file or Helm values file
still starts on the day of the upgrade. What is not kept is the validation
that REQUIRED one: a distributed frontend started with no bus URL is no
longer fatal. The TLS paths lose type:"existingfile" deliberately, so a
certificate deleted along with the broker cannot fail a startup.
One operator-visible behaviour change: --nats-require-auth no longer makes an
agent worker wait through admin approval. Ask for that wait with
--distributed-require-auth, which already implied it. It is documented in the
migration section and pinned from both sides.
A deployment now needs PostgreSQL and the frontends' own HTTP listener, and
nothing else.
coverage-baseline.txt moves from 54.2 to 62.0.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Every carrier had already moved and no process opened a bus connection, but
the surface an operator reads still described a deployment with a broker in
it: a compose service, a 220-line credential-generation script, two CI steps
pulling a container nothing started, two flag tables offering --nats-url, an
architecture diagram with a NATS box wired to the workers, a join-command
generator in the Nodes page that emitted --nats-url for agent workers, and a
test suite that stood a NATS server up for specs that no longer used it.
That is the one way this programme could still fail invisibly. Every test
passes, every binary works, and every production deployment goes on running
and paying for infrastructure that carries nothing.
Nothing in this repository starts a NATS server any more. The compose file is
four services, the docs say to shut the broker down and what to keep, and the
e2e suite runs on one PostgreSQL container.
The three LOCALAI_NATS_*_TIMEOUT env vars are KEPT, and are now documented
twice as being kept. They were never broker settings: each names a control-RPC
budget the frontend applies to a worker, still read and still enforced. They
carry the prefix only because they arrived with the bus, and renaming them
would break every existing deployment for cosmetics.
The agent worker's join command was the last surface still emitting the flag,
two tasks after the agent worker stopped dialling. The Playwright spec that
covered it asserted the opposite of what is now true, so it is inverted rather
than deleted, and it reads the rendered command string rather than the
component's variables: the variables are what the fix removes, so a spec
reading them would have stopped compiling instead of failing, and a compile
error is not evidence about what an operator is shown.
nats_jwt_test.go and its helpers are deleted. They pinned a real server
ENFORCING the minted permissions. The CONTENT of those allow lists is still
pinned, untouched, by pkg/natsauth's own suites, including the spec that
refuses to let the agent lists go empty, since an empty allow list in NATS
means unrestricted. The enforcement half is retired rather than moved:
enforcement is a property of a connection, and nothing opens one.
The suite's own NATS container goes with them, which the brief left for the
next task. Removing the pre-pull while BeforeSuite still ran the image would
have defeated the step rather than cleaned it up, and this change removes the
last reader of TestInfra.NC. agent_native_executor_test.go and
mcp_ci_job_test.go are moved onto infra.Bus() instead of deleted: they were
the last two specs building a bridge and a dispatcher on a client nobody uses,
which is exactly the drift TestInfra.Bus's own comment warns about.
cluster.Options.NatsURL is now fed a deliberately dead address rather than a
live container's. Frontends and agent workers still receive LOCALAI_NATS_URL,
because that is the coverage for the promise that an existing command line
still starts; sourcing it from a running server would have let a regression
that actually dialled it pass. The control in cluster_control_test.go keeps
its assertion and loses its explanation, which claimed the deployment had a
bus and no longer could.
One latent spec race surfaced and is fixed: the background-run spec waited for
a COUNT of events and then read a snapshot for the terminal status, which is
the last event of a run and therefore always arrives after the count is met.
Its immediate twin had already been fixed this way. Nothing in production
changed.
pkg/natsauth keeps its files. It is reachable from production only through the
natsauth.Config parameter thread, and that thread is the next task's.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Task 4 gave agent workers tunnels and deliberately left the NodeType skip in
HealthMonitor.tunnelDeparted, with a spec asserting that an agent node whose
presence reader answers PresenceGone is NOT marked unhealthy. That spec was
scaffolding. It was true while an agent worker took its jobs and its verbs over
the message bus: a departure row for one said nothing about whether it could
work, and an early bug in the new tunnel client could otherwise have demoted a
fleet of healthy agent workers.
There is no bus. An agent worker is reachable through its tunnel and through
nothing else, so a departed agent tunnel means exactly what a departed backend
tunnel means: no live replica holds it, the departure has outlived the reconnect
grace, and that is a routing fact the scheduler and a reaper may act on. The
skip would now hide the only symptom an unreachable agent worker has. This is
the deliberate removal Task 4's M6 predicted, and task-4-report.md is where that
mutation already stands recorded red against the spec this commit deletes.
The skip existed at ONE site. router_liveness.go has none: its candidates come
from queries that already filter node_type = 'backend'. The two skips in
managers_distributed.go stay, because an agent worker still runs no backend
processes, so it has no backend to list and no backend op to apply.
Two node types can depart now, which is why the second half exists. Before this,
one type could depart and every per-node cache a departure left stale was
dropped from wherever its owner happened to notice, so a reader could not tell
which caches a demotion invalidated by reading the demotion path. Departure gets
ONE notification point. DepartureNotifier is edge triggered, because the monitor
runs on a ticker and a departed node stays departed; its subscribers are NAMED,
because what has to be caught is a forgotten cache and a count can say only that
one of four is missing; and NewHealthMonitor takes it as a required positional
argument, so a caller that does not pass one fails to compile.
Four caches subscribe: prefix-cache affinity in every model, probe freshness at
every address, in-flight staging operations, and the per-node breakdown of every
open gallery operation. The prefix-cache one is registered only when
prefix-cache routing is enabled, so --distributed-prefix-cache=false stays a
true no-op. The notification carries the node's name as well as its id, because
the staging tracker keys on the name and the other two key on the id, and a
subscriber should not have to read the registry from inside an eviction hook.
A departure notification is an act on absence, so it fires only on the routing
fact. A tunnel lost inside the grace, a worker that never dialled, a presence
query that failed and a stale heartbeat all announce nothing, asserted per node
type. The stale-heartbeat branch is excluded on purpose: it already marks the
node offline, which deletes its rows and runs the registry's replica-removed
hooks, so firing there too would double-evict and make the notification mean two
different things at its subscribers.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
agent.<name>.cancel was the last family on a message bus, and the only
reason an agent worker dialled one. Its subscriber is the worker running
the execution, and a worker has no database, so the family could not move
to the PostgreSQL fan-out carrier: a cancel published there would reach no
worker while reporting that it had been sent.
It is a control verb now. An agent worker mounts workerctl.PathAgentCancel
on the loopback control plane behind its tunnel and applies the cancel to
the same registry the executor registers a run on. The frontend issues it
through nodes.AgentControlClient.CancelAgentRun.
That call is a FAN-OUT and not a pick, because nothing records which worker
holds a given execution: the claim row names the claiming replica, and it
is deleted when the run ends. Every agent worker a live replica can reach
is asked over its own tunnel, relayed by the peer mesh when a peer holds
it, and each worker answers only for itself.
The answers stay apart, which is why this family was held back. A cancel a
worker made is nil. A cancel some worker could not be asked is
ErrAgentCancelUndelivered, which is neither a refusal nor a missing run. A
cancel every reachable worker declined to own is ErrAgentRunNotOnAnyWorker.
A deployment with no agent worker is ErrNoAgentWorker. Neither new sentinel
wraps ErrWorkerUnroutable and neither is a worker answer, so nothing is
reaped, demoted or evicted because of a cancel.
A worker in the ABSENT CONNECTION condition, one whose tunnel was lost
inside the reconnect grace, counts as undelivered. It is not retried in the
call and not queued: a retry would spend a budget the caller did not
choose, and a queue would need durable state whose only consumer is a run
whose control stream went with the tunnel. A worker whose departure has
outlived the grace is the one routing fact a caller may act on and is
excluded, or a single retired agent node would make every cancel
undelivered for ever.
The fan-out reads a different node set from the pick. A draining worker
takes no new work but is still finishing what it holds, so it is offered
the cancel; a pending one is refused by the tunnel route on every dial and
is not.
With that, nothing in LocalAI connects to NATS. The agent worker's dial,
its credential ladder and its refresh loop are gone, and so is the
frontend's cancel carrier. LOCALAI_NATS_URL is accepted and ignored
everywhere, and distributed mode no longer requires it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Nothing in the tree publishes, subscribes, queue-subscribes or requests through
the MessagingClient interface any more, so it is deleted rather than shrunk to
Broadcaster: two exported names for one method set in one package is an
invitation for the next author to pick whichever the surrounding file already
imported.
$ grep -rn 'messaging\.MessagingClient' --include='*.go' .
core/services/syncstate/syncstate.go:54: // It is messaging.Broadcaster rather than messaging.MessagingClient because
(one hit, a comment; no live referent. The naive grep in the plan also matches
prose and the local test type names fakeMessagingClient and
countingMessagingClient, so it can never be empty.)
*messaging.Client is shrunk to exactly Broadcaster plus its own lifecycle.
QueueSubscribe, QueueSubscribeReply, SubscribeReply, Request, Conn and the
package helpers QueueSubscribeJSON and RequestJSON go with it; none had a
production caller. Deleting the methods rather than only the call sites is what
makes putting a family back on this carrier a build error instead of a line that
compiles, publishes successfully, and is delivered onto a carrier the deployment
is being taken off. Conn is in that list because while it existed every other
name was one c.Conn().X() away; the flush-and-verdict that its real consumers
needed is now ConfirmRoundTrip, which keeps the NATS JWT permission specs armed.
The client, its options and its TLS plumbing are NOT deleted, and both processes
stay on the bus. agent.<name>.cancel is the one fan-out family that could not
move: its only subscriber is the agent worker, which has no database and cannot
join the PostgreSQL carrier at all, so a cancel published there would reach no
worker and be reported as sent. The frontend passes the client to
newFanoutBridges as its cancelCarrier and the worker subscribes on it, so
--nats-url stays required on agent-worker. Both go with the tunnel cancel verb.
The struct field is renamed Nats -> CancelCarrier to say what it is for, and
agentpool loses the messaging.Publisher it held only to be non-nil: it never
published on it, and it was gating whether a frontend runs agents distributed or
in an in-process pool. Retiring the bus would have flipped every replica back to
the in-process pool silently. The gate now reads the agent store, which is the
dependency the mode actually requires.
Also deletes four subject builders with no production publisher
(SubjectFineTuneProgress, SubjectFineTuneCancel, SubjectCacheInvalidateSkills,
SubjectCacheInvalidateCollection), the queue and request/reply halves of the
shared test double, and the e2e specs that were their only callers. Every
surviving subject is now pinned to its exact literal, because a subject is a
cross-version wire format and a rename that looks internal stops half a fleet
hearing the other half.
Docs: distributed-mode.md and cli-reference.md no longer claim NATS carries the
agent-worker job subjects, the frontend's cross-replica events, or an agent
worker's real work.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Gallery progress and cancel, the operation cache's start and end, the model
and backend cache invalidations, staging progress, and the prefix cache's
observations and invalidations now travel on the LISTEN/NOTIFY carrier. No
subject is published or subscribed on messaging.Client anywhere in the tree,
which is what makes retiring that package a deletion rather than a migration:
$ grep -rn 'natsClient\.Publish\|nats\.Publish\|\.Nats\.Publish\|QueueSubscribe\|SubscribeReply\|\.Request(' \
--include='*.go' core/ pkg/ | grep -v _test \
| grep -v 'c\.Request()\|ctx\.Request()\|Request()\.Context' \
| grep -v 'core/services/testutil/fakebus.go'
core/services/messaging/client.go:168,170,172,227,234,236,250,252,254,268,269,287
core/services/messaging/interfaces.go:21,22,23
Every remaining hit is inside core/services/messaging itself. The production
reads of the NATS client are now three, all of them the documented agent-worker
exception: Close on shutdown, the agent pool's publisher, and the agent-cancel
carrier passed to newFanoutBridges.
Prefix-cache observations publish like every other family rather than through a
method that refuses a message too large for a notification. The plan proposed
such a refusal on the reasoning that a long prompt makes a chain of thousands of
entries; ExtractChain caps a chain at Config.MaxDepth blocks, MaxDepth is a
constant with no operator knob, and the chain reaching Sync.Observe has one
source, the router's own extraction hook. A worst-case observation is a few
kilobytes against an 8000-byte cap, so the hot-path spill the refusal was
designed to avoid cannot occur, and shipping it would have added the programme's
only deliberate message drop to guard a condition that cannot arise. pgbus gains
FitsInline instead, a predicate that shares one size decision with Publish and
decides nothing, and core/application refuses at startup to wire a prefix cache
whose configured depth would put every observation over the cap.
The carrier choice is no longer stated at four sites. StagingTracker.SetPublisher
and SubscribeBroadcasts become one SetBroadcaster, so a tracker that publishes
where its peers are not listening cannot be spelled; prefixcache.Sync gains
SubscribeBroadcasts, which reads the carrier it publishes on; and the gallery
service and the operation cache are wired by methods on DistributedServices that
name no carrier at all, so the NATS client beside it cannot be handed over.
OpCache.SetMessagingClient and GalleryService.SetNATSClient are renamed to
SetBroadcaster so a missed call site fails to compile.
Two pre-existing defects that the two-real-carrier specs surfaced are fixed. A
progress tick published before a cancel and delivered after it cleared Cancelled
and left the operation reading as still running on that replica; mergeStatus now
drops a stale tick rather than merging it. GetStatus and GetAllStatus handed out
the stored OpStatus pointer while the broadcast subscribers mutated it in place,
so an /api/operations response could be marshalled mid-write; both now copy.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Five of the six families whose subscriber is an open HTTP response rather
than a process-lifetime cache now travel on pgbus: jobs.<id>.progress,
jobs.<id>.result, jobs.<id>.cancel, agent.<name>.events.<user> and
responses.<id>.cancel. Both ends of each move together, so there is no
state where a publisher is on one carrier and its subscriber on the other.
agent.<name>.cancel does NOT move, and the plan was wrong about why. Its
only subscriber in the tree is the agent worker, which has no database and
so cannot join the PostgreSQL carrier at all. Publishing that cancel on
pgbus would have lost every cancel of a worker-run agent while returning
nil, which reports a cancel that reached nobody as a cancel that was sent.
EventBridge now names its cancel carrier separately, a frontend replica
sets it to the carrier the worker reads, and it stays there until a cancel
rides the worker's tunnel like every other verb addressed to a worker.
The carrier drops at 256 rather than blocking, which is not safe on its own
for a result: a lost result has no successor message. It is not the only
path. The claiming replica persists the terminal line before it releases
the claim, and an open progress stream re-reads the job row once after
subscribing and then periodically, so a dropped terminal broadcast costs
promptness and never the answer.
Both per-request subscriptions close in a defer instead of on one return
path, and pgbus grows Subscribers() so the leak they would otherwise cause
can be asserted. It has no other symptom: only the first subscriber of a
channel issues a LISTEN, so a leaked filter just adds one closure per
notification for every stream the replica has ever served. Subscribe now
issues its LISTEN before it registers, which makes that count a readiness
signal rather than a figure to compare against itself.
Two rules that were stated at several sites and pinned at none are now one
each. The re-broadcaster is built beside the dispatcher and the bridge and
handed to the dispatch loop, so no line is left that can point it at a
carrier nobody subscribes to while every spec stays green. The set of
statuses a job never leaves is one exported set that the SSE bridge and the
store both read. The last hand-written subject filter in production code
became messaging.SubjectAgentEventsWildcard.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
syncstate.Config held one carrier field typed as the NATS client, so a
pgbus.Bus could not be handed to a SyncedMap at all: it satisfies
messaging.Broadcaster and not MessagingClient. The durable re-hydration
path built for the responses map therefore had a NATS-only consumer and
nothing in the build said so.
The field becomes Bus messaging.Broadcaster, SubscribeJSON moves to its
own file and relaxes its parameter to Broadcaster, and the four adopters
fan out over PostgreSQL LISTEN/NOTIFY: fine-tune jobs, quantization jobs,
agent tasks with their per-tenant children, and Open Responses metadata.
A new spec proves it on a real database, over two Bus instances on two
pinned listener connections: a Set and a Delete carry, a payload past the
8000-byte notification cap comes back byte identical through the spill
row, two families sharing one LISTEN channel stay separate, and a
terminated listener re-hydrates a row written while it was gone.
The five sites that each chose a carrier for an adopter are collapsed
into one DistributedServices.Broadcast() accessor. Five field reads were
five chances to leave one family on NATS with nothing failing, because
messaging.Client satisfies Broadcaster too. The accessor also refuses to
hand out a nil pgbus.Bus wrapped in a non-nil interface, which every
adopter would read as "broadcast" and dereference on the first Set.
SetTaskSyncNATS and SetJobSyncNATS are renamed to SetTaskSyncBus and
SetJobSyncBus so a missed wiring site fails to compile.
The response metadata table gains a retention of its own, defaulting to
24 hours. It inherited the Open Responses store TTL, which defaults to 0
meaning no expiration. Zero is defensible for a map that dies with the
process and is not for a table: the table grew for the life of the
deployment and a restarting replica re-hydrated every response the
cluster had ever created. A row that names its own expiry is still judged
on that column alone, and "this row is dead" now has one SQL spelling
that PurgeExpired deletes by and ListUnexpired is the negation of, so a
hydrate cannot resurrect what a sweep has already retired.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Every AgentJobService built its tasks SyncedMap with the name "agent.tasks",
and there is one service per user. So every tenant published on and subscribed
to the same subject, state.agent-tasks.delta, and SyncedMap.apply scopes
nothing: a task tenant A created was written into tenant B's in-memory map on
every replica, and ListTasks reads that map. Nothing repaired it short of a
process restart.
The subject now carries the tenant in a token of its own,
state.<name>.<tenant>.delta. Four tokens where the unscoped builder makes
three, deliberately: SubjectMatches compares token count before anything else,
so a tenant's subject and the cluster-wide one cannot cross-match, and neither
can two tenants. Putting the tenant inside the name token would not do that,
because the sanitizer folds '.' to '-' and the only filter that could then span
tenants is state.*.delta, which spans every other family too.
The rule is stated once. subscribeFilters calls publishSubject rather than
restating the subject, so a map cannot end up publishing scoped and subscribing
unscoped, which would leak exactly as before while every publish assertion
passed. The one case that decides on its own is the cluster-wide administrative
view: it hydrates from every tenant's rows, so it also takes the per-tenant
wildcard, or it would be stale the moment any tenant wrote. A tenant hydrates
from its own rows and applies only its own deltas. PerTenant defaults to false,
so finetune, quantization and the responses store keep the subject they have.
The second half of the same defect was the delete. taskStoreAdapter.Delete
called DeleteTask(id) and JobStore deleted by primary key with no user
predicate, reachable from DELETE /api/agent/tasks/:id, which takes the id off
the URL. A tenant who learned another tenant's task id destroyed that tenant's
row. The user id now travels with the id and lands as a user_id predicate.
Empty stays the administrative any-owner scope, the same thing an empty id
already means for ListTasks and ListJobs. A foreign delete removes nothing and
returns no error: not yours and not there are the same answer to the caller,
and neither is a store failure.
SetUserID rebuilds the tasks map for the same reason SetTaskSyncNATS does.
GetJobs happens to set the user id first, nothing enforced it, and with the
order reversed the map would be built with an empty tenant and put that user's
tasks back on the cluster-wide subject.
Both halves predate this programme; they are surfaced here rather than caused.
Neither is fully closed for a deployment with the agent pool off, where the
task routes are still served by one cluster-wide service that every
authenticated caller shares; that is a separate gap and it is documented.
testutil.FakeBus grew a real defect this was the first change to trip:
Unsubscribe matched on the filter string, so with two subscribers on one
filter, closing one deafened the other. Subscriptions now carry an id.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The responses.metadata SyncedMap had no durable Store, so its reconnect
re-hydrate replaced nothing. That was survivable while responses converged
through deltas on a broker that mostly stayed up. It is not survivable on a
carrier whose listener is one pinned PostgreSQL session: every response created
while the subscription was down stays invisible on that replica forever, and the
symptom is a 404 from one replica and a 200 from another for the same
response_id.
State that must survive a gap now lives in a response_metadata table, and the
notification only says it changed. The map writes through on a Set and reads the
table on hydrate, on reconnect and on reconcile, so the gap closes instead of
becoming permanent.
The row carries the whole projection as JSON rather than one column per field. A
column-per-field schema would be a second definition of what a peer may act on,
and the two would drift the first time syncedResponse gained a field: the map
would broadcast the new field and hydrate without it, so a replica that had
reconnected would serve a different response body from one that had not, with
nothing failing anywhere. Only PayloadJSON is ever decoded; owner_replica and
owner are indexed copies for an operator reading the table by hand.
A missing row and an unreachable database are different facts. Every store and
adapter method returns a driver failure as an error and never as an empty
result, and syncstate replaces nothing when its source errors, so an outage
leaves the map holding what it had rather than blanking it into a cluster-wide
404.
Liveness is the database's clock, spelled expires_at IS NULL OR expires_at >
now(), because every replica hydrating from this table must agree on which rows
are live and a Go-side cutoff makes that a property of whichever process asked.
The test container shares the host clock, so no behavioural spec can tell the
two apart; the statement shape is pinned instead. The constructor refuses a
non-PostgreSQL handle, because an unguarded now() on the single-binary path
reads as a missing migration.
A ticker sweeps expired rows every five minutes on each replica, and Close waits
for it rather than racing it. Note that the sweep removes nothing while
LOCALAI_OPEN_RESPONSES_STORE_TTL is 0, which is the default: with no TTL nothing
ever expires and the table grows for the life of the deployment. The docs say so
plainly.
EnableDistributed takes the store positionally and last, so a call site that
forgets it fails to compile rather than silently restoring the deltas-only map
this change exists to replace. A nil store there is refused by name: it is
reached only from the distributed branch of route registration, so it is a
wiring bug and not a deployment shape.
What still never leaves the owning replica is unchanged: the resume buffer and
the CancelFunc. The write-through is one row per response state change, not one
per generated token.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The three NATS queue groups jobs.new, jobs.mcp-ci.new and agent.execute are
gone. Dispatching work is now a row in a work_claims table, taken by one
frontend replica with SELECT ... FOR UPDATE SKIP LOCKED and driven on an agent
worker as a streaming control RPC over that worker's tunnel. Exactly-one
delivery among competing consumers is a database problem, not a broker feature.
An agent worker has no database, so it never claims; it executes what the
claiming replica hands it.
A claim must not outlive the replica that took it. The reap releases a claim
whose owner is no longer a live replica in the instances table, on the database
clock, and never asks how long the claim has been held. A job that legitimately
runs for an hour on a heartbeating replica is left alone, while a claim whose
owner stopped heartbeating becomes claimable again within one liveness window.
A replica with no advertised address has no instances row at all, so it refuses
to claim rather than have its work reaped out from under it mid-run.
The settle rule is stated once, in settleClaim, and every exit path calls it. A
transport failure releases the claim and never completes or discards it; only a
decoded reply line completes it. That line is deliberately not
cluster.IsWorkerAnswer, which accepts the stream refusals a worker's tunnel
writes before any request body reaches its control server: completing on those
would discard work that never ran.
The terminal line is persisted before the claim is completed, so a store that
refuses leaves the claim standing rather than leaving the job running for ever.
That is the dropped-result defect fixed structurally rather than by retry.
This also surfaces a pre-existing gap rather than causing one: no worker has
ever served plain task jobs, and publishing them into an empty queue group left
them running with no trace. Such a claim is now failed with a reason.
Removes QueueWorkers, --agent-subject and --agent-queue, and narrows an agent
worker's minted JWT by agent.execute and jobs.mcp-ci.new.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
nodes.<id>.backend.stop was the last worker-facing NATS subject, and it existed
only because ONE publisher had not moved. An agent worker already mounted
workerctl.PathBackendStop on the tunnel it holds, and a backend worker already
took its stop there, so RemoteUnloaderAdapter branched on NodeType to pick a
carrier for a verb both kinds of worker served the same way.
The branch is gone, and with it nodeTypeOf and its NodeTypeBackend default,
which removes one of the ten NodeType branches left to sweep. The adapter loses
its messaging.MessagingClient outright rather than keeping an unused field: it
now holds no publisher, so re-routing any verb back onto the bus is a change to
the struct and to every caller of the constructor, and does not compile until
all of them agree. messaging.SubjectNodeBackendStop and subjectNodePrefix are
deleted, the agent worker's subscription with them.
pkg/natsauth drops the per-node backend.stop grant from the agent SUB list. That
is a narrowing of eleven entries to ten, never to nothing: NATS reads an EMPTY
allow list as unrestricted, so the coverage spec asserts both that the retired
subject is no longer covered and that the queue subjects an agent worker lives
on still are. The e2e half proves it against a real enforcing server: one spec
subscribes successfully on an agent-minted JWT, the next is refused the retired
subject on a JWT minted the same way.
Both halves of the old split were pinned, so both pins are re-aimed rather than
deleted, and the two node types are asserted separately rather than as one
parameterised case, because only two cases can show that the two used to differ.
Three assertions that the adapter published nothing are deleted instead: with no
publisher to hold, no change could ever redden them.
The CLI's handler set moves into agentWorkerControlHandlers so a spec can stand
it up and post to it. That wiring was a bare literal no spec pinned, and
deleting the subscription made it the ONLY carrier for backend.stop: a dropped
field would have been a 404 the frontend reads as a worker too old to serve the
verb, and nothing in the repo would have noticed.
Mutations: the agent branch restored off the control route reddens two specs;
the backend branch restored, separately, reddens five; PathBackendDelete in
place of PathBackendStop reddens nine across both node types; dropping the CLI
wiring line reddens the new wiring table; re-adding the allow-list entry reddens
the unit spec and the JWT e2e spec; and restoring the publisher for real does
not compile.
Four comments this change falsified are fixed, in core/cli, pkg/model and the
distributed-mode docs, which now say both kinds of worker serve
POST /v1/control/backend/stop and what each does with it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
