mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-29 17:44:30 -04:00
4d26e648bc1e13a8acda3ef56df36c8511b2bb91
252
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ettore Di Giacinto
|
4d26e648bc |
feat(cluster): open the door a worker dials its tunnel through
A worker needs no inbound port: it dials GET /api/cluster/connect, the connection becomes one multiplexed yamux session, and the frontend opens a stream on it per request. This adds the endpoint that accepts that dial and attaches it to the tunnel registry. The dial is authenticated against the NODE's own stored token hash rather than the deployment's registration token. That is the mechanism, not yet the isolation, since a worker still registers by presenting the shared token; what it rules out is the shortcut of comparing against the configured value, which would have to be unpicked the day workers get their own secrets. Every refusal happens BEFORE the WebSocket upgrade, so a dialer reads an HTTP status rather than a handshake error. The route is registered in every deployment, single-binary ones included, which is what puts it in front of the route-coverage test that holds that rule in place; with no node registry it refuses every dial, and tells a credentialed one the frontend has no cluster rather than that its token is wrong. A lookup that FAILED is answered as a failure. Reporting a database that could not be read as "unauthorized" would send a worker re-registering, throwing away the identity its tunnel and loaded models are keyed by. Wires the tunnel registry in core/application/distributed.go and hands it to the membership loop. Without that call the re-claim after a replica is reaped had no production caller and could never run. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] |
||
|
Ettore Di Giacinto
|
9e4df19045 |
fix(cluster): keep a session close out of the per-node claim gate
The gate is justified by being held for one claim round trip, and Attach held it across the close of the session it superseded. Closing a yamux session closes the underlying conn and then waits for both its send and recv loops to exit, and the send loop can be inside a write bounded only by ConnectionWriteTimeout, so that is a wait on other goroutines. It must not stand between a worker re-dialling this node and its claim. The gate is now released after the store and before the close, which also makes Attach match reclaimOne, where it has always been released explicitly on every path. This is safe because a superseded session is no longer reachable from the map by the time it is closed: the next re-dial replaces an entry that already names the new session. Pin the re-claim half of the gate too. A worker that re-dials between a re-claim's commit and its record leaves the row carrying the re-dial's epoch while the entry carries the re-claim's, so the attachment holding the socket releases an epoch the row does not have and the row outlives it, with nothing to sweep it while this replica is alive. Only Attach's half of the serialisation was asserted; keying the two apart left every spec green. Also take the test hook's action under the lock that guards whether it has fired. It was written from the spec's goroutine and read from whichever goroutine issued the statement, which is a race in the harness that pins the serialisation specs. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
2466e42906 |
fix(cluster): make a claim and its record indivisible per node
Two Attach calls for one node both claim, and PostgreSQL serialises the two upserts, but nothing ordered the two map writes against the two commits. The entry left installed could be the one whose claim lost the row, and its Detach then released an epoch the row does not carry, so the release matched nothing and the row survived the socket. Nothing swept that row. This replica is alive and heartbeating, so ReapStale leaves its rows alone, and no reconnect is coming for a worker that has gone. Owner kept naming this replica as the live owner of a tunnel it no longer held, and every dialer sent here was answered ErrNotOwner, which is the relay into a replica that cannot serve the request that this phase exists to prevent. Claims for one node now pass through a gate, so claim and record are indivisible. It is per node rather than one lock over the registry, the way PeerPool locks per peer: the claim is a database round trip, and a slow one for a single worker must not hold up Open for every other. Detach is not gated, because it takes no context and must never park behind an in-flight database call, and it changes no epoch. Reclaim takes the same gate, which makes its claim the newest one for that node, so it records the epoch on whatever attachment is installed rather than only on the one it listed. Refusing to record onto an attachment that replaced the listed one would leave that row with nothing able to release it. The interleave the gate does not cover is Detach, and a claim whose attachment detached while it was in flight is now released again rather than left behind. Also: restore Start's doc comment, which SetTunnels had swallowed; keep reaping other replicas when this one fails to rebuild its own row, rather than skipping the sweep along with the re-claim; scope the comment about an unnoticed dead socket to the keepalive of the session whoever accepted the tunnel built, since the worker session config does not exist yet; and pin the sortedness of Held, the nil-session refusal, and both re-claim interleaves with specs. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
032725c81b |
feat(cluster): hold worker tunnels, and re-claim them after a sweep
Phase 1 left the connection fence with a table and no sockets behind it. This adds the registry that holds them: Attach claims the node and then stores the session, Open hands out a stream over the tunnel this replica holds, Detach releases the claim it was handed, and Held names what this process is carrying. The claim is written before the session is stored. A claimant that installs itself and only then finds it cannot claim has, for that window, published a tunnel no row records, so Held names it while a peer asking Owner is told the worker is connected nowhere. ErrNotOwner is produced at one place, the map miss. It is a routing fact: some other replica may hold that worker perfectly well. A broken socket under a held entry is returned as itself, because answering "not held here" would send a dialer looking elsewhere for a worker this replica is holding. Epochs are compared for equality and never ordered. Claim guarantees an epoch is unique and never reissued; it does not guarantee the later claim draws the larger number, because the sequence value on the insert path is drawn before the row lock. The membership loop now re-claims on re-register, which closes the hole phase 1 named in ReapStale. A replica that stalls long enough is swept by a peer, losing its instance row and, in the same transaction, every connection it owned; Register rebuilds the instance row and nothing else, so without this it serves workers that every other replica reports as connected nowhere. Re-claiming draws a fresh epoch, so an attachment carries two: the token Attach handed back, which is what Detach matches and which never moves, and the epoch of the row currently held, which is what Release is given. Collapsing them would leave the re-claimed row outliving the socket with no caller able to remove it. A tunnel whose session is already closed is skipped rather than claimed back, because claiming is an upsert and would take the row from whoever holds the worker now. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
df7cebf642 |
fix(cluster): make the ownership comments say only what holds
Review round 1 on the joined Owner read. The behaviour was accepted; three
comments claimed more than the code delivered, one spec pinned less than its doc
promised, and one pre-existing spec ranked epochs.
instanceIsLive said every reader of instance liveness uses it, which was false:
ReapStale spelled the complement by hand. The complement is now written as
NOT (instanceIsLive), so "stale" is exactly "not live", including how each side
treats a NULL last_seen, and the sentence is true. Inverting that predicate
fails 3 reaper specs, so the routing is held.
The Select("node_connections.*") in Owner was justified by a SELECT * hazard
that cannot occur: with a join present and nothing selected, gorm expands the
model's own columns table-qualified (callbacks.BuildQuerySQL), and the suite is
green with the Select removed. It stays, because the projection should be a
property of this query, and the comment now says that instead.
Owner gained the dialect guard Claim has. now() and make_interval are
PostgreSQL, so on the SQLite single-binary path it failed with "no such
function: now", which reads as a missing migration; that regression already
shipped once in phase 1. The refusal is deliberately not ErrNoConnection: a
deployment with no cluster has no answer about ownership, and reporting absence
would let a caller conclude the worker is not connected. A spec in the
non-PostgreSQL block holds all three properties.
The new specs aged rows by ten minutes, which any window between zero and ten
minutes satisfies, so nothing tied Owner's window to the one the sweeper uses.
They now age to just past InstanceLiveness, and a sibling ages to half of it and
must still resolve. Widening the window tenfold fails 2 specs, narrowing it
tenfold fails 1; before this both were silent.
The concurrent-claim spec asserted the stored epoch was the highest handed out,
and justified it with claims drawing their epoch after the row lock, which
contradicts Claim's own doc: the insert path draws nextval while the tuple is
built. It now asserts the stored epoch is one of the epochs handed out, and
ranks nothing.
OwnerRow's doc justified the function with a sweeper that does not call it.
ReapStale deletes orphans with a set difference; the callers are this package's
specs and one e2e assertion. It says that, and states plainly that a caller
needing to know who owns a node in order to dial it wants Owner.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Ettore Di Giacinto
|
a275147145 |
feat(cluster): resolve tunnel ownership against a live owner
OwnerRow is a bare row read of node_connections. A connection row outlives the replica that wrote it: a replica that dies stops heartbeating, but its rows survive until a peer's sweep removes them, which is up to InstanceLiveness plus one InstanceHeartbeat later. For that whole window the table names a process that is gone. The next component phase 2 builds is the relaying dialer, and a dialer reading OwnerRow would relay into a corpse for roughly 35 seconds after every replica death, then report the worker as unreachable when it is in fact absent, which is the distinction the phase 1 end-to-end specs pinned. Owner is the resolving read: one statement joining instances, returning ErrNoConnection when the row is missing OR its owner is not live. Both cases are one answer on purpose, since both mean no replica here holds this tunnel; they differ only in which sweep has run. It is one statement, not a row read followed by an instance lookup, because between two statements the owner can die and the caller would act on an owner the second read would have rejected. OwnerRow stays, unjoined, for readers that need the row itself, and a spec holds the two apart: with an aged-out owner, OwnerRow still names it and Owner refuses, so neither can quietly become the other. The liveness predicate is now one string, instanceIsLive, shared by Live and by Owner's join. Two spellings of one fact drift, and this drift would show as a relay to a replica one query calls dead and another calls alive. It is table-qualified so it is unambiguous inside the join, and the cutoff stays on the database clock, so replica clock skew cannot widen or narrow the window. Both mutations were run. Dropping the liveness predicate from the join fails 3 specs, the aged-owner one among them. Replacing the database clock with a Go-side time.Now() fails 1: the aged-owner specs still pass, because the two clocks agree on one host, and only the recorded-SQL spec sees the literal timestamp. That is why that spec exists. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
3daf889f5e |
fix(cluster): share one lock order, and correct the phase 1 comments
ReapStale deleted from instances then node_connections while Deregister took them the other way round, both inside one transaction and both running concurrently by design: a replica shuts down while a peer sweeps it. Opposite orders let each hold the row the other waits for. PostgreSQL breaks the cycle by aborting one side, so the cost today is a warning rather than lost data, but the inversion costs nothing to remove. Deregister now deletes the instance row first. That is the order ReapStale is forced into anyway, since its connection delete asks which instance rows survived, so the sweeper is the fixed side. Both functions say the order is deliberate and shared, and name the other. A spec records the statements each path issues and asserts they delete from the same two tables in the same order; racing two transactions until they really deadlock would be flaky and could pass for the wrong reason. The rest is comment and spec accuracy, deferred from the phase 1 task reviews: - co-location does not imply loopback. Compose's usual host=postgres resolves to a bridge address and discovery works there; it is a DSN that NAMES localhost that yields a loopback source address. Corrected in the DiscoverAdvertisedAddr doc and in the spec comment that repeated it. - unroutableReason labelled every scoped address "link-local", including the class the check exists for, and formatted the IP with %s, which drops the %iface, so the reported address was not the one being rejected. Split into two cases, both rendered with their zone. CheckAdvertisedAddr passed zone "" and net.ParseIP rejects fe80::1%eth0, so a scoped literal looked like a name and collected no warning at all; the zone is now split off before parsing. - Splice's "Both callers satisfy it" claimed callers that still do not exist. It now names the two stream types the wake-on-Close property was verified against and says a phase 2 caller over anything else has to check it. - restored, short, why a socket-level ECONNRESET stays reported while a yamux reset does not: the yamux endings are the teardown Splice's own Close provokes, and whether an aborted request is routine is the relay's policy. - the real-yamux spec's far.Read had no deadline, so a stall parked the suite rather than failing it. - gorilla's SetWriteDeadline is conn.go:796, not 787. - ClusterPathPrefix is no longer derived from: the peer route spells its path out, because core/services/cluster must not import core/http/auth. The comment now points at the spec that holds them together instead of claiming a derivation the move removed. - the epoch spec asserted e2 > e1, an ordering Claim's doc tells callers not to rely on. It asserts uniqueness, which is what the fence guarantees, and is named for that. A sibling spec still described the epoch as incrementing in SQL when it is drawn from a sequence. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
bb4719e67d |
fix(cluster): hold the guarantees phase 1's comments were claiming
Review found the recurring class: assertions that a wrong implementation also satisfies. The "refuse promptly, never park the peer" guarantee was stated in three places and tested in none. Removing the Close from the no-relay branch left the whole cluster suite green, because the specs asserted only that some error arrived and yamux reports a read deadline as ErrTimeout: a parked stream satisfied that as well as a refused one. Both specs now require an ENDING, EOF or a reset, inside a deadline short enough that parking is unmistakable, and both go red when the Close is removed. Deregistration existed only in a comment. Membership.Stop ended the loop and left the row behind, so every clean rolling restart had peers dialling a corpse for the full liveness window; the shutdown comment described the opposite. Registry.Deregister deletes the row and the connections that replica owned, in one transaction, for the reason the sweeper does both, and an e2e spec pins departure inside a budget shorter than the liveness window so it cannot pass on the sweeper doing the work. Before: the spec times out with both replicas still live. After: 3.6s. The configured advertised address bypassed every check discovery makes, so the one value most likely to be copied between hosts, 127.0.0.1, was taken verbatim and would make every peer dial itself. Both paths now share one rejection rule: unparseable is refused, "this host" is warned about once and honoured, because a single-host deployment uses it correctly. Two comments claimed more than the code does. The sweeper said a stalled replica recovers via re-register; only its instance row does, while the connections another replica reaped stay gone and the sockets stay held here - phase 2 must re-claim, on re-register, every connection a replica still holds locally. And Owner became OwnerRow, documenting that the owner it names may be dead for up to InstanceLiveness plus a heartbeat and that any caller acting on it must join instances itself, so the deferred constraint lives at the call site rather than in a report; the plain name is left free for the joining version. Minors: warn once when the peer link mounts with no registration token, so an operator sees the cause rather than 401s; Stop no longer blocks forever when Start was never called; corrected the NewRegistry migration doc and an e2e comment that described a 6s window as "throughout". Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
dca8ce7149 |
feat(cluster): give phase 1 a call site, and prove it against real replicas
Tasks 1 to 5 built an instances table, a splice, both halves of a peer link and an epoch fence, and nothing in the tree called any of it: no replica registered, no route was mounted, no sweeper ran. Proving phase 1 end to end therefore had to start by wiring it. A frontend in distributed mode now publishes the address its peers dial, heartbeats it, and sweeps replicas that stopped answering along with the connection rows they owned, in one pass so the two can never disagree about who is alive. It serves the peer link and owns the sessions peers dial in, refusing streams on them until phase 2 installs a relay: a session nobody accepts on does not fail a peer's Open, it hangs it. The address is the one peers use, not the one the process binds, and it is derived from the route to PostgreSQL. That derivation only holds while the database is remote, so LOCALAI_DISTRIBUTED_ADVERTISE_ADDR sets it explicitly and a replica that can determine neither warns and keeps serving rather than failing to start. Three e2e scenarios run against real local-ai processes, real PostgreSQL and real dials: replicas publish addresses that can actually be connected to; a sibling opens a stream over the peer link and is refused without the cluster token; and a killed replica is reported unreachable, never absent, loses the claim it held, and takes no worker with it. Each was verified by mutation: eight injected defects, each failing the scenario that claims to catch it. Also moves RegisterClusterRoutes to core/http/routes beside every other registrar, folds AutoMigrate and the epoch sequence into one cluster.Migrate, and turns the peer route's auth-coverage spec into a real assertion: it drives the request through the actual auth middleware instead of comparing two string constants, which the old spec would have passed even with the exemption deleted. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
e9514334fa |
fix(cluster): keep the connection schema migratable on SQLite
The NodeConnection model carried `default:now()`, which is PostgreSQL syntax reaching the DDL, so AutoMigrate failed on the single-binary SQLite path and took every SQLite caller of nodes.NewNodeRegistry down with it. Stamp the database clock as an expression inside Claim instead, the way Register already does, and leave the column plain. CREATE SEQUENCE is Postgres-only for the same reason, so it is skipped on another dialect, and Claim refuses that dialect outright: a fence that cannot draw a token must say so rather than fail later as a missing function. Also correct a claim the previous commit made in both the doc comment and its message. An epoch is unique and never reissued, but it is not ordered: the insert path draws its sequence value before taking the row lock, so a claim that inserts after a release can be handed a lower number than one already issued. Uniqueness is what Release needs, since it matches by equality; callers must never compare epochs for order. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
6f53701f4c |
fix(cluster): draw connection epochs from a sequence so none is ever reused
Release deletes the row, so a per-row `epoch + 1` restarted the numbering at 1 for the next claim. A replica could then be handed an epoch it already held: claim w1 at epoch 1, lose the link silently, watch another replica claim and release, reclaim and be handed 1 again, and its delayed cleanup for the first dead link would match the live claim and delete it. The fence has to be unique per node over time, not per row lifetime. Every claim now draws nextval from a dedicated sequence on both the insert and the conflict paths, so an epoch is never issued twice. The draw still happens after the row lock on the conflict path, so the winning claim still holds the highest epoch handed out. Also drop last_seen. Nothing maintained it and it was always equal to connected_at, but an indexed column named that way invites a second liveness clock; whether the owner is alive is Instance.LastSeen, and whether a claim is current is the epoch. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
edc7313f3b |
feat(cluster): fence worker-connection ownership with a monotonic epoch
A worker whose link is silently broken reconnects to another replica while the old owner's socket has not yet noticed. Without a fence both believe they own it. Claim is a single atomic upsert returning the new epoch, and a release must match both owner and epoch so a stale owner cannot delete a live claim. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
393797e7e3 |
refactor(cluster): make the cluster service a leaf and blame the caller's deadline
The peer link's WebSocket adapter and route constant lived in core/http/endpoints/cluster, so the dialler in core/services/cluster had to import an HTTP endpoints package to reach them. That pulled echo, core/http/auth and core/config into a package whose doc says it is deliberately free of such dependencies, and it made core/services/nodes reach an endpoints package transitively. It also has no way forward: the worker-connect handler needs the tunnel registry and the node token store, both of which are cycles from there. Move WebsocketConn and PeerPath into core/services/cluster and let the endpoints package import it, which is the direction the rest of core/http flows. The route and the auth exemption still cannot drift apart, now asserted where both are visible rather than by a const reference across the boundary, and the assertion is stronger than the one it replaces: it pins the route under the prefix instead of pinning the prefix's spelling. Also guard the fresh-dial path with ctx.Err(), mirroring the cached path. A caller with a 300ms deadline dialling a live, listening peer was told the peer was unreachable, which would be enough for one impatient client to get a healthy replica routed around once the relay consults these errors. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
c7cb7e1ca5 |
feat(cluster): dial and pool yamux links to peer replicas
Distinguishes a peer missing from the registry from a peer that will not answer: the second must never be readable as node absence, or a network hiccup between replicas evicts healthy workers. The distinction is a property of the error type rather than of the call sites. The unreachable error formats its cause into its message and keeps it out of its unwrap chain, so an ErrInstanceNotFound picked up on the dial path cannot reach a caller's absence check. One yamux session is cached per peer and re-dialled when OpenStream on it fails, which is how both a dead transport and a graceful remote go-away arrive. A reset of one stream never reaches the pool, so an abandoned request cannot cost every other worker its link. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
73da8089d6 |
fix(cluster): stop reading a bare EOF as a clean ending
A dead yamux session does not always arrive wrapped. Session.close publishes shutdownErr and closes shutdownCh before it force-closes the streams, so a Write or Close landing in that window gets the raw cause back instead (session.go:507-510, 528-533), and for a peer that vanished the raw cause is a bare io.EOF. The generic io.EOF clause then reported the dead session as a clean completion. Remove the clause. A clean read-side EOF never reached it anyway: io.Copy consumes that and reports nil, and neither *yamux.Stream nor *net.TCPConn takes a WriteTo/ReadFrom path that would hand one back. Every existing spec still passes, the io.EOF entry in the normal-termination table included, which is what showed the branch was dead for legitimate endings and live only for the bug. Add a spec driving a real yamux session end to end. Every mux shape until now was a synthesized error, which is exactly why a race inside the real library stayed invisible. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
a0deca6385 |
fix(cluster): report a dead yamux session instead of swallowing it
Matching yamux errors with errors.Is was too broad. Session.close hands every live stream ErrStreamReset wrapped around whatever killed the connection, so a keepalive timeout, a broken TCP connection or a peer that simply vanished all matched, and a relayed request that died reported a clean ending. Nothing upstream would have retried or logged it. Match the plain sentinels by identity, since only identity separates a stream that was reset from the wrapped form that means the session died. Treat a StreamError as a per-stream reset, and a GoAwayError as normal only when it carries the no-error code, read off ErrRemoteGoAway because the constant is unexported. ErrSessionShutdown needs no entry of its own; it is a GoAwayError with that code. Order matters as much as the matching: session death wraps its cause, which is routinely io.EOF or a closed socket, so the mux checks run before the generic endings. Reversing them alone puts a vanished peer back to nil. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
c05246a366 |
fix(cluster): treat a yamux teardown in Splice as a normal ending
go-yamux/v5 matches none of its errors against net.ErrClosed, so the classifier reported an ordinary teardown as a failure: when the session has gone away, the FIN that Splice's own Close writes returns ErrSessionShutdown, and a stream torn down under a live copy surfaces as ErrStreamClosed or a reset. Splice owns that Close, so it owns the errors it produces; the sentinels are named here rather than injected by the caller, which would make a forgotten classifier reintroduce the same bug silently. Cover the error half of the contract, which no in-memory pipe could reach: a scripted stream now feeds Splice a genuine transport failure and each closed-stream ending in turn. Replacing the tail of Splice with "return nil" passed every previous spec. Also assert that Splice does not return until the second direction has finished, rename a spec that promised a leak check it never made, and correct two comments that claimed more than the code did. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
6caceb4b1d |
feat(cluster): add the bidirectional splice used by the relay and tunnel
Returns on the first direction to finish and closes both sides so the other unblocks; a sequential copy deadlocks on any protocol where the far side speaks first. EOF and use-of-closed are normal termination, not errors. The fourth spec covers a peer that stops reading mid-body, the case where a copy is parked in Write rather than in Read. The other three tear down an idle splice and pass even against a Splice that closes only one side. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
42be69d469 |
fix(cluster): stamp liveness on the database clock and refuse undialable addresses
DiscoverAdvertisedAddr promised to return an error rather than a fallback no peer can dial, but only rejected an unspecified address. With PostgreSQL on the same host or pod as a replica, which is compose, single-node and any sidecar layout, the route to it is loopback, so every replica advertised 127.0.0.1 and a peer dialling that reached itself. Loopback, link-local and zoned source addresses are now rejected with an error naming the remedy, and a port outside 1-65535 is rejected before it becomes an undialable address. Liveness was also measured on each replica's own clock: Register and Heartbeat stamped last_seen from the Go process, and Live compared those rows against the reading replica's time.Now(). Skew therefore shrank or stretched the window by writerBehind+readerAhead, evicting healthy peers or keeping dead ones. Both sides now use the database clock, which is the one clock every replica demonstrably shares. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
c72baa4ad9 |
feat(cluster): record frontend replicas in a shared instances table
Replicas need to find each other to relay worker traffic, and nothing in the tree recorded a replica's address. The advertised address is discovered by opening a UDP socket toward PostgreSQL and reading back the local address, which yields the interface every replica demonstrably shares without asking an operator to configure one. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
1e5f5dbe52 |
test(distributed): correct the claims the e2e comments make
Review of the whole branch found five comments that would send a reader to the wrong place, plus three smaller inaccuracies. Nothing here changes behaviour. The KNOWN RACE note on both backend-log WebSocket handlers said the fix needs an atomic snapshot-plus-subscribe "under the store lock". It does not: BackendLogStore.mu guards only the buffers map, and AppendLine enqueues and fans out under the per-buffer buf.mu. Whoever took the store lock would ship and the race would survive, so both notes now name buf.mu and say what s.mu does and does not exclude. Two comments in the cluster harness quoted Eventually(c.FrontendAlive) .Should(BeFalse()). FrontendAlive takes an index, so Gomega rejects that with "requested 1 arguments but received 0". Both now quote the closure form the specs actually use, and say why the closure is needed. proveHealthCheckingIsAlive claimed to prove the health monitor ran for the whole preceding window. It proves the monitor was alive at the end of it, and inferring backwards needs any wedge to be sticky. In the peer-replica-death spec that inverts: health checks are single-flighted by a session-scoped pg_try_advisory_lock, the spec SIGKILLs the replica that may hold it, and until Postgres reaps the session the survivor acquires nothing and checks nothing silently. Consistently(healthy) can then pass because nothing was checking, with the positive control still succeeding once the lock frees. The doc now states what is proven, names that gap, and says the assertion is a floor rather than a proof. The Makefile still called DISTRIBUTED_TEST_FLAKES a retry count, which is what seeded that error into the two docs just corrected against it, and the workflow called the 15s window a reconcile tick when the mechanism is HealthCheckInterval in the node health monitor. Also: the cluster suite measured 509.1s / 509.8s / 512.3s, so about 8m30s and not the 8m39s/8m40s three files claimed; the dead-worker spec title implied two independent detectors when both probes read one advisory-lock-serialised verdict out of the same row; and the sanitizeDBName length assertion used <= 50, which an empty string also satisfies, where the invariant for an over-long input is exactly 50. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
1dc7998520 |
test(distributed): scope the log-subscriber wait and mark the race it works around
Three corrections from review of the previous commit. The lock-order comment on SubscriberCount claimed no path takes s.mu and a buffer lock together. Subscribe does exactly that, holding s.mu.RLock across replica registrations that take buf.mu. State the rule that is actually true — s.mu precedes any buffer lock, so counting after releasing it preserves the order — and say what follows from it: the total is a sample, not a snapshot. waitForLogSubscriber read as general-purpose but unblocks on the first registered subscription. Subscribe attaches the exact-key buffer and each replica buffer one at a time, so for a replicated model the count goes positive while later replicas are still unattached and the race survives. Rename it waitForSingleLogSubscriber, document that it holds only where Subscribe resolves to one buffer, and assert on exactly 1: misuse then fails loudly on the count rather than going quietly back to being flaky. Taking the expected count as a parameter was the alternative, but that makes callers predict a store-internal number and an under-count fails the same silent way as the original bug. The snapshot-then-subscribe race had no artifact outside a report, and review found a second site carrying it. Mark both handlers identically, including the point that swapping the two calls duplicates rather than drops and so is not the fix. The race itself is left alone; this branch stays test infrastructure. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
c0b7e64973 |
feat(ui): show running models and host gauges on single-node installs (#12189)
On a single-node install nothing in Operate listed the models loaded on this machine or let an admin stop one. The System page that did was retired in #11548, and its replacements (the Nodes workbench) only work in distributed mode. The Nodes page also mis-detected single-node mode: the cluster routes are not registered there, so /api/nodes answers 404, but only 503 was treated as "distributed off", which sent every single-node install to the empty worker-registration card. The rail hid the entry anyway. Nodes route on a single node becomes "This machine": - the Nodes page's VRAM / RAM / CPU / models-disk gauges, fed from this host by mapping /api/resources onto the worker heartbeat fields - a memory bar splitting host RAM by running model - a running-models table (backend, RSS, CPU share, uptime, PID) with search, sorting, logs and a confirmed Stop - the distributed setup behind an "Add machines" button The Operate overview gains a "Running now" preview (heaviest five, with Stop) on single node and a pointer to Nodes > Running models on a cluster. The rail shows "This machine" in Runtime with a running count. Backend, additive only: - /system: each loaded model carries a `process` block (pid, rss_bytes, memory_percent, cpu_percent, started_at). A sampler keeps one gopsutil handle per PID so CPU is the share since the previous poll rather than the lifetime average; it is omitted on the first reading. - /api/resources: host `cpu` and models-path `disk`, the same readings workers send in their heartbeat. Also fixes the fleet tables widening the page on phones: the headers' absolutely positioned sr-only labels escaped the scroll wrapper. Assisted-by: Claude:claude-opus-5 [Playwright] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
237cf456fd |
feat(gallery): galleries published as OCI artifacts (#12167)
* feat(oci): pull ORAS artifacts into a directory Galleries published to an OCI registry arrive as an ORAS artifact whose layers carry their tree path in org.opencontainers.image.title. Nothing in pkg/oci could read a non-image manifest, so add PullArtifact: it lays the layers out under a destination directory and returns the manifest digest so callers can pin and verify signatures on it. A registry is remote input and this writes files, so the manifest is validated before any byte is fetched. A title that is absolute, escapes the destination or is missing is refused, an artifact of an unexpected artifactType is refused, and both the layer count and the total size are capped with caller-settable limits. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * feat(gallery): fetch and verify galleries published as OCI artifacts A gallery whose URL is oci://host/repo:tag is now pulled as an ORAS artifact and its index.yaml read from the unpacked tree, alongside the existing HTTP and file sources. Mirrors, the per-candidate timeout, the failure cooldown and the last known good copy work the same for both schemes, so a gallery can mirror an OCI primary with an HTTP fallback. When the gallery declares a verification policy the artifact reference is resolved to its manifest digest, the publisher signature is checked against that digest, and only then is the same digest pulled. Nothing unverified is written to disk or parsed. With strict integrity mode on, an OCI gallery without a verification policy is refused instead of warned about, which is what the flag already does for backend installs. The pull lands in a staging directory that is renamed into the cache only once the whole tree is on disk and the index reads back as an index, so a failed or interrupted pull leaves nothing a later fetch would serve. A fetch that finds a fresh cache entry does not contact the registry. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * feat(gallery): resolve entry urls relative to the gallery root A gallery published as a self-contained tree, which is what an OCI gallery is, names its base configs by their place in that tree. Such a url reached the HTTP client verbatim and no entry of the gallery could be installed. A scheme-less entry url now resolves against the root the index came from: the unpacked artifact for an oci:// gallery, and the directory of the index URL otherwise. The relative path may not climb out of that root, reusing the guard the artifact puller already applies to layer titles, and an entry that does is dropped from the listing rather than failing the whole gallery. An entry that names its own scheme is untouched, so .ref indirection and local github: resolution behave as before. Reading an entry of an OCI gallery also needs the unpacked artifact directory as its trusted root, since the cache is deliberately a sibling of the models directory the downloader otherwise confines a file read to. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(oci): restrict artifact file permissions Create artifact files with owner-only permissions to satisfy G302. Document the existing path validation for the G304 scanner finding. Check the permissions of downloaded layers in the artifact test. Assisted-by: Codex:gpt-6 gosec * fix(oci): confine artifact writes to their root Use root-relative filesystem operations to reject escapes through existing directory symlinks. Retain owner-only permissions and add a symlink escape regression test, without suppressing the path traversal finding. Assisted-by: Codex:gpt-6 --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler-agentandEttore Di Giacinto
|
96ccb8e96c |
fix: correct stale symbol names in doc comments (#12139)
Fixes 7 doc comments whose names no longer match their function: - ToggleStateModelEndpoint (was ToggleModelEndpoint) - Messages.ToProto (was MessagesToProto) - NewUserCancellableContext (was newUserCancellableContext) - CalculateRMS16 (was calculateRMS16) - NewLocalAIMetricsService + Shutdown TODO (was copied setupOTelSDK) Supersedes #12084 and #12085 (DCO not signed by contributors). Assisted-by: MAKI:regolo/glm5.2 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
2facfc0d88 |
feat: Add kimodo.cpp and 3D animation API/UI (#12095)
* fix(vulkan): preserve host ICD discovery for packaged backends Add bundled Mesa manifests through VK_ADD_DRIVER_FILES instead of replacing the system driver list. Merge inherited and model-specific additive paths while preserving explicit operator overrides, with regression coverage. Assisted-by: Codex:gpt-5 golangci-lint Assisted-by: Codex:gpt-5.6-sol Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(3d): add Kimodo CPU and Vulkan animation backend Introduce a distinct animation capability and model-described 3D operations, with a typed /3d/animate API, RPC transport, distributed media staging, permissions, and tracing. Add a persistent kimodo.cpp adapter, skeleton GLB export, CPU/Vulkan packages, model and backend galleries, importer support, CI builds, and documentation. Adapt Studio inputs to each model and provide real-time skeleton playback, seeking, and history. Cover backend validation, packaging, API behavior, importer inventories, distributed staging, and Studio workflows. Validate real-model CPU/Vulkan generation and deploy the integration to the local QA instance. Assisted-by: Codex:gpt-5 golangci-lint Assisted-by: Codex:gpt-5.6-sol Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(kimodocpp): adopt monolithic encoders and resident inference Update upstream for resident weights, packed execution paths, and cached motion graphs. Default to all 32 text layers while retaining configurable streaming and legacy bundle support. Use monolithic Q8_0 encoders by default and offer all six published quantizations through the gallery and importer. Refresh pinned hashes, tests, and documentation; remove the obsolete thread patch and ensure cached source checkouts follow the upstream pin. Validated CPU and Vulkan generation, lower-bit streaming, gallery/importer suites, packaging, lint, and cold/warm Studio generation on localai-dev. Assisted-by: Codex:gpt-5 golangci-lint Assisted-by: Codex:gpt-5.6-sol Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
997d403de4 |
feat(nodes): add fleet operations dashboard (#12046)
* feat(nodes): report CPU telemetry Assisted-by: Codex:gpt-6 * feat(nodes): add fleet view utilities Assisted-by: Codex:gpt-6 * feat(nodes): add fleet operations dashboard Replace the panel roster with aggregate capacity gauges, fleet filtering and selection, bounded bulk actions, and an on-demand node inspector. Extend node details and distributed-mode documentation with CPU and models-disk telemetry. Assisted-by: Codex:gpt-6 * fix(nodes): harden fleet lifecycle actions Assisted-by: Codex:gpt-6 * fix(nodes): restore compact fleet composition Keep fleet health, capacity, and attention in one compact overview at ordinary desktop widths. The inspector now overlays the roster until the workbench can preserve a useful table beside it. Assisted-by: Codex:gpt-6 * feat(nodes): add accessible running models workbench Assisted-by: Codex:gpt-6 * fix(nodes): correct model view ARIA links Keep each tab panel available for its controlling tab while native hidden state removes inactive content from accessibility navigation. Model controls now expose only supported state and valid inspector relationships. Assisted-by: Codex:gpt-6 * fix(nodes): align lifecycle and capacity states Pending nodes now expose approval wherever node actions appear, while other lifecycle controls follow the server transition rules. Capacity totals exclude incomplete readings so missing availability remains unknown. Assisted-by: Codex:gpt-6 * fix(nodes): restore approved dashboard composition Assisted-by: Codex:gpt-6 * fix(nodes): integrate operate navigation Assisted-by: Codex:gpt-6 * fix(nodes): restore low density fleet view Assisted-by: Codex:gpt-6 * fix(nodes): preserve complete operate menu Assisted-by: Codex:gpt-6 * fix(nodes): preserve inspector workspace height Assisted-by: Codex:gpt-6 * fix(nodes): restore standard operate navigation Assisted-by: Codex:gpt-6 * feat(ui): add collapsible console rail Assisted-by: Codex:gpt-6 * feat(nodes): stop models from fleet view Assisted-by: Codex:gpt-6 * fix(nodes): make inspector a full height drawer Assisted-by: Codex:gpt-6 * fix(model): stop mixed local and remote placements Assisted-by: Codex:gpt-6 * fix(ui): announce action menu navigation Assisted-by: Codex:gpt-6 * fix(nodes): keep inspector within viewport Assisted-by: Codex:gpt-6 * fix(ui): preserve focus across model actions Assisted-by: Codex:gpt-6 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
8321234275 |
fix(agents): pass pool limits to the LocalAGI agent pool
The LocalAGI bump in #11985 changed state.NewAgentPool to take a SkillsProvider and a PoolLimits value. The call site here was not updated, so master stopped compiling and every Go job went red. Pass the limits explicitly, mirroring LocalAGI's own defaults, so the pool prunes conversation dumps and scheduler run history instead of growing without end. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Adira Denis Muhando
|
869000ceb7 |
fix(distributed): exclude pinned models from cluster eviction and idle scale-down
pinned: true was only honoured by the per-node watchdog. Every distributed eviction path was pinned-blind: the router's LRU eviction (EvictLRU, evictLRUAndFreeNode) and the replica reconciler's idle scale-down would happily unload a pinned model — and since eviction is gated on in_flight = 0, a pinned model became eviction-eligible the instant each response completed. Under capacity pressure that surfaces as the backend being freed immediately after every request (#11101). Wire the model config loader into the router and reconciler through a new PinnedModelResolver seam (mirroring ConcurrencyConflictResolver): - EvictLRU passes the pinned set into FindLRUModel's query so the next-oldest unpinned model is selected instead of the attempt failing - evictLRUAndFreeNode filters pinned models inside its locked selection - scaleDownIdle skips pinned models entirely: trimming to the floor still means requests beyond the survivor's capacity pay a cold reload Deliberate teardown (admin unload, model delete, node drain) intentionally still applies to pinned models, as does dead-row reaping (state correction, not eviction). Regression specs verified to fail with the exclusion disabled. Addresses the cluster-side eviction gap in #11101 Assisted-by: Claude Code:claude-fable-5 [Claude Code] Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> |
||
|
Ettore Di Giacinto
|
bbd5488871 |
fix(distributed): make backend.stop report what it stopped
backend.stop was the one lifecycle subject a worker never answered. The controller published and returned nil as soon as the local publish succeeded, so a stop that killed nothing, and a stop that failed outright, were indistinguishable from one that worked. The unload endpoint calls model.unload and then StopBackend. Only the first is acknowledged, so the endpoint answered 200 while the backend kept running and held its VRAM, and its own "backend stop failed" branch could never run. The worker logged the failure and nobody saw it. Give the subject a reply. The worker now enumerates the process keys it terminated and reports any per-process error, so StopBackend fails when the stop failed. Resolving to nothing stays a success: stopping a backend that is not running leaves the caller in the state it asked for, and eviction paths stop already-gone models routinely. The empty list is what says nothing matched, and ReportsStoppedProcesses is what makes that emptiness trustworthy, the same way BackendDeleteReply handles it. A worker built before this reply still receives the request and still stops the backend, it only stays silent, so a timeout degrades to the old assumption rather than failing every stop on a fleet mid-upgrade. Only silence degrades: a transport error is still reported, because UnloadRemoteModel skips its registry cleanup for a node it could not reach and needs to keep hearing about that. Assisted-by: Claude:claude-opus-5 golangci-lint |
||
|
Ettore Di Giacinto
|
db09452d54 |
feat(tts): support multi-reference personalities
Saved profiles previously resolved to one audio path and transcript, so cloning backends could not use several examples of one personality. Store ordered audio and transcript pairs while preserving the legacy first-reference fields. Fish Speech and audio.cpp receive all pairs, including on distributed workers. Other backends retain their single-reference behavior. Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
783556bc93 |
feat(prefixcache): index reported KV residency
Add a NATS event contract and exact-residency provider for backend KV cache reports. Keep guessed request observations as the default routing source while maintaining the reported index for future producers. Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
a64fc865ba |
fix(distributed): aggregate prefix cache pressure
Broadcast forced-disturb events and successful scale-up resets so every frontend shares the same rolling autoscale signal. Deduplicate NATS echoes, expose an origin-only Prometheus counter, and document cluster behavior. Closes #10083 Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
a9f8062dd5 |
feat(nodes): add pluggable routing pipeline
Route prefix-cache replica selection through composable filters, weighted scorers, and a replaceable picker. Preserve the existing load guard and deterministic selection policy while exposing per-model scorer weights through scheduling configuration. Assisted-by: Codex:gpt-5 [go-vet] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
9bd7d17ff6 |
[model-config] feat: add environment variables support for backends (#10721)
* feat: add environment variables support for backends in model configurations - Add field to model configuration to pass environment variables to backend processes - Update backend options and model configuration handling - Add documentation for environment variables configuration with examples including CUDA_VISIBLE_DEVICES Assisted-by: qwen-agentworld-35b-a3b Signed-off-by: nold <nold42@pm.me> * fix(test): Test environment variables configuration parsing from YAML Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: nold <Nold360@users.noreply.github.com> --------- Signed-off-by: nold <nold42@pm.me> Signed-off-by: nold <Nold360@users.noreply.github.com> Co-authored-by: nold <nold42@pm.me> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
e78271563b |
fix(distributed): pass raw JPEG base64 through staging (#11956)
JPEG base64 starts with /9j and was classified as an absolute path in distributed mode. Detect valid JPEG base64 by its decoded magic bytes so vision requests reach the backend unchanged while malformed path-like inputs retain existing staging behavior. Assisted-by: Codex:gpt-5 Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
be5342ef05 |
fix(worker): resolve temporary paths in tests (#11944)
Capacity guards reject symlink components. On macOS, temporary paths start with /var, which links to /private/var, so the new staging tests fail before exercising cleanup or capacity accounting. Resolve the fixture directories before building guarded paths. Keep explicit symlinks within the fixtures for containment tests. Assisted-by: Codex:gpt-6 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
bf93008ef3 |
fix(backends): bound temporary scratch files (#11941)
Backend processes shared the host temporary directory, so crashes could leave request images and audio behind until the filesystem filled. Give each process a locked LocalAI-owned runtime, remove scratch on exit, and sweep only marked abandoned runtimes at the next start. Also close known request error-path leaks in the Python media backends, CrispASR, LongCat Video, and stable-diffusion.cpp. Assisted-by: Codex:gpt-5 Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
8b5f62cc02 |
fix(distributed): bound ephemeral staging (#11924)
* docs: design ephemeral staging retention High-frequency camera and audio inputs can fill a worker before the current six-hour cleanup window expires. Define a one-hour retention policy that preserves recently modified request payloads. Assisted-by: Codex:gpt-6 * docs: make ephemeral staging request-owned Time-based retention can still fill a worker under bursty or high-rate input. Define request-lifecycle cleanup with capacity reservation and crash recovery. Assisted-by: Codex:gpt-6 * feat(distributed): release exact staged keys Request inputs need transport-neutral cleanup after backend calls. Add authenticated exact-key deletion for HTTP and coordinated cache eviction before shared-object deletion for S3/NATS. Preserve URL metacharacters as filename data, reject unsafe keys, and remove upload sidecars while pruning empty request directories. Assisted-by: Codex:gpt-6 * fix(distributed): release staged request inputs Ephemeral inputs remained on workers after inference completed. Release each exact key after synchronous and streaming calls, including partial staging failures. Use a bounded cleanup context so caller cancellation cannot suppress release. Preserve caller requests and backend results when cleanup fails. Assisted-by: Codex:gpt-6 * feat(worker): bound ephemeral staging capacity Concurrent staging can otherwise exceed its byte limit or consume reserved filesystem headroom. Explicit states keep bytes charged through each reservation, write, and commit transition. Use a synchronized waiter count to prove Commit blocks until bounded writers close, and retain committed baselines across re-reservation. Assisted-by: Codex:gpt-6 * feat(worker): enforce ephemeral staging bounds Share capacity accounting across HTTP and S3 request inputs so workers reject uploads before exhausting their filesystem. Reconcile exact release and crash recovery with the same guard. Assisted-by: Codex:gpt-6 * fix(distributed): make staged release race-safe Pin each release path component before removing request-owned inputs and sidecars. Stop pruning when a directory identity changes. Assisted-by: Codex:gpt-6 * fix(worker): retain staged input ownership Keep committed request inputs protected from age recovery until exact release ends their ownership. Startup-scanned files remain reclaimable and can acquire ownership through reservation. Assisted-by: Codex:gpt-6 * fix(worker): claim cached ephemeral inputs Keep startup-scanned cache hits owned while inference uses them and reconcile their actual size against capacity. Assisted-by: Codex:gpt-6 * fix(distributed): enforce staging admission Propagate multimodal staging failures before inference and claim matching ephemeral HTTP cache entries. Fall back to PUT when an older worker does not support claims. Assisted-by: Codex:gpt-6 * fix(distributed): close staging accounting gaps Keep unknown-length reservations charged until bytes reach disk and bound NATS release waits by the lifecycle cleanup deadline. Assisted-by: Codex:gpt-6 * fix(distributed): restage swept cache hits Treat files removed between cache probing and ownership claims as misses so HTTP and S3 workers can stage them again. Assisted-by: Codex:gpt-6 * fix(distributed): release staged inputs by request Release every input from one inference with one fixed-size worker coordination request. Fence request ingress against cleanup, bound staging capacity and cleanup state, and retain exact-key release for rolling upgrades. Assisted-by: Codex:gpt-6 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
aff9db9758 |
fix(distributed): resolve paths for virtual models (#11911)
Virtual model names have no primary file to anchor the worker path. Companion assets still stage successfully, but relative options retain an incorrect model directory and fail to load. Derive the worker root from successfully staged option assets when the primary path is absent. Cover Buffalo packs, files, directories, overrides, and failed transfers. Document the frontend upgrade. Assisted-by: Codex:gpt-6 golangci-lint Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
e494033607 |
fix(distributed): finalize stalled model uploads (#11910)
A worker can retain all model bytes with an unfinished-upload marker. Retries then start at zero and repeatedly fail with HTTP 416. Verify the existing bytes and finalize same-file retries at full size. Reuse the normal integrity checks so corrupt content cannot be accepted. Add regression coverage and document worker recovery. Assisted-by: Codex:gpt-6 golangci-lint Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
88d19567b8 |
feat(faces): replay saved face enrollments (#11908)
Accept original embeddings and timestamps so clients can restore faces when the in-memory store restarts. Derive stable IDs from exact vectors to make registration retries preserve identity without duplicate entries. Assisted-by: Codex:GPT-6 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
fb8b7a359a |
fix(distributed): stage sound detection audio (#11907)
* fix(distributed): stage sound detection audio Sound detection passes frontend temporary paths directly to remote workers, unlike transcription. Stage the WAV before classification so CED can read it without a shared temporary directory. Preserve the original request for retries and propagate staging errors without calling the backend. Cover staging, request preservation, and error handling with regression tests. Assisted-by: Codex:GPT-6 golangci-lint * test(distributed): verify routed sound staging Call sound detection through the client returned by SmartRouter.Route. This checks interface dispatch through both routing wrappers, rather than constructing FileStagingClient directly. The test fails without the sound-staging override and passes with it. Assisted-by: Codex:GPT-6 golangci-lint --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Tai An
|
37e18a4bf3 |
fix(quantization): record job progress without an attached listener (#11874)
A quantization job that runs with no client on its progress stream stays "queued" forever, in the API and in state.json, while the finished artifact sits on disk. state.json was written once by StartJob, and the only code that advanced a job afterwards lived inside the stream callback of StreamProgress, so job state depended on somebody watching it. The backend's progress stream cannot simply gain a second reader: each job owns one queue.Queue and QuantizationProgress pops from it, so two consumers split the updates rather than both seeing them. The stream has to be opened exactly once per job. StartJob now starts watchProgress on the application context (the request context is done as soon as the handler returns). That goroutine is the single reader: it applies each update to the job -- in the cross-replica store and in state.json, terminal statuses still winning over late updates -- and republishes it in-process. StreamProgress becomes a pure reader over that fan-out and no longer loads a backend or opens a stream. A client attaching to a job that has already finished, including one hydrated from disk after a restart, gets a final event built from the stored job instead of blocking. Two paths used to end a client's stream by breaking the gRPC connection and now release it explicitly: StopJob kills the backend, so it publishes the stopped event itself; and a stream that ends without a terminal update means the backend is gone, so the job is recorded as failed rather than left running forever. Signed-off-by: Tai An <antai12232931@outlook.com> |
||
|
localai-org-maint-botandEttore Di Giacinto
|
9afe10ba21 |
fix(distributed): survive a slow control-plane database (#11837)
* fix(distributed): evict only when a node is known to be full scheduleNewModel asked the registry for a free replica slot and treated every error as "this node is full", so a control-plane database slow enough to time out the lookup evicted a healthy loaded model. The evicted process died, a peer frontend still holding its address dialled the dead port and retried, and the model thrashed between nodes. The comment on the branch already said it meant a full node; the code never tested for it. Evict only on ErrNoFreeSlot. Any other error now returns and names the lookup that failed, so a slow database degrades into a diagnosable load failure instead of into lost work. An audit of the rest of the router found one branch of the same shape: node selection discarded the error from its last-resort finder, so a database timeout there also produced a nil node and evicted for it. That path now returns unless the finder said gorm.ErrRecordNotFound, which is the only answer that means the cluster had no node to give. No other destructive branch in router.go fires on a generic error. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): checkpoint heartbeat writes instead of writing every beat Every heartbeat UPDATEd backend_nodes. Six nodes at a ten second beat is roughly 52,000 writes a day against a six-row table, and that churn is what turned a blocked autovacuum into a 460 MB table whose six-row scan cost 867 ms and timed out the queries that place models. A beat carrying only a fresher timestamp now waits for the checkpoint interval. Each reported field is compared against the value last persisted rather than tested for presence, because a worker sends its disk figures on every beat and presence alone would suppress nothing. A node's first beat, a changed total VRAM, total disk or GPU vendor, and a free VRAM, RAM or disk reading that has moved more than 256 MiB from the persisted value all still write at once. A node that is not active is never suppressed, because it recovers only when the health monitor sees a fresh timestamp. The persisted column is up to one interval stale by design, so the stale-node threshold moves from 60s to 5m to cover it. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): fail worker readiness when a held backend is unreachable The readiness gate tracked only the NATS link, so a worker whose backend processes had died still answered /readyz with 200 and kept receiving loads. One node did exactly that during an incident: it reported healthy while its backend port refused connections, and every load routed to it failed. Readiness is now the NATS link and, for each backend process the worker believes it is running, a short dial of its recorded address. A worker holding no backends stays ready, because idle is a healthy state. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): keep a starting backend out of the readiness dial set A backend process is inserted into the supervisor map with its gRPC address already recorded, but the address refuses connections until the gRPC server binds, which the startup poll allows up to 30 seconds for and which takes 10 to 15 seconds on a slow node. The new data-path readiness probe dialled that address straight away, so a worker answered /readyz with 503 for the whole of every cold backend start. The container HEALTHCHECK absorbs that, but a Kubernetes readinessProbe at 10s does not, and the worker would leave rotation each time it loaded a model. The skip for a stopping process had no counterpart at the other end of the lifecycle. Backend processes now carry a serving flag, set where the startup health-check gate succeeds, and the probe dials only processes that are serving and not yet stopping. backendStartStillValid becomes markBackendServing: the check and the mark must share one lock hold, so the flag can only ever land on the entry the key currently owns. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): export control-plane database health gauges Four transactions wedged on a corrupt index held the vacuum horizon open for 42 days. Nothing measured it, so the first symptom anyone saw was models failing to load six weeks later, by which time a six-row table had grown to 460 MB. Export the oldest xmin age, the longest open transaction, and the dead tuple ratio on the registry tables. The first is the number that would have caught it: it sits near zero in health and was 21,002,291. Sampling is scrape-driven behind a cache, and a failed sample reports the last good values rather than failing the scrape, because these gauges matter most when the database is already struggling. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): rate-limit failed control-plane database samples The cache advanced its clock only on a successful sample, so once the database started failing every scrape retried the query immediately. That turned the cache off in the one regime it exists for: a retry storm at scrape cadence aimed at a database already in trouble. A catalog read that consistently exceeds the 5 second timeout also paid that cost on every scrape, with all scrapes serialised behind the sampler mutex. Time every attempt rather than every success, so failures and timeouts cost the same interval as good samples. Whether a good sample exists moves to its own field, keeping the gauges absent until the first success and holding the last good values through later failures. Also note in the runbook that pg_stat_activity cannot see prepared transactions or replication slot xmins, so a healthy-looking xmin age does not by itself rule out a blocked horizon. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(distributed): pin that a failing database evicts nothing Exercises the real distributed stack against a control-plane database that refuses the router's slot lookup, and asserts the scheduler reports the lookup it could not answer instead of falling through to eviction. The failure is injected with privileges rather than a statement timeout. A timeout set with ALTER DATABASE also breaks AutoMigrate, and it leaks into every later spec in the suite unless it is reset, so the spec would end up testing the migration rather than the scheduler. Instead the spec creates a dedicated login role, points a second gorm handle at it, and revokes that role's SELECT on node_models.replica_index. This has to be a separate role: the test container's owner is a PostgreSQL superuser, and superusers bypass every privilege check, so revoking from CURRENT_USER is recorded and then ignored. The revoke is scoped to one column on purpose. Revoking the whole table would also blind node selection, which runs first and has a guard of its own, so the scheduler would never reach the slot lookup this spec is about. Leaving every other column readable lets selection succeed and lands the refusal exactly on NextFreeReplicaIndex, which plucks replica_index. The grant is restored from BeforeEach via DeferCleanup, so a failing assertion or a panic cannot hand the next spec a role that cannot read. Reverting the eviction guard fails this spec, which is the point of it: the router then reports "no replica slot on keeper and eviction failed" for an error that was never evidence the node was full. The surviving-row assertions are secondary under this injection, because the eviction path reads whole node_models rows and the same revoke blinds it too; a comment in the spec says so, so nobody mistakes them for the load-bearing ones. Also documents why the vector store and the control plane must not share a database: the removable-tuple cutoff is per database, not per table, so one transaction left open anywhere stops autovacuum reclaiming the node registry, and a six-row table bloats into hundreds of megabytes. The note names LOCALAI_AUTH_DATABASE_URL and LOCALAI_AGENT_POOL_DATABASE_URL as the two knobs that must differ, and the localai_control_plane_oldest_xmin_age gauge as the way to see it coming. grep for StaleNodeThreshold and HealthCheckInterval in core/config/runtime_settings_registry.go returns no matches: the distributed duration knobs are not exposed as runtime settings, so the new heartbeat checkpoint interval follows them and needs no registry entry. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): close the review gaps in the heartbeat and health path The stale-node threshold moved from 60 seconds to 5 minutes in this branch because checkpointing makes last_heartbeat up to one checkpoint interval behind by design. Two things were left inconsistent with that. NewHealthMonitor still fell back to a hardcoded 60 seconds when handed a zero threshold, so any future caller that stopped passing the configured value would mark every healthy, beating node offline on every cycle. And the threshold itself had a flag-name constant but no AppOption, no CLI field and no env binding, so an operator who widened --node-heartbeat-checkpoint had no way to widen the threshold to match. The fallback now tracks config.DefaultStaleNodeThreshold, and --stale-node-threshold / LOCALAI_STALE_NODE_THRESHOLD is wired the same way its sibling is. Heartbeat suppression compared the RAW reported free VRAM against the snapshot, but the column persists capAvailable(raw, ceiling). On any node with a VRAM budget set, whose actual free VRAM oscillates above that ceiling, every beat looked material while the persisted value never moved: suppression was defeated on exactly the nodes an operator had configured, and the write amplification this branch exists to remove came straight back there. The comparison and the snapshot now both hold the capped figure, so they measure the same quantity as the column. Fixing that needs the ceiling, and reading it cost a SELECT on every beat, including suppressed ones. The skip decision therefore moved ahead of the updates map and now reuses the ceiling cached on the last durable write, while the write path still re-reads it before capping anything. A ceiling that changed inside the checkpoint window can cost one extra or one late write; it cannot persist a wrong figure. A suppressed beat now costs no query at all. Also: the operations section now says to grant pg_read_all_stats to the LocalAI role, because PostgreSQL blanks backend_xmin and xact_start for sessions owned by other roles, and the transaction that wedged the horizon in the incident was a co-located vector store connecting as a different role, so without the grant the new gauge sees only our own sessions. The compose healthcheck comment now describes readiness covering the backend data path, and the control-plane gauge registration records the otel.SetMeterProvider ordering it depends on. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): resolve the gauge's table names through gorm The dead-tuple gauge queried pg_stat_user_tables against a hardcoded list of three table names. Those three do not agree on where their name comes from: BackendNode and NodeModel take gorm's default pluralisation, while GalleryOperationRecord overrides TableName, and gallery_operations already had a constant of its own that the list duplicated. A literal list keeps compiling after any of that moves, and the query then matches nothing. The failure is silent and it points the wrong way: a dead-tuple ratio that matched no rows reports the same numbers as a cluster with no bloat, so the gauge would look healthiest exactly when it had stopped working. Ask gorm what each model is stored as instead, which follows a TableName override and the default pluralisation alike. A spec pins that the override really is consulted: naive pluralisation of the type would give gallery_operation_records, so the resolution cannot quietly stop asking the model. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di GiacintoandClaude Opus 5
|
5555a3c569 |
fix(nodes): skip checksum sidecars when staging option dirs
stageDirectory and countStageableFiles already skip them, but stageOptionDir did not - and it is the path sherpa-onnx voices take for espeak-ng-data. The receiver writes "<file>.sha256" for every file it accepts, so staging the sidecars made it write sidecars for those in turn, one level deeper on every load. Observed on a live node: "<file>.sha256" repeated eleven times, 5077 junk files out of 7832 in the models dir, and still growing. Staging never finished, so vits-piper-it_IT-paola-sherpa stayed permanently "staging on node" and every realtime warmup needing that voice failed with the session then going silent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0142UfUh8HWxdim5JZqf8Tr6 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
80e3240f2d |
feat(distributed): key scheduling rules by a model alias (#11771)
Node placement and replica rules could only name a model, so an operator who pinned "llama3" to the GPU tier had to rewrite the rule whenever a different model took over that job. An alias already gives a stable name for whichever model serves it, and a rule on that name makes it a deployment slot: repoint the alias and the placement follows. A rule keeps the name the operator chose. Reads resolve that name through the config loader to the model the rule governs, so the reconciler counts, schedules and trims replicas of the target, and the router finds an alias-keyed rule from the target it is already routing. An alias that resolves to nothing governs nothing loadable, so the reconciler skips it and the write paths refuse it. A replica is shared by every name that resolves to it, so only one rule can decide where it runs. The REST and MCP write paths reject a rule whose target another rule already governs. A pair that arrives some other way, such as a seed file or an alias repointed onto a model that already has a rule, resolves in favour of the rule named after the model itself and then the oldest, and the rest are listed as shadowed. The eviction guard is the exception: it matches rules to replicas in raw SQL inside a locking transaction and cannot resolve an alias. It reads a stored target that the reconciler refreshes each tick, and falls back to the rule's own name when that target is empty. Assisted-by: Claude:claude-opus-5 golangci-lint eslint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
29899cd1e0 |
fix(ui): size model fit against the cluster and move node labels into the selector (#11765)
* fix(ui): move node labels into the scheduling selector field The scheduling page kept a node-label browser open above the rules whether or not anyone was writing one, while the field that actually needs labels, the rule's node selector, was two bare text inputs with no hint of what the cluster reports. The browser is gone. The selector's key input now completes against the label keys the cluster uses, and the value input offers only the values that key takes. The roster already loads for the page, so the suggestions cost no request, and a roster that fails to load costs the admin the hints and nothing else. Suggestions stay suggestions: a key no node reports yet still commits as typed, which is how an admin writes a rule before labelling the nodes for it. Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): size model fit against the cluster, not the frontend The models page asked the frontend how much memory a model may occupy. In distributed mode the frontend is usually a GPU-less pod while every model runs on a worker, so a fleet of GPU nodes was told it could only run the smallest CPU build. The variant picker's fits flag and its auto-selection came from the same place, as did the hardware recommendations. The registry now reports the largest single healthy backend node. The largest node, not the fleet total: a model loads into one node, so four 16GB workers are not a home for a 40GB model. An operator-set VRAM budget caps a node's contribution, because the scheduler refuses a load above that ceiling anyway, and a GPU node beats a CPU node holding more system RAM. GET /api/resources and GET /api/models carry this as an additional cluster object. Their aggregate and ram fields keep reporting the frontend's own hardware, which is what the resource monitor shows. Variant selection judges backends against the union of the capabilities present in the cluster, the way backend discovery already did. Every path degrades to the local host: no cluster object in single-node mode, and none when the registry cannot be read, so a hiccup narrows the answer back to single-node behaviour rather than marking the whole catalog too large. The verdicts now name the node they belong to, since a model fits somewhere or nowhere. Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Copilotandmudler
|
e7b83ef7c0 |
Fix flaky "tests-apple" CI job in modeladmin test suite (#11717)
* Initial plan * tests: raise default Eventually timeout for modeladmin suite to fix flaky macOS CI Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
Ettore Di Giacinto
|
f7ded96b1e |
fix(distributed): probe liveness on a subject every worker answers
The scheduler's liveness probe asks a worker a question over NATS and reads "no responders" as proof the worker is gone. That is only sound when every worker in the fleet subscribes to the subject asked. It asked models.running, which arrived in 4.6. A 4.5 worker is alive and serving, answers backend.list, and never subscribes to models.running, so the probe condemned it on every scheduling attempt and marked it unhealthy. A model pinned to such a node by its selector could then never be placed at all: on this cluster an embedding model pinned to the one Apple node was unschedulable for exactly this reason, while that node's log showed it handling backend.list throughout. Ask backend.list, which has been in the worker protocol far longer, and treat a worker that answers anything as alive. Only a node that reports no responders on every subject is absent, so adding a newer subject here can never condemn an older worker. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |