mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-12 22:33:54 -04:00
c19ed5ab3250faa3c29c2bf0941448312d4f35f7
763
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ettore Di Giacinto
|
c19ed5ab32 |
fix(cluster): stop a late request frame reading as the worker's verdict
Making a worker's refusal reaping evidence created a defect one layer along, at the producer. The worker refused a ReadStreamRequest failure with ErrStreamRequestInvalid and its own comment said "Includes the deadline above expiring", which was harmless while every refusal reached the frontend as "no route" and became a reap the moment one of them did not. So a request frame that had merely not ARRIVED yet was reported as a non-transient verdict about a backend. It is reachable on the relay path, which carries most production traffic: the worker's header timer starts when the OWNING replica opens the stream, while the frame is written by the DIALLING replica only after the relay's acceptance travels back to it, so a whole peer-link round trip runs inside that window, on a link this design deliberately loads with multi-gigabyte artifacts beside token streams. For a long-deadline caller the endpoint is ConnectionEvictingClient, which stops the model across the fleet. It also falsified the "neither clears on its own" argument that licensed the reap. There is now a fourth refusal, ErrStreamNotServed, for what a worker could not serve for a reason of its OWN. It is deliberately outside IsWorkerAnswer, so it reaches a consumer under the no-route umbrella and reaps nothing, which is the same treatment an unrecognised code already gets. Four producers move onto it: a request frame that timed out (a malformed one stays a verdict, because that is a frontend bug no retry fixes), both SetReadDeadline failures, which are facts about the stream and not about a target nothing has dialled yet, and WriteStreamRefusal's default for a reason nobody classified. classifyServiceFailure keeps ErrStreamTargetUnavailable as its default on purpose: inverting it would make errno enumeration the single point of failure for the reap, and a miss there is a row nothing can ever delete. What it gains is a deny-list of two causes that are provably this worker's own clock or its own context. Also: - The read-site caller-deadline guard in the handshake was unpinned: the existing seam spends the budget before the handshake starts, so only the write could ever fail. A spec whose deadline falls between the request and the reply pins it, and each guard now reddens on its own. - The documented worker-first failure line omitted the JSON error envelope the old frontend returns, so an operator grepping it found nothing. - The peer-link disclosure names the aimable per-session receive window in all four places, and LastDialErrorOf records why a third consumer must go through IsWorkerAnswer rather than roll its own list. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
d26263f9c0 |
fix(distributed): let a worker's own refusal be evidence about its backend
A worker that refuses a stream has answered, and cluster.Dial keeps the three tunnelproto sentinels out of the ErrNoRoute umbrella precisely so a consumer can act on that. No consumer did. Since workers stopped listening, a backend process that crashed on a healthy worker is no longer a dead listener's codes.Unavailable: the worker refuses the stream with ErrStreamTargetUnavailable, gRPC flattens it into Unavailable anyway, and nodes.unroutable reported the whole thing as "this frontend has no route". Every reap path then answered ProbeUnknown and left the row, so the replica slot never freed and at the default MaxReplicasPerModel=1 the only cleanup left was LRU eviction of models that were working. isWorkerAnswer is exported as cluster.IsWorkerAnswer, so the errors the dialer keeps out of the umbrella are by construction the errors the consumers treat as the worker answering. nodes.unroutable and pkg/model's transportFailure both use it; ConnectionEvictingClient, the site reached during inference, goes through transportFailure rather than asking the transport directly. A reply code this frontend does not recognise is still not an answer, so a newer worker's vocabulary costs a retry and not a replica. The reap guards keep the allow-list rather than requiring ErrNoRoute: an unrecognised dial error must mean "no route", never "the backend is gone". Also in this final pass over the branch: - Docs: recommend upgrading FRONTENDS first, with the symptom of each order. Workers-first fails now that a 4xx registration is a verdict rather than an outage, so an old frontend's "address is required for backend workers" makes each restarted worker exit and drains the fleet a node per restart. - Docs: LOCALAI_WORKER_TUNNEL=false is a fatal startup error, not a degraded mode, in both places that described it; and a frontend rollback needs every worker restarted, because re-registration force-clears the address columns. - A replica with no advertised address now says so every five minutes and names the workers only it can reach, instead of one startup warning for a cost paid for the life of the process. - callerRanOut's rule now holds at all three siblings, so an expired caller deadline stops reading as a broken tunnel; probeHealth's withdrawn reason for using the raw client is corrected; the dead DoOrCached is deleted and its coverage kept on DoOrCachedResult; sweepLeakedInFlight enumerates the outcomes that reach it. - The peer route's self-declared id is recorded as a phase-3 deferral, in the handler, in the isolation claim it narrows, and in the operator docs. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
0dc6ebd525 |
fix(cluster): stop blaming a peer for the caller's own expired deadline
Review round 1 on the end-to-end proof. Zero blocking items, eleven non-blocking, and three of them turned out to be production defects rather than notes on the report. The one that matters is a misclassification the phase is built to prevent. A dial carries the caller's deadline down to the socket, so when the budget runs out the socket's timer fires and the error travels back up through the WebSocket handshake and the multiplexer. The context's cancellation is a separate timer whose func the scheduler has to run before ctx.Err() stops returning nil, and nothing orders the two. Under contention the socket's error is back in PeerPool.Open first, ctx.Err() reads nil, and a peer that is listening and healthy is reported as ErrPeerUnreachable to a caller that simply ran out of time. An unreachable peer is a fact a caller may act on and an expired deadline is not, and core/services/nodes routes around a replica it is told is unreachable. callerRanOut answers that question in one place: ctx.Err() when it is set, and otherwise the wall clock against the caller's own deadline. That is sound because it is the same instant the socket compared itself against, so if the socket's timer fired this comparison is past it too. The ambiguous instant resolves towards the caller, which is the direction that never blames a peer. The spec that caught it, peerlink_test.go's "blames the caller's deadline", was red in three of seven -race runs and had been since Task 5, which is often enough to read as noise and is why single-run verification never saw it. Rather than leave the proof to a coin flip, a second spec makes the window deterministic: Open is handed a context whose deadline has passed and whose cancellation has not been delivered, against an address nothing is listening on, so the dial fails for real. It reddens without the fix. The peer link's yamux windows were applied to one end only. A receive window is advertised by the side that RECEIVES, so configuring the dialler alone tunes exactly one direction, and the direction left on the 256 KiB default is the one that carries a relayed model artifact INTO the replica that owns the worker's tunnel. That is the largest thing the link ever moves and it is the direction the load measurement exercises: the review read it as flowing toward the dialler and it does not. PeerLinkConfig is now exported and used on both ends. Measured, same box, 128 MiB staged through the relay against the same transfer without one: the relayed path cost 1.6x to 2.0x the direct path's transfer window before, and 1.06x to 1.25x after. The SSRF reachability spec could be fooled into reporting an SSRF that did not happen. It bound the victim on 127.0.0.2 at an ephemeral port and required 127.0.0.1 at the same port to refuse, so any other spec in the run holding that number made the dial succeed; red one run in seven, green five of five in isolation. It now picks from below the kernel's ephemeral range, the same fix the harness got for the adjacent-port collision. The rest are the specs and the report saying what they mean. Scenario 1's advertisement assertion could not tell "the worker advertises nothing" from "the JSON key moved", which matters because removing the advertisement is the change it covers. It was green against a renamed key. The roster now keeps the raw key set beside the decoded fields and the spec requires both keys present before reading them as empty. Scenario 4's refusal-body check was a four-way disjunction admitting bare "tunnel", "not connected" and "unroutable". Those alternatives were inert and each would be satisfied by refusals that say nothing about routing, in the one assertion the whole negative control rests on. It is "no route" alone. The head-of-line gate bounded the worst probe by the whole transfer window, which admits about eightfold degradation and loosens as the box slows. It is now half the window, plus a scale-free ratio against the worst probe under the SAME cold load with nothing to transfer, which is the control that isolates the transfer from the load. Not tighter than that, and the reason is measured rather than cautious: under a concurrent -race suite the worst relayed probe reached a fifth of its window, so a quarter-window gate would have had 1.2x of margin, and a spec that fails one run in three is worse than no spec. The report entry printed p90 and p99 off samples of twenty, where both land on the same element and p99 often lands on the max, so one number appeared three times under three names. A quantile is now printed only when the sample can separate it. Two claims in the report were wrong and are withdrawn rather than softened. Scenario 2's race is closed by the trailing re-read of the owner, not by the pre-assertion the report credited: a move to the non-owner mid-request would serve directly and still return 200, and only the trailing read reddens on it. And "the median request is unchanged" holds on this box and not on the reviewer's, where the relayed median rises up to 82% and p99 up to 3.5x. What survives on both is structural: the worst probe is a small fraction of the window in which bytes are moving, so the session interleaves rather than serialising. Sharing a session with a bulk transfer costs latency; it does not cost service. The disk footprint note undercounted, and the reviewer lost a run to a full disk on this box, so it is worth having right: two bulk models seeded into two frontends and staged to the worker is about 768 MiB, not 512 MiB. Left alone deliberately: the worker's backend port allocator still hands out ports without checking they are free, and its default range still overlaps the kernel's ephemeral range. It is confirmed, it is out of scope here, and it is being tracked as a named follow-up rather than fixed under an e2e task. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
1cf847f29e |
feat(distributed): stop workers listening, and stop them advertising
A worker now opens no listener on a routable interface and states no endpoint at registration. Backend processes and the file-transfer server bind loopback, and the frontend reaches both through the tunnel the worker dials. The bind address is built from loopbackHost, the same constant the tunnel's grpc tag dials, so "the worker binds where its tunnel dials" is one fact in one place rather than two literals that can drift. All three advertisement sites are closed, not one: the registration body, RegisterNodeRequest, and the per-backend address in the install reply. That third one was hiding a live bug. stopModelExact refuses a stop whose ExpectedAddress does not match what the worker recorded for the process. The worker recorded 127.0.0.1:port; handleBackendInstall reported advertiseHost:port; the router stored the reported one and sent it straight back. On any worker whose advertise host was not 127.0.0.1, every acknowledged model stop failed with an address mismatch. Nothing caught it because the e2e harness set LOCALAI_ADVERTISE_ADDR=127.0.0.1, which made the rewrite a no-op. Removing the rewrite makes the two strings the same by construction. The brief was wrong about two of the four functions it called dead. effectiveBasePort is the base of the backend port allocator and resolveHTTPAddr is the file server's bind address; deleting them would have deleted the port allocator and the file server. Only the two advertise* helpers were dead, and addr_test.go is rewritten rather than deleted, because the port arithmetic it pinned still needs pinning. NodeModel.Address survives with a narrowed meaning and is renamed WorkerLocalAddress, along with the install reply field that feeds it. The frontend still has to say WHICH backend process on a worker it means, and the port in this string is how it says it: it travels as a stream target and the worker dials its own loopback. The gorm column and the json key stay "address", so neither a migration nor an API break rides along. Every fall-back to the node's address is gone. installBackendOnNode now errors when a worker reports success without naming one, because substituting the now-always-empty node address would name an empty target, and the worker refuses that as an invalid stream, which is classified as the worker answering about its backend. That is the "a present worker reads as something it is not" class this phase forbids. DistributedModelStore.Range had the same shape and was already wrong: it built each remote model's client from the node's base gRPC port, never the port a backend process listens on, so Free and Status went to the wrong place. It uses the replica's address now. BackendNode.Address and HTTPAddress are kept but made provably inert: no writer, no reader that acts on them, and Register force-clears both on re-registration so an upgraded worker's stale advertisement does not outlive its own upgrade in the API and the Nodes page. Dropping the columns is a ~90-site edit across the specs, the e2e suite, the MCP dto and the UI; it is recorded as a follow-up rather than folded in here. A persistent tunnel 401 still does not trigger re-registration, and now for a reason rather than a deferral. Register CLEARS the node's replica rows, so re-registering on a 401 would delete a live worker's rows on every retry, and under the name collision that causes the 401 the two workers would take turns doing it forever: a credential failure causing model reclamation. It also cannot fix the named cause, since a collision is indistinguishable from a restart. The 401 log now names both causes and says nothing can reach this worker, which is true only now that it has no listener. The container healthcheck did not break the way the brief expected, since the listener still exists on loopback and the probe runs inside the container. It did have a real #10987 defect that this change makes the common case: it read LOCALAI_SERVE_ADDR only, while effectiveBasePort reads LOCALAI_ADDR first, so a worker on a non-default base port was probed on 50050 and reported unhealthy while working. It follows the same precedence now. Docs, the compose file and the e2e harness are updated in step: no inbound rule or published port is needed for a worker, the two advertise variables are gone, the remaining address variables are read for their port only, the firewall-the-file-transfer-port warning is narrowed to the LOCALAI_HTTP_ADDR opt-out, and the upgrade-order note no longer claims the worker still listens. The Nodes page showed node.address, which is now always blank, so it shows the node id instead. Eight mutations, all red on a named spec, including reverting the loopback bind, re-adding the address to the registration body, restoring both node-address fall-backs, dropping the force-clear, storing the endpoint's address again, and un-fixing the healthcheck. One of them caught a defect in a spec I had just written: it asserted 200 where the endpoint returns 201, which went unnoticed because core/http/endpoints/localai is not on the task's verify list. It is run here. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
a8ac2af167 |
fix(cluster): make "no route" a condition of its own, and let it out of the package
Review round 1 on task 6. Five blocking findings, all with the same root: the conditions the dialer kept apart were erased one layer out, because every one of them arrived at core/services/nodes as a gRPC codes.Unavailable, which is also what a backend process that died produces. Four call sites acted on that by deleting a replica row, one of them after a single failed probe. The fifth condition is ErrNoRoute: this replica could not get a request to a worker's backend, and no claim at all about the worker. A worker's presence is its HEARTBEAT, which nodes owns; a route is a separate fact that cluster owns, and the two now differ. They differ in normal operation, not exotically: a worker that has not dialled its tunnel yet after a frontend-first upgrade is unroutable on every request while it heartbeats and serves. Two properties, both mutation-tested. Every failure to resolve or open a route carries ErrNoRoute, so a consumer has one check to make. No failure carries an absence sentinel: routeFailure is the single place that rule lives, and it keeps ErrNoConnection and ErrInstanceNotFound in the message and out of the unwrap chain, the guarantee unreachableError already made for peers. Everything else stays matchable, so ErrNotOwner and ErrPeerUnreachable are unchanged for anyone who can act on them. A worker's own refusal carries no umbrella, because a worker that answers has demonstrated it is there and that is the only real evidence on the path. Crossing the boundary needed a value, not a code. NewClientWithDialer wraps the dialer and records each outcome; LastDialError hands it back behind a narrow interface, and nodes.unroutable turns it into ErrWorkerUnroutable with the cluster sentinels still in the chain. A spec asserts a dial failing with ErrNoRoute plus ErrPeerUnreachable arrives matching all three and matching neither absence sentinel. The sweep found a fourth site the review had not named: pkg/model checkIsLoaded evicts a remote model on a connection error, and a tunnel dial failure is one. Four other reap sites were cleared with reasons - inflight and the worker authoritative pass reap only on semantic answers, scale-down is driven by last_used, abandoned loads decide on the node's heartbeat. Every fixed site also grew the opposite spec, so the new check cannot pass by never reaping. probeCache carries the reason through singleflight rather than a closed-over variable. A variable is only written by the goroutine that runs the probe, so the leader would correctly decline to reap while every joiner reaped on the leader's own observation; a mutation reproduces exactly that. The docs sentence promising LOCALAI_WORKER_TUNNEL=false restores direct dialling is gone. There is no such path, so it said the operator could take a worker dark and call it a rollback. Replaced with the upgrade order that is actually safe. The deadline spec the reviewer found vacuous now waits on the dial context's own Done channel before touching the stream, so the armed deadline has really expired; the mutation that survived for the reviewer reddens it. Nine mutations, each reddening a named spec, including both halves of isAbsenceClaim independently. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
75953d9f63 |
feat(cluster): reach every worker through its tunnel, never its address
The tunnel, the fence, the registry and the relay were all built and none of them carried a byte: every dial from the frontend still went to the address a worker registered. This is where that stops. One WorkerDialer resolves where a worker's tunnel is held, opens a stream on it locally or relays through the owning replica, and hands back a conn past both handshakes; gRPC, the file stager's HTTP client and the log-streaming WebSocket are all pointed at it. A worker's address stops being somewhere to connect to and becomes the name of which backend process a stream is for. It still appears in URLs, logs and errors, because that is what identifies the process; what it no longer decides is where the bytes go. Nothing falls back to dialling it. BackendClientFactory now has exactly one method, NewClientForNode, and returns an error where there is no way to reach the worker. The direct-dial constructor was removed rather than kept beside it, because leaving one on the interface keeps the bypass one word away from every call site that holds an address, which is all of them. The second construction path is closed too. DistributedModelStore built remote models with a nil client, and pkg/model.Model.GRPC then dialled the raw address lazily on first use - reached in production by ShutdownModel's Free and by the backend monitor's Status. Those models now carry the tunnel-backed client, and a model that cannot be given one is logged and not listed. Four conditions stay unmixable, and one path produces absence: the dialer answers ErrNoConnection only where Owner's liveness join did. A peer that will not answer, a stale ownership row, a worker's own refusal and a missing relay path are each reported as themselves. This matters because nodes ACTS on absence, and the collapse would have it reclaim the models of a worker that is connected and busy. That is not hypothetical. Writing the mutation for it exposed the bug in this change's own first draft: probeHealth returned bare false when it could not build a client, and tryWarmPath deletes the replica row on a false probe. A frontend whose dialer broke would have emptied node_models for the whole deployment while every model kept running. probeHealth now returns alive and probed separately, the reconciler gets a ProbeUnknown outcome that neither advances nor clears a failure streak, and the health monitor skips rather than counting a miss. Task 5 left the relay's open timeout at a fixed 15s and said so: no operator has the information to set it, because the number that matters is the original client's remaining budget, which is invisible on the relay side. The dialer has that budget, so it now states it in the relay request frame and the owner takes the smaller of the two. It can only shorten - a patient client must not be able to park a relay goroutine and a stream slot on a worker that stopped accepting. Zero is written as no budget at all, since on the far side the number zero is a caller with nothing left and would refuse healthy traffic. Seven mutations, each reddening a named spec: peer-unreachable as absence; the local-failure guard dropped; max instead of min on the budget; the nil-client model restored; ProbeUnknown falling through to the reaper; OwnerRow instead of Owner; probed collapsed into alive. The first budget spec passed for the wrong reason - a handshake deadline, not the relay - and was replaced by three that each assert one link, including one where the spec plays the owning replica and reads the budget out of the frame instead of inferring it from a clock. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
3b6d32c1c4 |
fix(worker): make the tunnel credential's node-type gate actually structural
Re-review follow-up, three items. Two are the overclaiming-comment class again, and the first is that class with a real defect underneath it. attachTunnelToken said "enforcement is therefore structural": an ineligible node never gets a credential, so its hash stays empty and the tunnel route's empty-hash branch does the refusing. That was true for a node that had always been an agent and false for one that had not. Register upserts by NAME, so a backend node re-registering as an agent keeps its ID, and Register's struct Updates zero-skips the credential column while writing the new node_type. The early return left the credential the node earned as a backend sitting on a row that is now an agent, and ConnectHandler never looks at node_type. Fixed by making the claim true rather than by softening it, because the mint-site gate was chosen precisely on the grounds that it was structural: an ineligible node now has its column CLEARED, unconditionally, so the invariant does not depend on what the row happened to contain. A spec pins it and was red before the change. Same shape as the Register-upserts-by-name hazard already carried forward: a name is not an identity. Second, loopbackHost claimed to be the only host any tunnel stream is ever dialled on. It is not: fixedService dials whatever Run built it from, which is this worker's own LOCALAI_HTTP_ADDR, and loopbackAddr rewrites only a wildcard bind, so an operator who binds the file-transfer server to a routable address gets a routable dial. The property that matters is narrower and is what the comment says now: the frontend cannot STEER the dial. The grpc tag builds its address from a constant and a validated port with nothing from the wire reaching the dialler, and the http tag ignores its target entirely. Worth stating exactly rather than summarising, because the argument about what a stream can reach rests on knowing which hosts are reachable, and an overstatement at that site is what would let someone conclude the constant alone is doing the work. Third, a spec named "without allocating it" measured no allocation. It now asserts the mechanism the defence actually rests on, that the reader consumes the two length bytes and not one byte of the body, through a counting reader. The input carries a body on purpose: against input that ends after the header the assertion would pass with the limit check deleted. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] |
||
|
Ettore Di Giacinto
|
5108be222d |
fix(worker): spec the tunnel's routing table, which was the SSRF boundary
Review follow-up. One blocking finding and seven others. The blocking one first, and it is this project's recurring shape: the untested path. loopbackService is the function whose comment calls the discarded host "the security property this function exists for", and nothing tested it. The reviewer replaced its body with a dial of whatever the frontend named, no port range, and all 131 specs passed. Every spec installed the permissive test dialler, so the real routing table was exercised nowhere. It now has specs, and the property is stated as reachability rather than as a property of the code: a listener on 127.0.0.2 that only the frontend's target names must NOT be reached. Plus the port-range table, fixedService, loopbackAddr, tunnelEndpoint, and the table itself, which moved out of Run into tunnelServices so it can be built without starting a worker. One spec drives a real stream through that table over the wire, so the routing rules are exercised end to end at least once rather than only in isolation. The reviewer's mutation now reddens ten specs, and six narrower ones redden between two and four each, so no spec is riding on another. The shape changed too, not only the coverage. The dial address is built from a loopbackHost constant and strconv.Itoa of a validated int, so nothing derived from the wire reaches DialContext at all: restoring the hole takes ADDING a data flow, not deleting a check. And a taxonomy fix found while specifying it. A port outside this worker's allocator range was reported as unavailable, which tells a frontend to retry something that can never work. It is a bad request now, and a backend that is merely not listening yet stays unavailable, which is the retryable one. Agent nodes no longer get a tunnel credential. Nothing dials into an agent worker, so a tunnel replaces nothing for it and no client would open one, and the gate is at the mint site rather than in the handler: with no credential minted the hash stays empty and the existing empty-hash refusal covers it, so enforcement is structural. Two comments and one doc paragraph said an anonymous registrant gets a "working" credential. With auto-approve off the node is pending and the credential is inert, which is the distinction this same change argues three files away to justify minting for pending nodes at all. A refusal reason over the frame limit was cut on a byte boundary and could split a rune. It cuts on a rune boundary now, and the code survives truncation, which is what keeps a refusal classifiable. Also: the pending-node spec asserted only that a credential was non-empty, so a credential derived from the shared token passed it; it now pins per-node-ness the way the headline spec does. The tunnel handler's citations into nodes.go were stale before this branch landed, having been written against a file the same commit was editing, and are by function name now. The static-NATS path says plainly that an externally forced rotation locks it out until restart, and where that gets fixed. tunnelproto gained direct specs, including that a read failure is never reported as a refusal. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] |
||
|
Ettore Di Giacinto
|
29a2020f3d |
feat(worker): dial, hold and serve the tunnel, on a credential of its own
The worker end of the tunnel. It dials wss://<register-to>/api/cluster/connect, holds one yamux session as the CLIENT, and serves every stream the frontend opens on it. Nothing dials into the worker, which is the point: no inbound port, no reachable address. Each stream opens with a length-prefixed frame naming a tag and a target, and the worker answers before either side speaks the tunnelled protocol. The reply is sent on every stream, not only on refusal, because the protocols carried here are client-speaks-first and a reply sent only sometimes would arrive interleaved with a response body. Two tags today: grpc reaches a backend process, and only on 127.0.0.1 within this worker's own backend port range, because a tunnel terminates inside the worker and letting the frontend name a host would make every worker a proxy into its own LAN; http reaches the worker's file-transfer server, whose address the frontend is not asked about. An unknown tag, an unreachable local service and an unparseable request are three refusals and stay three on the wire. A frontend gives up on the first and retries the second. Each is answered AND the stream is ended: a worker that says why and leaves the stream open has parked the caller on a request nobody will answer, and a deadline on the far side cannot tell that from a slow worker. The specs assert the stream ends rather than that an error occurred, which is what phase 1 shipped in three places and held in none. Reconnects double from 500ms to a 30s ceiling, each wait drawn between half the interval and all of it, and the interval returns to its floor only after a session that LASTED. Resetting on connect is how a rolling restart, where every dial succeeds and dies moments later, becomes a retry storm against the first replica back up. Nothing is assumed to survive a reconnect: the credential is read at dial time, never captured. And the credential is now real. The tunnel endpoint advertised authenticating a worker against its own secret, but registration stored the hash of the shared registration token, so a leak plus a known node ID still opened a tunnel. Registration now mints a per-node secret, returns the plaintext once as tunnel_token, and stores only its SHA-256 in a new column; the endpoint compares against that and does not fall back to the old one. Rotating on every registration follows from storing only the hash, since a re-registering worker cannot be told the secret it already holds; its live tunnel is unaffected, because the credential is checked when a tunnel is dialled and never again. Unlike the agent API key and the NATS JWT next to it, the credential IS issued to a node awaiting approval: the tunnel route re-reads the node's status on every dial and refuses a pending one, so it is inert until an admin acts, and withholding it would strand every worker that registers exactly once. A node that has not registered since this change cannot tunnel, and the column cannot be back-filled because the plaintext only ever existed in the response that minted it. The boot warning that said tunnels need LOCALAI_REGISTRATION_TOKEN is replaced: it was true while the tunnel authenticated against that token's hash, and says the wrong thing now. What is still true, and is what it warns about instead, is that without one, registration itself is unauthenticated. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
a816bf9b84 |
fix(testutil): stop the shared-database change from disarming two regressions
Two advisory-lock specs named their database by literal, ALTER DATABASE testdb. Once the test helper started handing every spec its own database on a shared server, that statement landed on the maintenance database and did nothing to the one the spec was holding, so both specs went green having never reproduced the condition they exist for. They regress a model-load advisory-lock wedge that has already shipped to production once, so the previous commit's de-flaking silently disarmed a regression test for a real deployed bug. Both sites now read the name back with current_database() and, more importantly, assert the override actually landed before relying on it. A literal name can go stale again; an assertion that the setting is in force cannot pass while it is not. Removing either production override now fails the matching spec with the real 55P03 and 57014 again. That literal also meant every CREATE DATABASE and every DROP ... WITH (FORCE) ran under the 300ms bound it set on the maintenance database, which is a new load-dependent single-spec flake inside the change that was meant to remove one. The helper's maintenance connections now pin one connection and clear both timeouts on it, so no setting a spec makes can bound them, and a white-box spec imposes the leak deliberately and proves it does not reach them. Also pins the reclaimOne gate deferral the previous commit added without a test, by panicking inside the re-claim's own claim statement, and drops the per-dial empty-token log line to debug now that the boot warning says it once. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
48ece89c63 |
fix(cluster): harden the worker tunnel, and stop starting a database per spec
Review follow-up. Twelve findings, none blocking, grouped here by what they protect. Panics. The handler now recovers between the WebSocket upgrade and the hand-off, the way the peer link next door already did: net/http recovers the panic but leaves the hijacked socket open, so without this a worker keeps a session this replica has no entry for and will never detach. The claim gate in Attach and reclaimOne is now released with defer, so a panic under Claim cannot wedge one node's gate for the life of the process. SetTunnels gained the nil-receiver guard its sibling Stop has. Operability. A deployment with no registration token stores an empty token_hash on every worker, so every tunnel dial 401s forever on a frontend that looks correctly configured. That now warns at startup, logs its own line rather than sharing the "wrong token" one, and is stated in the docs together with the fact that setting the token later needs the workers to register again. Authorization. A node still awaiting admin approval is refused with 403. The rest of /api/node/ gates on nothing, but the two places that hand a node something durable, its API key and its NATS credential, both refuse a pending one, and a tunnel is that kind of grant. Draining and unhealthy nodes keep their tunnels on purpose. Comments that claimed more than the code. The global auth middleware does run on this path and then declines to reject; the future per-node secret only lands without a change here if it lands in TokenHash; the empty-hash guard is defensive rather than deciding; ClusterPathPrefix is no longer only replica-to-replica; the docs no longer say a reaped replica re-claims unconditionally. And the test harness. SetupTestDB started a PostgreSQL container per BeforeEach with a readiness deadline it asserted on, which is one chance per spec to fail one spec inside its setup, anywhere, never twice in the same place: the shape of the flake seen twice here and never reproduced. It now starts one container per process and creates a database per call, which is the pattern tests/e2e already proved. Isolation is unchanged and is now asserted for the first time. All 69 call sites are untouched; the eleven consumer packages run 1404 specs green, and jobs went from 34.3s to 3.3s, agents from 13.8s to 1.9s, cluster from 97.4s to 37.5s. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] |
||
|
Ettore Di Giacinto
|
6e55092a4b |
feat(cluster): open the door a worker dials its tunnel through
A worker needs no inbound port: it dials GET /api/cluster/connect, the connection becomes one multiplexed yamux session, and the frontend opens a stream on it per request. This adds the endpoint that accepts that dial and attaches it to the tunnel registry. The dial is authenticated against the NODE's own stored token hash rather than the deployment's registration token. That is the mechanism, not yet the isolation, since a worker still registers by presenting the shared token; what it rules out is the shortcut of comparing against the configured value, which would have to be unpicked the day workers get their own secrets. Every refusal happens BEFORE the WebSocket upgrade, so a dialer reads an HTTP status rather than a handshake error. The route is registered in every deployment, single-binary ones included, which is what puts it in front of the route-coverage test that holds that rule in place; with no node registry it refuses every dial, and tells a credentialed one the frontend has no cluster rather than that its token is wrong. A lookup that FAILED is answered as a failure. Reporting a database that could not be read as "unauthorized" would send a worker re-registering, throwing away the identity its tunnel and loaded models are keyed by. Wires the tunnel registry in core/application/distributed.go and hands it to the membership loop. Without that call the re-claim after a replica is reaped had no production caller and could never run. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] |
||
|
Ettore Di Giacinto
|
0d13056d53 |
fix(cluster): share one lock order, and correct the phase 1 comments
ReapStale deleted from instances then node_connections while Deregister took them the other way round, both inside one transaction and both running concurrently by design: a replica shuts down while a peer sweeps it. Opposite orders let each hold the row the other waits for. PostgreSQL breaks the cycle by aborting one side, so the cost today is a warning rather than lost data, but the inversion costs nothing to remove. Deregister now deletes the instance row first. That is the order ReapStale is forced into anyway, since its connection delete asks which instance rows survived, so the sweeper is the fixed side. Both functions say the order is deliberate and shared, and name the other. A spec records the statements each path issues and asserts they delete from the same two tables in the same order; racing two transactions until they really deadlock would be flaky and could pass for the wrong reason. The rest is comment and spec accuracy, deferred from the phase 1 task reviews: - co-location does not imply loopback. Compose's usual host=postgres resolves to a bridge address and discovery works there; it is a DSN that NAMES localhost that yields a loopback source address. Corrected in the DiscoverAdvertisedAddr doc and in the spec comment that repeated it. - unroutableReason labelled every scoped address "link-local", including the class the check exists for, and formatted the IP with %s, which drops the %iface, so the reported address was not the one being rejected. Split into two cases, both rendered with their zone. CheckAdvertisedAddr passed zone "" and net.ParseIP rejects fe80::1%eth0, so a scoped literal looked like a name and collected no warning at all; the zone is now split off before parsing. - Splice's "Both callers satisfy it" claimed callers that still do not exist. It now names the two stream types the wake-on-Close property was verified against and says a phase 2 caller over anything else has to check it. - restored, short, why a socket-level ECONNRESET stays reported while a yamux reset does not: the yamux endings are the teardown Splice's own Close provokes, and whether an aborted request is routine is the relay's policy. - the real-yamux spec's far.Read had no deadline, so a stall parked the suite rather than failing it. - gorilla's SetWriteDeadline is conn.go:796, not 787. - ClusterPathPrefix is no longer derived from: the peer route spells its path out, because core/services/cluster must not import core/http/auth. The comment now points at the spec that holds them together instead of claiming a derivation the move removed. - the epoch spec asserted e2 > e1, an ordering Claim's doc tells callers not to rely on. It asserts uniqueness, which is what the fence guarantees, and is named for that. A sibling spec still described the epoch as incrementing in SQL when it is drawn from a sequence. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
e26d556594 |
fix(cluster): hold the guarantees phase 1's comments were claiming
Review found the recurring class: assertions that a wrong implementation also satisfies. The "refuse promptly, never park the peer" guarantee was stated in three places and tested in none. Removing the Close from the no-relay branch left the whole cluster suite green, because the specs asserted only that some error arrived and yamux reports a read deadline as ErrTimeout: a parked stream satisfied that as well as a refused one. Both specs now require an ENDING, EOF or a reset, inside a deadline short enough that parking is unmistakable, and both go red when the Close is removed. Deregistration existed only in a comment. Membership.Stop ended the loop and left the row behind, so every clean rolling restart had peers dialling a corpse for the full liveness window; the shutdown comment described the opposite. Registry.Deregister deletes the row and the connections that replica owned, in one transaction, for the reason the sweeper does both, and an e2e spec pins departure inside a budget shorter than the liveness window so it cannot pass on the sweeper doing the work. Before: the spec times out with both replicas still live. After: 3.6s. The configured advertised address bypassed every check discovery makes, so the one value most likely to be copied between hosts, 127.0.0.1, was taken verbatim and would make every peer dial itself. Both paths now share one rejection rule: unparseable is refused, "this host" is warned about once and honoured, because a single-host deployment uses it correctly. Two comments claimed more than the code does. The sweeper said a stalled replica recovers via re-register; only its instance row does, while the connections another replica reaped stay gone and the sockets stay held here - phase 2 must re-claim, on re-register, every connection a replica still holds locally. And Owner became OwnerRow, documenting that the owner it names may be dead for up to InstanceLiveness plus a heartbeat and that any caller acting on it must join instances itself, so the deferred constraint lives at the call site rather than in a report; the plain name is left free for the joining version. Minors: warn once when the peer link mounts with no registration token, so an operator sees the cause rather than 401s; Stop no longer blocks forever when Start was never called; corrected the NewRegistry migration doc and an e2e comment that described a 6s window as "throughout". Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
aca383d263 |
feat(cluster): give phase 1 a call site, and prove it against real replicas
Tasks 1 to 5 built an instances table, a splice, both halves of a peer link and an epoch fence, and nothing in the tree called any of it: no replica registered, no route was mounted, no sweeper ran. Proving phase 1 end to end therefore had to start by wiring it. A frontend in distributed mode now publishes the address its peers dial, heartbeats it, and sweeps replicas that stopped answering along with the connection rows they owned, in one pass so the two can never disagree about who is alive. It serves the peer link and owns the sessions peers dial in, refusing streams on them until phase 2 installs a relay: a session nobody accepts on does not fail a peer's Open, it hangs it. The address is the one peers use, not the one the process binds, and it is derived from the route to PostgreSQL. That derivation only holds while the database is remote, so LOCALAI_DISTRIBUTED_ADVERTISE_ADDR sets it explicitly and a replica that can determine neither warns and keeps serving rather than failing to start. Three e2e scenarios run against real local-ai processes, real PostgreSQL and real dials: replicas publish addresses that can actually be connected to; a sibling opens a stream over the peer link and is refused without the cluster token; and a killed replica is reported unreachable, never absent, loses the claim it held, and takes no worker with it. Each was verified by mutation: eight injected defects, each failing the scenario that claims to catch it. Also moves RegisterClusterRoutes to core/http/routes beside every other registrar, folds AutoMigrate and the epoch sequence into one cluster.Migrate, and turns the peer route's auth-coverage spec into a real assertion: it drives the request through the actual auth middleware instead of comparing two string constants, which the old spec would have passed even with the exemption deleted. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
d73730b545 |
refactor(cluster): make the cluster service a leaf and blame the caller's deadline
The peer link's WebSocket adapter and route constant lived in core/http/endpoints/cluster, so the dialler in core/services/cluster had to import an HTTP endpoints package to reach them. That pulled echo, core/http/auth and core/config into a package whose doc says it is deliberately free of such dependencies, and it made core/services/nodes reach an endpoints package transitively. It also has no way forward: the worker-connect handler needs the tunnel registry and the node token store, both of which are cycles from there. Move WebsocketConn and PeerPath into core/services/cluster and let the endpoints package import it, which is the direction the rest of core/http flows. The route and the auth exemption still cannot drift apart, now asserted where both are visible rather than by a const reference across the boundary, and the assertion is stronger than the one it replaces: it pins the route under the prefix instead of pinning the prefix's spelling. Also guard the fresh-dial path with ctx.Err(), mirroring the cached path. A caller with a 300ms deadline dialling a live, listening peer was told the peer was unreachable, which would be enough for one impatient client to get a healthy replica routed around once the relay consults these errors. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
78958beef1 |
fix(cluster): own the peer auth prefix in auth, and pin what the specs claimed
The prefix constant moves to core/http/auth beside the check that uses it, and the endpoints package derives its route from there. Seven sibling endpoint packages already import auth, so the previous direction would have deadlocked the build as soon as this one registered in RouteFeatureRegistry, and it was dragging echo, gorilla/websocket and yamux into unrelated service packages. Four properties were argued in comments and held by nothing. Flipping the empty-token check to fail open, making SetWriteDeadline a no-op, returning a zero-length read for a zero-length message, and dropping the recover around the callback all left the suite green. Each now fails a spec that asserts the behaviour rather than the setter's return value. SetWriteDeadline takes the write mutex because gorilla keeps that deadline in a plain struct field applied at the next flush; SetReadDeadline must not take the read mutex, since it goes straight to the net.Conn and would otherwise block behind the read it exists to unblock. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
e957ff1ca2 |
feat(cluster): accept authenticated peer links on /api/cluster/peer
Upgrades to a WebSocket, wraps it as a yamux server session and hands it to the caller. Rejects before upgrading so an unauthenticated dial sees a 401 rather than a WebSocket error, which is what the route-coverage test asserts. The adapter keeps the reader of a partially consumed message across Read calls. yamux reads through a 4 KiB bufio.Reader, so a small-payload test cannot see a dropped message tail; the framing specs drive the adapter directly with buffers smaller than the message. An empty configured token authorizes nobody here, unlike the worker file transfer server's check: this route is registered in every deployment, so failing open would publish an unauthenticated mux. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
c5796d407f |
test(distributed): correct the claims the e2e comments make
Review of the whole branch found five comments that would send a reader to the wrong place, plus three smaller inaccuracies. Nothing here changes behaviour. The KNOWN RACE note on both backend-log WebSocket handlers said the fix needs an atomic snapshot-plus-subscribe "under the store lock". It does not: BackendLogStore.mu guards only the buffers map, and AppendLine enqueues and fans out under the per-buffer buf.mu. Whoever took the store lock would ship and the race would survive, so both notes now name buf.mu and say what s.mu does and does not exclude. Two comments in the cluster harness quoted Eventually(c.FrontendAlive) .Should(BeFalse()). FrontendAlive takes an index, so Gomega rejects that with "requested 1 arguments but received 0". Both now quote the closure form the specs actually use, and say why the closure is needed. proveHealthCheckingIsAlive claimed to prove the health monitor ran for the whole preceding window. It proves the monitor was alive at the end of it, and inferring backwards needs any wedge to be sticky. In the peer-replica-death spec that inverts: health checks are single-flighted by a session-scoped pg_try_advisory_lock, the spec SIGKILLs the replica that may hold it, and until Postgres reaps the session the survivor acquires nothing and checks nothing silently. Consistently(healthy) can then pass because nothing was checking, with the positive control still succeeding once the lock frees. The doc now states what is proven, names that gap, and says the assertion is a floor rather than a proof. The Makefile still called DISTRIBUTED_TEST_FLAKES a retry count, which is what seeded that error into the two docs just corrected against it, and the workflow called the 15s window a reconcile tick when the mechanism is HealthCheckInterval in the node health monitor. Also: the cluster suite measured 509.1s / 509.8s / 512.3s, so about 8m30s and not the 8m39s/8m40s three files claimed; the dead-worker spec title implied two independent detectors when both probes read one advisory-lock-serialised verdict out of the same row; and the sanitizeDBName length assertion used <= 50, which an empty string also satisfies, where the invariant for an over-long input is exactly 50. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
53639c4df3 |
test(distributed): scope the log-subscriber wait and mark the race it works around
Three corrections from review of the previous commit. The lock-order comment on SubscriberCount claimed no path takes s.mu and a buffer lock together. Subscribe does exactly that, holding s.mu.RLock across replica registrations that take buf.mu. State the rule that is actually true — s.mu precedes any buffer lock, so counting after releasing it preserves the order — and say what follows from it: the total is a sample, not a snapshot. waitForLogSubscriber read as general-purpose but unblocks on the first registered subscription. Subscribe attaches the exact-key buffer and each replica buffer one at a time, so for a replicated model the count goes positive while later replicas are still unattached and the race survives. Rename it waitForSingleLogSubscriber, document that it holds only where Subscribe resolves to one buffer, and assert on exactly 1: misuse then fails loudly on the count rather than going quietly back to being flaky. Taking the expected count as a parameter was the alternative, but that makes callers predict a store-internal number and an under-count fails the same silent way as the original bug. The snapshot-then-subscribe race had no artifact outside a report, and review found a second site carrying it. Mark both handlers identically, including the point that swapping the two calls duplicates rather than drops and so is not the fix. The race itself is left alone; this branch stays test infrastructure. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
893a45141c |
fix(realtime): accept GA WebRTC signaling (#11778)
OpenAI GA clients send multipart or raw SDP requests. They expect a bare SDP answer. LocalAI only accepted its legacy JSON envelope, so signaling failed before media setup. Keep the JSON contract for existing clients. Accept both GA request shapes and choose the matching response format. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
80e3240f2d |
feat(distributed): key scheduling rules by a model alias (#11771)
Node placement and replica rules could only name a model, so an operator who pinned "llama3" to the GPU tier had to rewrite the rule whenever a different model took over that job. An alias already gives a stable name for whichever model serves it, and a rule on that name makes it a deployment slot: repoint the alias and the placement follows. A rule keeps the name the operator chose. Reads resolve that name through the config loader to the model the rule governs, so the reconciler counts, schedules and trims replicas of the target, and the router finds an alias-keyed rule from the target it is already routing. An alias that resolves to nothing governs nothing loadable, so the reconciler skips it and the write paths refuse it. A replica is shared by every name that resolves to it, so only one rule can decide where it runs. The REST and MCP write paths reject a rule whose target another rule already governs. A pair that arrives some other way, such as a seed file or an alias repointed onto a model that already has a rule, resolves in favour of the rule named after the model itself and then the oldest, and the rest are listed as shadowed. The eviction guard is the exception: it matches rules to replicas in raw SQL inside a locking transaction and cannot resolve an alias. It reads a stored target that the reconciler refreshes each tick, and falls back to the rule's own name when that target is empty. Assisted-by: Claude:claude-opus-5 golangci-lint eslint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
29899cd1e0 |
fix(ui): size model fit against the cluster and move node labels into the selector (#11765)
* fix(ui): move node labels into the scheduling selector field The scheduling page kept a node-label browser open above the rules whether or not anyone was writing one, while the field that actually needs labels, the rule's node selector, was two bare text inputs with no hint of what the cluster reports. The browser is gone. The selector's key input now completes against the label keys the cluster uses, and the value input offers only the values that key takes. The roster already loads for the page, so the suggestions cost no request, and a roster that fails to load costs the admin the hints and nothing else. Suggestions stay suggestions: a key no node reports yet still commits as typed, which is how an admin writes a rule before labelling the nodes for it. Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): size model fit against the cluster, not the frontend The models page asked the frontend how much memory a model may occupy. In distributed mode the frontend is usually a GPU-less pod while every model runs on a worker, so a fleet of GPU nodes was told it could only run the smallest CPU build. The variant picker's fits flag and its auto-selection came from the same place, as did the hardware recommendations. The registry now reports the largest single healthy backend node. The largest node, not the fleet total: a model loads into one node, so four 16GB workers are not a home for a 40GB model. An operator-set VRAM budget caps a node's contribution, because the scheduler refuses a load above that ceiling anyway, and a GPU node beats a CPU node holding more system RAM. GET /api/resources and GET /api/models carry this as an additional cluster object. Their aggregate and ram fields keep reporting the frontend's own hardware, which is what the resource monitor shows. Variant selection judges backends against the union of the capabilities present in the cluster, the way backend discovery already did. Every path degrades to the local host: no cluster object in single-node mode, and none when the registry cannot be read, so a hiccup narrows the answer back to single-node behaviour rather than marking the whole catalog too large. The verdicts now name the node they belong to, since a model fits somewhere or nowhere. Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Plamen K. Kosseff
|
e58dabf75f |
feat(ui): add 'Focus mode' option in chat settings to persistently toggle the sidebar auto-collapse (#11750)
Assisted-by: Claude:claude-fable-5 Signed-off-by: Plamen K. Kosseff <p.kosseff@gmail.com> |
||
|
lei_lei
|
f28e8b24e6 |
fix(ollama): accept :latest tag on model lookup (#11732)
/api/tags appends :latest to untagged names, but chat and the other model endpoints looked the tagged name up as-is and 404'd. Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com> |
||
|
Ettore Di Giacinto
|
1dc3aeef87 |
fix(distributed): resolve config revisions through one entry point
A model's revision is published by administration and checked against on every inference request. Those were computed by separate code: the request path resolves through the loader, while each publisher hashed whatever ModelConfig it happened to hold. By then SetDefaults had folded in the GGUF guess and app-level options, so the published value was one no request would ever carry and the model became unroutable until the row was deleted by hand. Fixing the publishers one at a time did not hold. Three rounds each found another: the startup resync, then a saved edit and a toggle, then a rename and the peer-change path. ModelConfigLoader.RevisionFor is now the only way to obtain a revision, and the raw hash is unexported, so a caller outside this package cannot hash a config it holds. A publisher and a request agree by construction rather than by two implementations happening to match. The request path no longer falls back to hashing its merged config either: an unstamped config is routed without a revision rather than with a wrong one. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
Ettore Di Giacinto
|
df1a40f9c0 |
fix(distributed): hash the config as persisted, not as defaulted
The revision was computed after SetDefaults, which folds in things that are not persisted configuration: the GGUF guess, the hardware defaults, and app-level options such as threads. The GGUF guess is the damaging one. It parses the model file to fill in values like context size, and when that parse fails it falls back to a different default. Whether a multi-gigabyte file on network storage parses at a given moment is not a property of the configuration, so one unchanged YAML produced two different revisions depending on when it was read. The controller rejected every request carrying the other one, and the model stayed unroutable until the stored value happened to match again. This is why it never reproduced against a model directory with no weights in it: the guess is skipped there and both values agree. The app-level defaults are the same class of bug with a slower fuse: changing threads in the settings UI changed every model's revision and made every model unroutable. The revision is now stamped when the file is parsed, before any defaults are applied, so it is a function of the file alone. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
a8bc64cd09 |
fix(ci): bound Discord release summaries (#11695)
* fix(ci): bound Discord release summaries The release model can return more than Discord's 2,000-character message limit. Discord then rejects the entire release notification. Ask the model for a smaller response and truncate extracted content to 1,800 characters before the notification step. The smaller bound leaves room below Discord's hard limit when model output varies. Assisted-by: Codex:gpt-5 * fix(tests): implement node liveness stub NodeCommandSender now requires PingNode. The endpoint test stub must implement it before the package can compile. Assisted-by: Codex:gpt-5 [Codex] --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Ettore Di Giacinto
|
f3fabe8c5c |
fix(distributed): order derived usecases deterministically
syncKnownUsecasesFromString rebuilds KnownUsecaseStrings by ranging GetAllModelConfigUsecases, which is a map. Go randomizes that order per call, and the field is part of the serialized config, so one unchanged YAML hashed to a different config revision on every load. A model that derives a single usecase hid the problem. One that derives several, such as a chat model with an mmproj, alternated between as many revisions as there are orderings. The router treats a revision it did not establish as a config change, so requests failed with "stale model config revision" until the stored value happened to match again. Sorting the list makes the revision a function of the file alone. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
Ettore Di Giacinto
|
04735cd1f6 |
fix(distributed): stamp config revision at load time
The request middleware merges the caller's prediction parameters into its copy of the model config. core/backend.ModelOptions then hashed that copy, so the revision identified the request body rather than the persisted configuration. EstablishModelConfigRevision stores the first revision it sees and requires an exact match afterwards. The first request after a restart therefore pinned the model to its own temperature, top_p and stop values, and every later request that sent different ones failed with "stale model config revision". No config edit was involved. The loader now stamps the revision when it materializes a config, before any request override reaches it, and ModelOptions reads that stamp. Model administration keeps hashing the same persisted config, so both paths agree on one revision per configuration. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
8f56e4e042 |
fix(vram): persist remote probe metadata (#11487)
* fix(vram): persist remote probe metadata The startup warmer repeated remote size and GGUF metadata probes after every restart because both caches lived only in memory. Store successful HTTP probes for 24 hours so frequent restarts reuse the prior results. Bound the cache, reject invalid records, and purge it when gallery data changes. Local model files continue to bypass persistence. Assisted-by: Codex:gpt-5 * fix(vram): check temporary file cleanup The lint gate rejects the unchecked cleanup call in the persistent cache writer. Assisted-by: Codex:gpt-5.6 [golangci-lint] * fix(vram): make persistent cache optional Remote metadata probes can transfer enough data that operators need control over disk reuse and startup warming. Gallery autoload now gates both behaviors, and the runtime setting applies changes immediately. Assisted-by: Codex:gpt-5 * fix(ui): expose gallery startup pre-warm The existing gallery autoload setting also gates the startup metadata warmer. Name both effects in Settings so operators can find the requested boot control. Assisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
82c191afad |
fix(distributed): keep model replicas config-consistent (#11664)
* docs: design configurable copy buffering Document the context-aware copy buffer option and its validation plan. Assisted-by: Codex:gpt-5 * docs: design durable distributed staging operations Assisted-by: Codex:gpt-5 * docs: design distributed model config revisions Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] * feat(config): add stable model revisions Hash typed model configuration and effective protobuf options deterministically for distributed revision comparisons. Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] * feat(worker): acknowledge exact model stops Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] * feat(nodes): track model config revisions Assisted-by: Codex:GPT-5 [apply_patch] * fix(distributed): retry quarantined model cleanup Stop quarantined replicas by exact process identity, retain failed cleanup as durable capped retries, and compare-and-delete only the claimed registry row. Process one sufficiently leased row at a time so multiple frontends cannot duplicate slow cleanup work. Assisted-by: Codex:gpt-5 * fix(distributed): bind loads to config revisions Assisted-by: Codex: GPT-5 [OpenAI Codex] * fix(modeladmin): apply config revisions consistently Route model edits, patches, state changes, deletion, and peer refreshes through the same revision lifecycle. Quarantine stale replicas before exact cleanup and report durable pending cleanup without failing successful config writes. Assisted-by: Codex: GPT-5 [OpenAI Codex] * feat(distributed): expose model config revision state Document replica revision observability and durable cleanup behavior. Keep pending cleanup explicit in model mutation responses and verify endpoint contracts expose revision state without serialized load options. Assisted-by: Codex:GPT-5 [OpenAI Codex] * test(distributed): cover model revision convergence Exercise cross-frontend quarantine, stale replay rejection, exact cleanup retry, worker re-registration, and current-generation replica convergence against the distributed PostgreSQL harness. Assisted-by: Codex:gpt-5 * fix(distributed): pass config revision CI checks Keep configured gallery sources out of authoritative runtime snapshots only after validating their real schema, and harden rollback snapshots against symlink races and non-regular files. Assisted-by: Codex: GPT-5 [OpenAI Codex] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
9d92139de4 |
feat(ui): edit scheduling rules in place (#11667)
* docs(ui): design scheduling rule editing Document the approved in-place rule editing flow and scalable node-label reference for the scheduling view. Assisted-by: Codex:gpt-5 * feat(ui): improve scheduling rule management Add scalable node-label discovery and editable scheduling rules with responsive, accessible controls. Assisted-by: Codex:gpt-5 * chore(ui): ratchet inline style baseline Record the static inline style removed by the scheduling view enhancement. Assisted-by: Codex:gpt-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
a0252ad6a1 |
fix(distributed): keep staging operations stable (#11663)
* docs: design configurable copy buffering Document the context-aware copy buffer option and its validation plan. Assisted-by: Codex:gpt-5 * docs: design durable distributed staging operations Assisted-by: Codex:gpt-5 * fix(distributed): merge durable staging operations Use active model load jobs as the durable operations baseline and overlay replica-local staging progress without duplication. Preserve tracker-only operations when the registry cannot be read. Assisted-by: Codex:gpt-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
5429f569e0 |
fix(progress): stop status updates throttling downloads (#11661)
* feat(progress): aggregate and coalesce gallery downloads Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): show rolling transfer speed Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): preserve legacy import byte labels Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Anupam Mediratta
|
ff6043b811 |
fix: upgrade react-router to 7.18.2, 8.3.0 (GHSA-qwww-vcr4-c8h2) (#11644)
* fix: GHSA-qwww-vcr4-c8h2 security vulnerability Automated dependency upgrade by OrbisAI Security Signed-off-by: anupamme <mediratta@gmail.com> * fix: upgrade react-router-dom to 7.18.2 to fully remediate GHSA-qwww-vcr4-c8h2 The prior fix pinned react-router@7.18.2 directly but left react-router-dom at ^7.18.1, which bun resolved to 7.18.1. That package bundles its own react-router@7.18.1 sub-dep, leaving the vulnerable version in bun.lock via the react-router-dom/react-router scoped resolution. Pinning react-router-dom to 7.18.2 and regenerating the lockfile removes all 7.18.1 resolutions. Assisted-by: Claude Code:claude-sonnet-4-6 Signed-off-by: Anupam Mediratta <mediratta@gmail.com> --------- Signed-off-by: anupamme <mediratta@gmail.com> Signed-off-by: Anupam Mediratta <mediratta@gmail.com> |
||
|
Anupam Mediratta
|
9ba4bbf9bb |
fix: upgrade ip-address to 10.3.1 (CVE-2026-69192) (#11632)
fix: CVE-2026-69192 security vulnerability Automated dependency upgrade by OrbisAI Security Signed-off-by: anupamme <mediratta@gmail.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
0ab632b6bd |
fix(auth): protect HTTP routes by default (#11602)
* fix(auth): default to protected HTTP routes Use a method-aware registry for the small anonymous bootstrap surface. Unknown routes now require credentials instead of inheriting fail-open path classification. Keep node self-service routes behind their registration-token middleware. Global auth no longer rejects valid worker credentials first. Assisted-by: Codex:gpt-5 * docs(auth): document public HTTP surface Assisted-by: Codex:gpt-5 * test(auth): align route coverage with default denial Assisted-by: Codex:gpt-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
2383726d6d |
Revert "chore(tests): Avoid network, sleep and more during tests" (#11601)
Revert "chore(tests): Avoid network, sleep and more during tests (#11050)"
This reverts commit
|
||
|
Richard Palethorpeandlocalai-org-maint-bot
|
cb3bf7af3f |
chore(tests): Avoid network, sleep and more during tests (#11050)
* test: make coverage failures observable Keep per-root logs, reject concurrent coverage runs, and avoid relying on /bin/sleep in the worker timeout test. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: parallelize coverage without remote fixtures Assisted-by: Codex:gpt-5 [apply_patch] [exec_command] Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: add offline resource infrastructure Introduce versioned resource manifests, a checksum-verified CAS preparer, offline test wrappers, and a guarded network transport. Replace live Hugging Face, GitHub, and OCI cases with deterministic fixtures and inject fixture metadata into importer discovery. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: enforce offline resource replay Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: harden offline resource refresh Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: expose slow coverage waits Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: eliminate avoidable wall-clock waits Inject a clock into Hugging Face retry handling, reuse a process-scoped PostgreSQL container with per-spec schemas in the nodes suite, and poll local import jobs promptly. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: remove repeated fixture startup waits Share PostgreSQL fixtures across parallel endpoint and agent suite workers, and make the worker Free deadline injectable so the wedged-backend test does not spend five seconds on wall-clock time. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: fix offline resource CI portability Normalize Docker archive metadata before content addressing, derive archive checksums during explicit refreshes, make network lint portable to macOS, and prepare distributed images before running their offline suite. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: cache Go modules before offline tests Warm the complete module graph before the Linux and macOS test jobs enter offline replay mode, so tool dependencies such as Ginkgo are not fetched through the guarded proxy. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: drop the static network lint in favour of real isolation The offline test suite already prevents tests from reaching the network twice over: run-test-linux-offline.sh puts the test process in a cgroup and REJECTs egress outside the private ranges, and HardenedTransport installs testnetwork.LocalGuard to refuse dials that resolve to a public address. Both fail the test with a precise error at the moment of the dial. test-network-lint.sh added neither. Its diff stage defaulted to a HEAD base, so on a clean checkout it compared the tree against itself and inspected nothing; the branch's own commits were never examined. It only produced output when an earlier job step dirtied the tree, and then it matched a bare https?:// against whatever changed. make react-ui runs npm install rather than npm ci, so CI rewrote core/http/react-ui/package-lock.json and the lint reported an npm registry URL as forbidden test network access: + "resolved": "https://registry.npmjs.org/hono/-/hono-4.12.25.tgz", Its fingerprint stage was self-defeating in a quieter way: hashing the whole tree's network-mechanism inventory meant every rebase onto a master that touched any _test.go needed a manual baseline bump, so the check mostly caught its own staleness. Remove the script, its make target and the two prerequisite edges, along with the test-network: fixture markers that existed only to suppress it. The isolation itself is untouched. Assisted-by: Claude:claude-opus-5 [go vet] Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: keep hidden files in the offline test bundle artifact Cherry-picked from |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
4cad809003 |
fix(ci): test stale chunks in split bundle (#11595)
The V8 coverage build inlines every dynamic import, so the stale chunk tests cannot intercept a page chunk. Run those tests against the normal code-split bundle and exclude them from the inlined coverage pass. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Ettore Di Giacinto
|
b806b1fec3 |
fix(ui): reload once when a page chunk 404s
A deploy replaces the whole content-hashed asset set at once. A tab holding an older index.html, or one whose request lands on a replica that the rollout has not swapped yet, asks for a page chunk the server no longer has. The dynamic import rejects and React Router's default error boundary replaces the app with "Unexpected Application Error!" until someone reloads by hand. The router now reloads the page itself when a chunk fails to load. index.html is served no-cache, so the reload lands on a self-consistent asset set. A timestamp in sessionStorage bounds this to one reload per 10 seconds, so a chunk that is genuinely gone reaches the error boundary instead of looping forever. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
0761bd02c7 |
feat(chat): add end-to-end context compression (#11556)
* feat(config): add context compression policy Define the opt-in model configuration contract before the chat middleware consumes it. Document each policy field so later request handling does not invent a second schema.\n\nRefs #9534\n\nAssisted-by: Codex:gpt-5 * fix(config): register compression fields The model editor metadata gate rejects new config fields without descriptions and suitable controls. Register the compression policy so operators can edit its six fields safely. Assisted-by: Codex:gpt-5 [monitoring-prs] * feat(chat): compress long contexts Long conversations currently fail once they reach the model context window. The opt-in policy now summarizes complete older turns before primary inference and preserves the newest tool chains. Both OpenAI and MCP chat routes share the same transformation. Usage metadata and metrics expose each compression event. Refs #9534 Assisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Nicholas Ciechanowski
|
cf93c04878 |
fix: tts text wrap (#11576)
Signed-off-by: Nicholas Ciechanowski <nicholas@ciech.anow.ski> |
||
|
Richard Palethorpe
|
d10374f849 |
feat(router): make KNN a first-class classifier with a persisted, curated corpus (#10652)
* feat(router): make KNN a first-class classifier with a persisted, curated corpus
Add `classifier: knn` — similarity-weighted voting over labelled
example prompts. Unlike score/colbert it needs no classifier model:
label knowledge lives in a corpus seeded and curated through the
admin API, so routing decisions are deterministic, auditable, and
grounded in graded experience rather than a model's opinion.
Epistemic gate: corpus entries below knn.similarity_threshold cannot
vote; when none clears it the classifier activates no labels and the
router uses the fallback — a prompt unlike all labelled experience is
treated as undecidable, not guessed. Decisions record
nearest_similarity (also on fallback rows) so admins can see how far
the nearest labelled experience was; the Routing tab explains
out-of-corpus fallbacks and shows per-label corpus counts.
Persistence: one JSONL file per router under
<data path>/router-corpus (text, labels, vector, embedder
fingerprint). The file is the source of truth; the local-store index
is rebuilt from it at classifier build time and stays a pure
in-memory index. Entries recorded under a different embedding model
re-embed on load. Also corrects the docs' false claim that
local-store collections persist — the embedding cache never survived
restarts (and still doesn't); the corpus does.
Corpus input is API-only by design (entries may contain example user
content): POST /api/router/{name}/corpus seeds (labels validated
against declared policies, embedded server-side, indexed
immediately), GET .../corpus/stats inspects — label counts only,
entry texts are never returned by any surface — DELETE .../corpus
wipes. Admin-gated like the sibling router endpoints, and exposed as
MCP tools (seed_router_corpus / get_router_corpus_stats /
clear_router_corpus) in both the httpapi and inproc clients with
coverage-test route mappings.
Plumbing: VectorStore gains SearchK (top-K was hardcoded to 1);
local-store gets InsertBatch/Delete as optional fast paths;
RouterConfig gains a knn block (embedding_model, k,
similarity_threshold, vote_threshold, store_name) with meta-registry
fields; the classifier dropdown now offers knn and the
previously-missing colbert; embedding_cache is ignored (with a
warning) for knn — it IS an embedding-KNN lookup; the stale
/api/instructions intelligent-routing entry is rewritten (it
described a classifier that no longer exists); swagger regenerated.
Tests: KNN vote/gate specs with hand-computed vote shares, corpus
manager suite (restart reload without re-embedding, fingerprint
re-embed, dedupe, hostile store names), middleware specs (corpus
routing, gate fallback, config validation, cache-wrap refusal),
corpus endpoint specs pinning the texts-never-returned contract, MCP
catalog + route-mapping gates, and a Playwright spec for corpus
stats and the out-of-corpus decision detail.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(router): name consulted corpus neighbours in knn decisions
Every knn decision (decision log rows and the /api/router/decide
response) now carries neighbors: the K retrieved corpus entries by
descending similarity - including ones below the epistemic gate, which
is what makes fallback decisions diagnosable - each as {id, similarity,
labels}. The id is the entry's content hash (first 8 bytes of the
SHA-256 of its text, hex): stable across reseeds and re-embeds, and
text-free, so an external platform that seeded the corpus can recompute
text->id on its own copy and bucket decisions by corpus region (per-
region reliability accounting) without corpus text ever leaving the
server. A corrupt index payload surfaces as an id-less neighbour at a
real similarity instead of disappearing.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* refactor(router): deduplicate knn plumbing and cut corpus hot-path waste
Post-review cleanup of the knn-first-class-router branch; no behaviour
changes on the API surface.
Reuse/altitude:
- RouterKNNConfig.ResolvedStoreName is now the single source of the
router-corpus-<name> default (was hand-derived in four files).
- corpus.ResolveKNNRouter + corpus.Seed carry the shared model
resolution and seed validation; the REST endpoints and the assistant
MCP client are thin transport adapters over them, with sentinel
errors mapped to HTTP statuses at the echo boundary.
- middleware.NewClassifierDeps assembles the classifier dependency set
once for all five entry points (OpenAI, Anthropic, realtime, decide,
corpus) instead of five hand-copied literals.
- router.AllClassifiers feeds both the status endpoint and the
unknown-classifier error, ending the classifier-list drift.
- Per-classifier requirements moved out of validateRouterPolicies into
their buildClassifier arms; the knn arm owns its embedding_cache
opt-out instead of a name-check in the shared wrap tail.
- adminOnly replaces four inline copies of the admin gate in the
middleware routes.
- localVectorStore.Search delegates to SearchK (identical traces).
Efficiency:
- Manager.Add embeds outside the manager mutex and appends to the
JSONL file (O(new) instead of O(corpus) rewrite); a torn tail from a
crash mid-append is tolerated on read and repaired on next write.
- Stats memoises per store keyed on the file's stat fingerprint and no
longer takes the manager mutex, so the 5s status poll stops parsing
vector-laden JSONL and stops blocking behind seeds.
- KNN Classify decodes each neighbour payload once (was twice) and
builds refs and votes in a single pass with one fallback return.
- Corpus file writes fsync before rename/close.
- The corpus manager is built eagerly in newApplication (sync.Once
dropped); test helper dead branch removed.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(router): bind knn corpus vectors to an embedder fingerprint and fail closed on mismatch
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* chore(mcp): align corpus tool prompts and the mutating-tool safety list
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(proto,backend): report embedding shape from the llama-cpp backend
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(embeddings): Go-side pooling — mean/last/decayed_mean with half-life
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(embeddings): accept chat messages[] and per-request pooling on /v1/embeddings
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* chore(middleware): name the failing fields when post-merge validation 400s
An intermittent post-merge validation failure surfaced as an opaque 400
during integration (pooling scheme mismatch that no client had sent).
Log the model, the request's pooling override, and the merged config's
pooling fields at the failure point so the next occurrence identifies
whether the request or the stored config carried the bad value.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(embeddings): scheme override must not inherit the config's half-life
A model config defaulting to decayed_mean pooling carries
pooling_half_life_tokens; a request overriding the scheme to mean/last
without its own half-life inherited that value, and post-merge
validation rejected the pair the server itself had assembled. Zero the
inherited half-life when the overridden scheme is not decayed_mean; a
request that explicitly pairs a half-life with a non-decayed scheme
still 400s.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix embedding pooling validation and router bounds
Declare backend embedding layouts and reject incompatible pooling modes. Reset local-store dimensions after a full clear, validate KNN thresholds, and add real backend and store integration coverage.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* ci: run local-store integration tests
Build and install the local-store backend in the Linux test job, then run the existing store integration suite so new specs are discovered automatically.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
|
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
a7bce6a128 |
fix(audio): reject incompatible transform streams (#11565)
The transform WebSocket accepted any model and opened its frame-based RPC. Any-to-any models use a different stream contract, so liquid-audio failed with an unimplemented RPC after the handshake. Reject incompatible model use cases before loading the backend. Direct realtime-audio callers to the OpenAI Realtime API. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Richard Palethorpe
|
799cc9f211 |
feat: bound global admission and expose running backend traces (#11560)
feat: bound backend admission and expose running traces Add process-wide backend execution admission without blocking UI or administrative HTTP work. Represent backend operations while they are in flight, surface running traces with immediate log links, and tie streaming admission leases to the gRPC receive lifecycle. Assisted-by: OpenAI Codex: GPT-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
0aaff91ebd |
feat(ui): unify model and backend lifecycle (#11548)
* feat(ui): add installed model lifecycle Models now owns catalog exploration and installed runtime controls under one canonical route. URL-owned state keeps lifecycle context recoverable through links and browser history. Assisted-by: Codex:gpt-5 Playwright * feat(ui): add installed backend lifecycle Backends split discovery from backend-binary management. The canonical page now keeps both lifecycle views under one URL-backed shell while it preserves target-node placement. Assisted-by: Codex:gpt-5 Playwright * fix(ui): repair lifecycle state updates Installed models lost distributed refreshes and kept a deleted selection. Backend searches also stopped tracking URL changes, while batch upgrades stopped after their first error. Preserve background refreshes and finish each requested batch action. Drive catalog results from URL-backed state without losing full metadata. Assisted-by: Codex:gpt-5 [Playwright] * feat(ui): make resource pages canonical Replace Host navigation with canonical Models and Backends lifecycle routes, preserve legacy management URLs, and surface shared host capacity on the Operate overview. Assisted-by: Codex:gpt-5 [Playwright] * feat(ui): complete canonical resource lifecycle Finish the responsive list-to-detail behavior, remove the retired Host implementation, and keep Explore focused on discovery while Installed owns destructive actions. Update regression coverage, localization, documentation, and development binding for the canonical resource pages. Assisted-by: Codex:gpt-5 [Playwright] * docs(ui): record the UI design context Record the approved users, brand character, and design principles so future interface work uses the same product direction. Index the context from the repository's agent instructions. Assisted-by: Codex:gpt-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
88edd7fc7f |
fix(distributed): run cold model loads as durable jobs instead of holding the advisory lock (#11514)
* fix(advisorylock): set statement_timeout alongside lock_timeout
WithLockCtx already overrides a deployment-wide lock_timeout on its
dedicated connection so a blocking pg_advisory_lock() waits its turn
instead of failing with 55P03. statement_timeout aborts that exact same
statement independently, with SQLSTATE 57014, and was not overridden.
Production roles commonly carry statement_timeout=60s. Any guarded
section longer than that (a cold model load stages for tens of minutes)
therefore killed every concurrent waiter:
advisorylock: acquiring lock 9003261067483446873: ERROR: canceling
statement due to statement timeout (SQLSTATE 57014)
Derive it from the same context budget as lock_timeout, with a matching
RESET so the pooled connection is returned clean.
Assisted-by: Claude Opus 5 [claude-code]
* feat(distributed): add ModelLoadJob, the durable cold-load record
A cold load in distributed mode is a long-running background job, but it
was modelled as a synchronous side effect of an inference request: the
whole of it (backend install, multi-GB staging, checkpoint load) ran
inside the per-model advisory lock. Loading a 35.7 GB GGUF held that lock
for ~20 minutes, so every concurrent request for the same model blocked
on pg_advisory_lock and died at the role's 60s statement_timeout.
Introduce the row that lets the lock shrink to a decision. Exactly one
ModelLoadJob may be active per tracking key; that uniqueness — not the
lifetime of a lock — is what de-duplicates concurrent loaders across
replicas. ClaimLoadJob does its read-then-write under the advisory lock
and nothing else: no network, file or gRPC I/O inside the guarded
section, so a claim costs milliseconds no matter how long the resulting
load takes.
LastProgress is a heartbeat rather than a byte counter. A checkpoint load
legitimately moves zero bytes for many minutes, so a reaper keyed on byte
movement would reclaim a healthy job mid-load; byte progress stays the
concern of load_deadline.go. A job whose heartbeat stops for longer than
the orphan window is reclaimable, so a replica killed mid-load cannot
wedge a model permanently.
Failed jobs keep their row for a short grace so an immediately-following
request reports the real cause instead of silently starting a fresh load
of a model that just failed.
No caller yet — the router moves onto this in the next commit.
Assisted-by: Claude Opus 5 [claude-code]
* refactor(distributed): run cold loads as jobs, outside the advisory lock
Route wrapped the entire cold load — node selection, backend install,
multi-GB staging and the remote LoadModel — in the per-model advisory
lock. The lock's job is to de-duplicate concurrent loaders, a decision
that takes milliseconds; holding it for the tens of minutes the resulting
work takes is what turned a dedup mechanism into a cluster-wide outage
for that model.
Split it into a claim and a run. The claim is the only thing left inside
the lock. The run is a background job owned by the claiming replica and
bounded by the same progress-extended deadline as before; every other
request for that model — local or on another replica — attaches as a
waiter and is served the moment the model is ready, with no duplicate
load and no lock contention.
Waiters share one broadcast rather than an ordered queue: they all want
the identical outcome, so ordering them would add fairness machinery that
changes no result. The local channel wakes same-replica waiters instantly
and a 2s DB poll is the authority, because a waiter on another replica
has no channel to close. On wake a waiter re-runs the warm path rather
than trusting the signal — the model may have been evicted in between.
A waiter whose client disconnects returns immediately and the job keeps
running; it belongs to the job record, not to the request. A failure is
recorded on the row so every waiter reports the real cause, and the row
survives briefly so the next request does not read "no job" as "not
loading" and start a duplicate load of a model that just failed.
The runner heartbeats the row on a fixed interval whether or not bytes
are moving, which is what keeps a legitimately silent checkpoint load
from being reclaimed as an orphan. Phase (installing/staging/loading) and
placement ride to the heartbeat on the context, the same seam
load_deadline.go already uses, so single-host paths are untouched.
Non-distributed mode (no DB) keeps the inline load exactly as it was.
Assisted-by: Claude Opus 5 [claude-code]
* feat(distributed): bound the wait for a loading model and answer with progress
A request whose model is cold-loading now attaches to the running job and
is served the moment the model is ready. That wait has to be bounded: a
held HTTP request cannot survive real infrastructure, and an ingress or LB
idle timeout kills a twenty-minute request regardless of what LocalAI
does.
New LOCALAI_MODEL_LOAD_WAIT (default 60s) bounds the CALLER, never the
load — the job keeps running either way. On expiry the request gets 503
with Retry-After and a structured body naming the model, the node, the
phase, byte progress and an ETA. The `error` envelope keeps OpenAI
clients working; `loading` is additive so they ignore it.
The ETA comes from the job's own observed rate and is omitted rather than
guessed until enough bytes have moved for that rate to mean anything: a
confidently wrong ETA on a twenty-minute wait is worse than none.
Retry-After is that ETA when known, clamped to [5s, 300s], and the wait
budget otherwise.
LOCALAI_MODEL_LOAD_WAIT=0 waits unbounded, for deployments with no proxy
in front. Zero in the config struct still means "unset, use the default",
so the CLI records the operator's zero as ModelLoadWaitUnbounded rather
than losing the distinction.
The distributed branch of ModelLoader.loadModel wrapped the router's
error with %s, which flattened it to a string. Use %w: the typed error is
what the HTTP layer keys the 503 off.
Assisted-by: Claude Opus 5 [claude-code]
* feat(api): add GET /api/models/{id}/load-status
A client that receives 503 while a model stages onto a worker needs
somewhere to poll. This returns the same `loading` object the 503 carries
— phase, node, byte progress and ETA — or 404 when no load is running.
Read-only and observability-shaped, so it is deliberately neither
admin-gated nor feature-gated: it explains a 503 the caller just
received, and hiding that behind a per-modality feature would make the
explanation for a failed image request depend on chat permissions. It
also gets no MCP tool, since there is nothing here an admin would manage
conversationally.
Registered on the surfaces from .agents/api-endpoints-and-auth.md: the
swagger block (existing `models` tag, so /api/instructions needs no new
area), the endpoint discovery maps in RegisterLocalAIRoutes, regenerated
swagger, and the distributed-mode docs page. No FLAG_* usecase is
involved, so capabilities.js is unchanged.
Assisted-by: Claude Opus 5 [claude-code]
* feat(ui): show cold-load progress in Chat and retry when the model is ready
A chat request for a model that is still staging onto a worker now gets a
503 carrying live progress instead of an error. Render it: the composer
shows the phase (installing / staging / loading), the node, the percent
and the ETA, then polls load-status and re-sends the request the moment
the model is ready.
Reuses the staging progress idiom the page already had rather than
inventing a second one — the two sources are folded into one
loadProgress, with the load job winning because it is authoritative
across frontend replicas and knows the phase, where the staging operation
only knows about a byte transfer this replica happens to be performing.
Waiting is bounded (three send attempts, ~30 min of polling each), so a
load that never finishes still surfaces as an error rather than as a
spinner nobody questions. An aborted generation stops the polling too.
Assisted-by: Claude Opus 5 [claude-code]
* fix(distributed): check warm-path cleanup errors
The router moved legacy cleanup calls onto newly linted lines. Report
cleanup failures while preserving the fallback to a cold load.
Assisted-by: Codex:gpt-5 [golangci-lint]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
|
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
95653f221e |
fix(ui): keep agent import action visible (#11488)
* fix(ui): keep agent import action visible The header hid its full import label after the agent list became non-empty. Hide only the nested file input so users can import more agents. Assisted-by: Codex:gpt-5 * test(ui): match the agent import label The Agents page renders the action as Import. The test searched for Import Agent, so it failed before checking visibility. Mock the observables request to remove backend timing from the fixture. Assisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |