mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-12 22:33:54 -04:00
6b712e76dbce1a683c2c96a17959080cd0fbf850
712
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ettore Di Giacinto
|
6b712e76db |
fix(cluster): keep the refusal vocabulary in one table
The worker re-classified a failure a local service had already classified. classifyServiceFailure preserved exactly one of the four refusal codes, which was faithful to its own comment for as long as there was one worth keeping; once ErrStreamNotServed existed, a service returning the code whose whole job is to say "I learned nothing" had it promoted to ErrStreamTargetUnavailable, which every reap guard acts on. ErrStreamTagUnknown was promoted too, and cost nothing only because both sides of that one reap. No in-tree service produces either, which is the same "unreachable, therefore safe" argument that let the request-frame merge survive a whole phase, and LocalService is exported. The cause was a fifth site enumerating the vocabulary by hand, so the fix is one table. streamRefusals pairs each sentinel with its wire code and with whether a frontend may act on it as evidence about a backend, and the writer, the reader, IsWorkerAnswer and the new IsStreamRefusal all read it. A fifth code is now taught to every one of them at once. The codes are also pinned against literals written out in a spec, the way this branch already pinned the NATS vocabulary. The round-trip table cannot see a rename, because a rename moves the writer and the reader together; an unrecognised code is deliberately not the worker's answer, so renaming "unavailable" would turn every crashed backend on a tunnelled worker into a row nothing can ever reap, silently and with the suite green. Three comments the previous fix falsified, corrected: - tunnelHeaderTimeout still said the window bounds only framing the frontend writes immediately after opening the stream. That is true on the direct path and false on the relay path, and it was the argument for treating an expiry as the frontend's fault. - classifyServiceFailure's deny-list is three causes, not two: on a dial error net.Error.Timeout also covers ETIMEDOUT and EAGAIN. Both are kept deliberately, because reaping a wedged or resource-starved backend is the eviction this phase exists to prevent, and ECONNREFUSED still reaps. isReadTimeout is renamed reportsTimeout, which is what it asks. - The operator table named three refusals and said a refusal is acted on. It now lists four, with when each is sent and whether the row is reaped. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
c19ed5ab32 |
fix(cluster): stop a late request frame reading as the worker's verdict
Making a worker's refusal reaping evidence created a defect one layer along, at the producer. The worker refused a ReadStreamRequest failure with ErrStreamRequestInvalid and its own comment said "Includes the deadline above expiring", which was harmless while every refusal reached the frontend as "no route" and became a reap the moment one of them did not. So a request frame that had merely not ARRIVED yet was reported as a non-transient verdict about a backend. It is reachable on the relay path, which carries most production traffic: the worker's header timer starts when the OWNING replica opens the stream, while the frame is written by the DIALLING replica only after the relay's acceptance travels back to it, so a whole peer-link round trip runs inside that window, on a link this design deliberately loads with multi-gigabyte artifacts beside token streams. For a long-deadline caller the endpoint is ConnectionEvictingClient, which stops the model across the fleet. It also falsified the "neither clears on its own" argument that licensed the reap. There is now a fourth refusal, ErrStreamNotServed, for what a worker could not serve for a reason of its OWN. It is deliberately outside IsWorkerAnswer, so it reaches a consumer under the no-route umbrella and reaps nothing, which is the same treatment an unrecognised code already gets. Four producers move onto it: a request frame that timed out (a malformed one stays a verdict, because that is a frontend bug no retry fixes), both SetReadDeadline failures, which are facts about the stream and not about a target nothing has dialled yet, and WriteStreamRefusal's default for a reason nobody classified. classifyServiceFailure keeps ErrStreamTargetUnavailable as its default on purpose: inverting it would make errno enumeration the single point of failure for the reap, and a miss there is a row nothing can ever delete. What it gains is a deny-list of two causes that are provably this worker's own clock or its own context. Also: - The read-site caller-deadline guard in the handshake was unpinned: the existing seam spends the budget before the handshake starts, so only the write could ever fail. A spec whose deadline falls between the request and the reply pins it, and each guard now reddens on its own. - The documented worker-first failure line omitted the JSON error envelope the old frontend returns, so an operator grepping it found nothing. - The peer-link disclosure names the aimable per-session receive window in all four places, and LastDialErrorOf records why a third consumer must go through IsWorkerAnswer rather than roll its own list. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
d26263f9c0 |
fix(distributed): let a worker's own refusal be evidence about its backend
A worker that refuses a stream has answered, and cluster.Dial keeps the three tunnelproto sentinels out of the ErrNoRoute umbrella precisely so a consumer can act on that. No consumer did. Since workers stopped listening, a backend process that crashed on a healthy worker is no longer a dead listener's codes.Unavailable: the worker refuses the stream with ErrStreamTargetUnavailable, gRPC flattens it into Unavailable anyway, and nodes.unroutable reported the whole thing as "this frontend has no route". Every reap path then answered ProbeUnknown and left the row, so the replica slot never freed and at the default MaxReplicasPerModel=1 the only cleanup left was LRU eviction of models that were working. isWorkerAnswer is exported as cluster.IsWorkerAnswer, so the errors the dialer keeps out of the umbrella are by construction the errors the consumers treat as the worker answering. nodes.unroutable and pkg/model's transportFailure both use it; ConnectionEvictingClient, the site reached during inference, goes through transportFailure rather than asking the transport directly. A reply code this frontend does not recognise is still not an answer, so a newer worker's vocabulary costs a retry and not a replica. The reap guards keep the allow-list rather than requiring ErrNoRoute: an unrecognised dial error must mean "no route", never "the backend is gone". Also in this final pass over the branch: - Docs: recommend upgrading FRONTENDS first, with the symptom of each order. Workers-first fails now that a 4xx registration is a verdict rather than an outage, so an old frontend's "address is required for backend workers" makes each restarted worker exit and drains the fleet a node per restart. - Docs: LOCALAI_WORKER_TUNNEL=false is a fatal startup error, not a degraded mode, in both places that described it; and a frontend rollback needs every worker restarted, because re-registration force-clears the address columns. - A replica with no advertised address now says so every five minutes and names the workers only it can reach, instead of one startup warning for a cost paid for the life of the process. - callerRanOut's rule now holds at all three siblings, so an expired caller deadline stops reading as a broken tunnel; probeHealth's withdrawn reason for using the raw client is corrected; the dead DoOrCached is deleted and its coverage kept on DoOrCachedResult; sweepLeakedInFlight enumerates the outcomes that reach it. - The peer route's self-declared id is recorded as a phase-3 deferral, in the handler, in the isolation claim it narrows, and in the operator docs. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
0683fb1579 |
test(distributed): prove the worker tunnel end to end, under real inference
Everything this phase built was proven by unit and integration specs. This is the first run of it against the real binaries: a frontend replica per process, a worker that binds nothing routable, real inference over the result. Four scenarios, each with the question "what would make this pass if the tunnel were doing nothing" answered rather than left open. A worker with no advertised address is reached through its tunnel. The roster is asserted to report it advertising nothing, so there is no address a frontend could have dialled instead, and node_connections is asserted to name the replica that serves the request. A request landing on the replica that does NOT own the worker is relayed to the one that does. With N replicas behind round robin that is (N-1)/N of production traffic, so it gets the FIRST request for its model: the backend install, the file staging on the http tag, and the gRPC load and predict all cross the relay. Which replica owns the tunnel is read from the ownership table through the production Owner query and mapped to a frontend index through the address the harness pins per replica; the non-owner is derived from that reading and asserted to be a non-owner immediately before the request, rather than assumed from the harness default. Sending the same request to the owner reddens it. Killing the owning replica re-homes the worker onto the survivor. The worker dials a balancer rather than a replica, because LOCALAI_REGISTER_TO is resolved once at boot and is the tunnel endpoint as well as the registration one: aimed at a single replica, a worker has nowhere to reconnect to when that replica dies, and the re-home cannot happen at all. Removing the kill reddens it. And the negative control for the whole suite, which is why the other three mean anything. Frontend and worker share a host here, so every backend port the frontend names in a stream target is one it could have dialled directly; if it did, the first three would pass with the tunnel inert. LOCALAI_WORKER_TUNNEL is no longer usable for this, because it is a fatal startup error and a worker that never started says nothing about a worker reachable some other way. The balancer answers the tunnel connect path itself instead, leaving a worker that registers, heartbeats, reports healthy and holds no tunnel. It is asserted to have dialled and been refused, asserted to be held by nobody, and then asserted unreachable with the refusal naming the missing route. Then the block is lifted, nothing else changes, and the same request succeeds: that is what attributes the refusal to the tunnel rather than to any of the ordinary reasons an e2e inference fails. The fifth spec measures the head-of-line blocking this phase deferred three times. 128 MiB crosses the session while a warm model is probed back to back, direct and relayed. Median latency is unchanged, the worst probe is about 3x the baseline median and about a seventeenth of the transfer window, and the transfer runs at 415-490 MB/s direct and 222-268 MB/s relayed. A session that head-of-line blocked would park a probe for the length of the window. Leave the yamux windows untuned; and note this is loopback, so it says the multiplexing does not serialise and says nothing about a link with a bandwidth-delay product. The load spec is measured against a control that the first version did not have. It passed with the bulk artifact cut to 4 KiB, because the window it read probes against was mostly cold-load overhead: it would have reported a clean bill on a session carrying no large message. The same cold load now runs twice, once empty and once bulk, and the difference between the windows is asserted to be real before any latency is read from it. Two defects on the base commit came out of this. cluster_peerlink_test.go has been red since the relay landed, deterministically, in isolation and in the suite. It asserted that an accepted peer stream is refused at once, on the premise that phase 1 installs no relay. The relay correctly waits fifteen seconds for a frame naming the worker, and the spec's budget was five. It now writes a relay request for a node no replica holds and asserts the refusal is ErrNotOwner and specifically not ErrNoConnection, which is a stronger spec than the one it replaces and the only thing in the e2e suite that exercises the relay's refusal path. The harness handed a worker's own HTTP port to a backend process. It took two ports from freeport and used one as the gRPC base and the other for the file transfer server; freeport returns adjacent ports often, and the backend allocator hands out base, base+1, base+2, so the second backend started on a worker was regularly given the HTTP server's port and died with EADDRINUSE. No spec had started two backends on one worker before, so it had never fired; the load spec starts five and it failed about one run in three. Each worker now reserves a contiguous bind-probed block laid out the way production lays it out, below the kernel's ephemeral range, with LOCALAI_GRPC_MAX_PORT bounding the allocator to it. The underlying production defect is not fixed here and is recorded in the report: allocatePort never checks that a port is free, and its default range overlaps the ephemeral range on every Linux box. Constraint 6, whether distributed mode should now refuse to start without an advertised address, is DEFERRED, and the comment and the docs that described the cost were understating it. A replica with no advertised address writes no instances row, and Owner joins a connection against a live instance, so a worker whose tunnel lands there is unroutable from every OTHER replica while being registered and healthy. Refusing to start would still be wrong, because the deployments it would break are single-host ones with no peers to be unreachable by, and telling those apart at startup is a design with its own specs. Both places now say what actually happens. Suite wall clock 592s for 15 specs, up from 502s for 10 of which 2 were red. The CI budget of 20 minutes does not move. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
3338d7bc56 |
fix(distributed): refuse a worker that cannot tunnel, and say why it was refused
Review round 1 on the change that stopped workers listening. One blocking item
and seven notes.
LOCALAI_WORKER_TUNNEL=false was the blocking one, and the ruling was to make it
fatal rather than to correct the comment that still promised it fell back to the
advertised address. There is no fallback left: a worker on this branch
advertises nothing and binds only loopback, so turning the tunnel off leaves it
reachable by nothing while it registers, heartbeats and reports healthy, and the
scheduler keeps placing models on it. That is the worst available failure shape,
so a new Config.validateStartup refuses it before prefetch, registration and
NATS, while the worker is still invisible to the cluster. It absorbs the
pre-existing empty-registration-token check, which had the same shape and no
spec. The flag is kept rather than deleted so an operator who set it is told the
promise is gone instead of having the setting ignored, and the guard around
StartTunnel is removed, because a branch nothing can take reads as a supported
no-tunnel mode that does not exist.
The justification for erroring on an install that names no address was wrong,
and the review is right that this is the dangerous form of overclaiming, because
the conclusion holds and the mechanism does not. It said the resulting empty
target would be refused as an invalid stream and that the refusal would read as
the worker answering about its backend. Nothing in this repo branches on
cluster.ErrNoRoute, and nodes.unroutable treats any recorded dial error as
unroutable, so that refusal reaches every reap guard as ProbeUnknown and deletes
nothing. The site now stands on what holds, that an install naming no port
produced nothing routable and the failure belongs to the install rather than to
a later probe, and records the retracted claim so nobody re-derives it. This
retracts the same paragraph in the body of
|
||
|
Ettore Di Giacinto
|
1cf847f29e |
feat(distributed): stop workers listening, and stop them advertising
A worker now opens no listener on a routable interface and states no endpoint at registration. Backend processes and the file-transfer server bind loopback, and the frontend reaches both through the tunnel the worker dials. The bind address is built from loopbackHost, the same constant the tunnel's grpc tag dials, so "the worker binds where its tunnel dials" is one fact in one place rather than two literals that can drift. All three advertisement sites are closed, not one: the registration body, RegisterNodeRequest, and the per-backend address in the install reply. That third one was hiding a live bug. stopModelExact refuses a stop whose ExpectedAddress does not match what the worker recorded for the process. The worker recorded 127.0.0.1:port; handleBackendInstall reported advertiseHost:port; the router stored the reported one and sent it straight back. On any worker whose advertise host was not 127.0.0.1, every acknowledged model stop failed with an address mismatch. Nothing caught it because the e2e harness set LOCALAI_ADVERTISE_ADDR=127.0.0.1, which made the rewrite a no-op. Removing the rewrite makes the two strings the same by construction. The brief was wrong about two of the four functions it called dead. effectiveBasePort is the base of the backend port allocator and resolveHTTPAddr is the file server's bind address; deleting them would have deleted the port allocator and the file server. Only the two advertise* helpers were dead, and addr_test.go is rewritten rather than deleted, because the port arithmetic it pinned still needs pinning. NodeModel.Address survives with a narrowed meaning and is renamed WorkerLocalAddress, along with the install reply field that feeds it. The frontend still has to say WHICH backend process on a worker it means, and the port in this string is how it says it: it travels as a stream target and the worker dials its own loopback. The gorm column and the json key stay "address", so neither a migration nor an API break rides along. Every fall-back to the node's address is gone. installBackendOnNode now errors when a worker reports success without naming one, because substituting the now-always-empty node address would name an empty target, and the worker refuses that as an invalid stream, which is classified as the worker answering about its backend. That is the "a present worker reads as something it is not" class this phase forbids. DistributedModelStore.Range had the same shape and was already wrong: it built each remote model's client from the node's base gRPC port, never the port a backend process listens on, so Free and Status went to the wrong place. It uses the replica's address now. BackendNode.Address and HTTPAddress are kept but made provably inert: no writer, no reader that acts on them, and Register force-clears both on re-registration so an upgraded worker's stale advertisement does not outlive its own upgrade in the API and the Nodes page. Dropping the columns is a ~90-site edit across the specs, the e2e suite, the MCP dto and the UI; it is recorded as a follow-up rather than folded in here. A persistent tunnel 401 still does not trigger re-registration, and now for a reason rather than a deferral. Register CLEARS the node's replica rows, so re-registering on a 401 would delete a live worker's rows on every retry, and under the name collision that causes the 401 the two workers would take turns doing it forever: a credential failure causing model reclamation. It also cannot fix the named cause, since a collision is indistinguishable from a restart. The 401 log now names both causes and says nothing can reach this worker, which is true only now that it has no listener. The container healthcheck did not break the way the brief expected, since the listener still exists on loopback and the probe runs inside the container. It did have a real #10987 defect that this change makes the common case: it read LOCALAI_SERVE_ADDR only, while effectiveBasePort reads LOCALAI_ADDR first, so a worker on a non-default base port was probed on 50050 and reported unhealthy while working. It follows the same precedence now. Docs, the compose file and the e2e harness are updated in step: no inbound rule or published port is needed for a worker, the two advertise variables are gone, the remaining address variables are read for their port only, the firewall-the-file-transfer-port warning is narrowed to the LOCALAI_HTTP_ADDR opt-out, and the upgrade-order note no longer claims the worker still listens. The Nodes page showed node.address, which is now always blank, so it shows the node id instead. Eight mutations, all red on a named spec, including reverting the loopback bind, re-adding the address to the registration body, restoring both node-address fall-backs, dropping the force-clear, storing the endpoint's address again, and un-fixing the healthcheck. One of them caught a defect in a spec I had just written: it asserted 200 where the endpoint returns 201, which went unnoticed because core/http/endpoints/localai is not on the task's verify list. It is run here. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
b4d8e23abb |
fix(grpc): let the transport answer through the wrappers, not only past gRPC
Re-review round 2. One blocking defect, and it was the concern I filed myself last round and mis-scoped as a future trap. It was live, and it sat on the most destructive reaping path of the five. RouteResult.Client is an InFlightTrackingClient, over a FileStagingClient when a stager is configured. model_router puts that on the cached remote model and pkg/model's checkIsLoaded asks IT whether the transport failed. Both wrappers embed grpc.Backend, which does not declare LastDialError, so the type assertion read nil and the guard added last round fell straight through to the old eviction. That eviction sends backend.stop over NATS to every node holding the model and deletes every replica row, where the other sites delete one. The spec covering it built a bare client by hand, which is why it passed while production did not. This is the third time in this task a correct fix was disarmed one layer out, so the fix is a mechanism rather than two methods. BackendUnwrapper is one line per decorator, LastDialErrorOf walks the chain, and both consumers now call it instead of each keeping its own assertion. One implementation, no per-caller policy to get wrong. Sweeping every type that embeds or holds a grpc.Backend found a third decorator the review had not named, and it is itself a reaping consumer of the same collapsed signal. ConnectionEvictingClient is built for remote models in initializers.go and its evict callback runs ShutdownModel; it fires during INFERENCE rather than on a health check, so a tunnel blip mid-request was enough to stop a model that was loaded and serving. It consults the transport first now. A locally spawned backend has no custom transport, so that path is unchanged byte for byte. Everything else touching a Backend is a consumer rather than a decorator; there is no fourth. The probe cache joiner shape is pinned. It was the right design last round with nothing holding it: the mutation back to a closed-over variable passed all 602 specs in the package. Eight goroutines coalesced on a probe that blocks on a channel now assert every joiner gets the leader's REASON and not just its answer, which is the difference between a leader declining to reap and its seven joiners reaping on the leader's own observation. The LastDialError scope note claimed an exactness it does not have at checkIsLoaded, which reads a shared long-lived client after releasing opMutex. It now says which caller is not exact, why the imprecision is accepted there, and what making it exact would cost. The four-outcome table in the docs still said a worker with no live owner is treated as absent and rescheduled, contradicting the code and the paragraph nine lines below it. None of those outcomes is absence any more, and the table says so, names the fifth, and points at the heartbeat as the thing that does decide presence. Five mutations, each reddening named specs, including the two the reviewer found surviving. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
a8ac2af167 |
fix(cluster): make "no route" a condition of its own, and let it out of the package
Review round 1 on task 6. Five blocking findings, all with the same root: the conditions the dialer kept apart were erased one layer out, because every one of them arrived at core/services/nodes as a gRPC codes.Unavailable, which is also what a backend process that died produces. Four call sites acted on that by deleting a replica row, one of them after a single failed probe. The fifth condition is ErrNoRoute: this replica could not get a request to a worker's backend, and no claim at all about the worker. A worker's presence is its HEARTBEAT, which nodes owns; a route is a separate fact that cluster owns, and the two now differ. They differ in normal operation, not exotically: a worker that has not dialled its tunnel yet after a frontend-first upgrade is unroutable on every request while it heartbeats and serves. Two properties, both mutation-tested. Every failure to resolve or open a route carries ErrNoRoute, so a consumer has one check to make. No failure carries an absence sentinel: routeFailure is the single place that rule lives, and it keeps ErrNoConnection and ErrInstanceNotFound in the message and out of the unwrap chain, the guarantee unreachableError already made for peers. Everything else stays matchable, so ErrNotOwner and ErrPeerUnreachable are unchanged for anyone who can act on them. A worker's own refusal carries no umbrella, because a worker that answers has demonstrated it is there and that is the only real evidence on the path. Crossing the boundary needed a value, not a code. NewClientWithDialer wraps the dialer and records each outcome; LastDialError hands it back behind a narrow interface, and nodes.unroutable turns it into ErrWorkerUnroutable with the cluster sentinels still in the chain. A spec asserts a dial failing with ErrNoRoute plus ErrPeerUnreachable arrives matching all three and matching neither absence sentinel. The sweep found a fourth site the review had not named: pkg/model checkIsLoaded evicts a remote model on a connection error, and a tunnel dial failure is one. Four other reap sites were cleared with reasons - inflight and the worker authoritative pass reap only on semantic answers, scale-down is driven by last_used, abandoned loads decide on the node's heartbeat. Every fixed site also grew the opposite spec, so the new check cannot pass by never reaping. probeCache carries the reason through singleflight rather than a closed-over variable. A variable is only written by the goroutine that runs the probe, so the leader would correctly decline to reap while every joiner reaped on the leader's own observation; a mutation reproduces exactly that. The docs sentence promising LOCALAI_WORKER_TUNNEL=false restores direct dialling is gone. There is no such path, so it said the operator could take a worker dark and call it a rollback. Replaced with the upgrade order that is actually safe. The deadline spec the reviewer found vacuous now waits on the dial context's own Done channel before touching the stream, so the armed deadline has really expired; the mutation that survived for the reviewer reddens it. Nine mutations, each reddening a named spec, including both halves of isAbsenceClaim independently. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
75953d9f63 |
feat(cluster): reach every worker through its tunnel, never its address
The tunnel, the fence, the registry and the relay were all built and none of them carried a byte: every dial from the frontend still went to the address a worker registered. This is where that stops. One WorkerDialer resolves where a worker's tunnel is held, opens a stream on it locally or relays through the owning replica, and hands back a conn past both handshakes; gRPC, the file stager's HTTP client and the log-streaming WebSocket are all pointed at it. A worker's address stops being somewhere to connect to and becomes the name of which backend process a stream is for. It still appears in URLs, logs and errors, because that is what identifies the process; what it no longer decides is where the bytes go. Nothing falls back to dialling it. BackendClientFactory now has exactly one method, NewClientForNode, and returns an error where there is no way to reach the worker. The direct-dial constructor was removed rather than kept beside it, because leaving one on the interface keeps the bypass one word away from every call site that holds an address, which is all of them. The second construction path is closed too. DistributedModelStore built remote models with a nil client, and pkg/model.Model.GRPC then dialled the raw address lazily on first use - reached in production by ShutdownModel's Free and by the backend monitor's Status. Those models now carry the tunnel-backed client, and a model that cannot be given one is logged and not listed. Four conditions stay unmixable, and one path produces absence: the dialer answers ErrNoConnection only where Owner's liveness join did. A peer that will not answer, a stale ownership row, a worker's own refusal and a missing relay path are each reported as themselves. This matters because nodes ACTS on absence, and the collapse would have it reclaim the models of a worker that is connected and busy. That is not hypothetical. Writing the mutation for it exposed the bug in this change's own first draft: probeHealth returned bare false when it could not build a client, and tryWarmPath deletes the replica row on a false probe. A frontend whose dialer broke would have emptied node_models for the whole deployment while every model kept running. probeHealth now returns alive and probed separately, the reconciler gets a ProbeUnknown outcome that neither advances nor clears a failure streak, and the health monitor skips rather than counting a miss. Task 5 left the relay's open timeout at a fixed 15s and said so: no operator has the information to set it, because the number that matters is the original client's remaining budget, which is invisible on the relay side. The dialer has that budget, so it now states it in the relay request frame and the owner takes the smaller of the two. It can only shorten - a patient client must not be able to park a relay goroutine and a stream slot on a worker that stopped accepting. Zero is written as no budget at all, since on the far side the number zero is a caller with nothing left and would refuse healthy traffic. Seven mutations, each reddening a named spec: peer-unreachable as absence; the local-failure guard dropped; max instead of min on the budget; the nil-client model restored; ProbeUnknown falling through to the reaper; OwnerRow instead of Owner; probed collapsed into alive. The first budget spec passed for the wrong reason - a handshake deadline, not the relay - and was replaced by three that each assert one link, including one where the spec plays the owning replica and reads the budget out of the frame instead of inferring it from a clock. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
5108be222d |
fix(worker): spec the tunnel's routing table, which was the SSRF boundary
Review follow-up. One blocking finding and seven others. The blocking one first, and it is this project's recurring shape: the untested path. loopbackService is the function whose comment calls the discarded host "the security property this function exists for", and nothing tested it. The reviewer replaced its body with a dial of whatever the frontend named, no port range, and all 131 specs passed. Every spec installed the permissive test dialler, so the real routing table was exercised nowhere. It now has specs, and the property is stated as reachability rather than as a property of the code: a listener on 127.0.0.2 that only the frontend's target names must NOT be reached. Plus the port-range table, fixedService, loopbackAddr, tunnelEndpoint, and the table itself, which moved out of Run into tunnelServices so it can be built without starting a worker. One spec drives a real stream through that table over the wire, so the routing rules are exercised end to end at least once rather than only in isolation. The reviewer's mutation now reddens ten specs, and six narrower ones redden between two and four each, so no spec is riding on another. The shape changed too, not only the coverage. The dial address is built from a loopbackHost constant and strconv.Itoa of a validated int, so nothing derived from the wire reaches DialContext at all: restoring the hole takes ADDING a data flow, not deleting a check. And a taxonomy fix found while specifying it. A port outside this worker's allocator range was reported as unavailable, which tells a frontend to retry something that can never work. It is a bad request now, and a backend that is merely not listening yet stays unavailable, which is the retryable one. Agent nodes no longer get a tunnel credential. Nothing dials into an agent worker, so a tunnel replaces nothing for it and no client would open one, and the gate is at the mint site rather than in the handler: with no credential minted the hash stays empty and the existing empty-hash refusal covers it, so enforcement is structural. Two comments and one doc paragraph said an anonymous registrant gets a "working" credential. With auto-approve off the node is pending and the credential is inert, which is the distinction this same change argues three files away to justify minting for pending nodes at all. A refusal reason over the frame limit was cut on a byte boundary and could split a rune. It cuts on a rune boundary now, and the code survives truncation, which is what keeps a refusal classifiable. Also: the pending-node spec asserted only that a credential was non-empty, so a credential derived from the shared token passed it; it now pins per-node-ness the way the headline spec does. The tunnel handler's citations into nodes.go were stale before this branch landed, having been written against a file the same commit was editing, and are by function name now. The static-NATS path says plainly that an externally forced rotation locks it out until restart, and where that gets fixed. tunnelproto gained direct specs, including that a read failure is never reported as a refusal. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] |
||
|
Ettore Di Giacinto
|
29a2020f3d |
feat(worker): dial, hold and serve the tunnel, on a credential of its own
The worker end of the tunnel. It dials wss://<register-to>/api/cluster/connect, holds one yamux session as the CLIENT, and serves every stream the frontend opens on it. Nothing dials into the worker, which is the point: no inbound port, no reachable address. Each stream opens with a length-prefixed frame naming a tag and a target, and the worker answers before either side speaks the tunnelled protocol. The reply is sent on every stream, not only on refusal, because the protocols carried here are client-speaks-first and a reply sent only sometimes would arrive interleaved with a response body. Two tags today: grpc reaches a backend process, and only on 127.0.0.1 within this worker's own backend port range, because a tunnel terminates inside the worker and letting the frontend name a host would make every worker a proxy into its own LAN; http reaches the worker's file-transfer server, whose address the frontend is not asked about. An unknown tag, an unreachable local service and an unparseable request are three refusals and stay three on the wire. A frontend gives up on the first and retries the second. Each is answered AND the stream is ended: a worker that says why and leaves the stream open has parked the caller on a request nobody will answer, and a deadline on the far side cannot tell that from a slow worker. The specs assert the stream ends rather than that an error occurred, which is what phase 1 shipped in three places and held in none. Reconnects double from 500ms to a 30s ceiling, each wait drawn between half the interval and all of it, and the interval returns to its floor only after a session that LASTED. Resetting on connect is how a rolling restart, where every dial succeeds and dies moments later, becomes a retry storm against the first replica back up. Nothing is assumed to survive a reconnect: the credential is read at dial time, never captured. And the credential is now real. The tunnel endpoint advertised authenticating a worker against its own secret, but registration stored the hash of the shared registration token, so a leak plus a known node ID still opened a tunnel. Registration now mints a per-node secret, returns the plaintext once as tunnel_token, and stores only its SHA-256 in a new column; the endpoint compares against that and does not fall back to the old one. Rotating on every registration follows from storing only the hash, since a re-registering worker cannot be told the secret it already holds; its live tunnel is unaffected, because the credential is checked when a tunnel is dialled and never again. Unlike the agent API key and the NATS JWT next to it, the credential IS issued to a node awaiting approval: the tunnel route re-reads the node's status on every dial and refuses a pending one, so it is inert until an admin acts, and withholding it would strand every worker that registers exactly once. A node that has not registered since this change cannot tunnel, and the column cannot be back-filled because the plaintext only ever existed in the response that minted it. The boot warning that said tunnels need LOCALAI_REGISTRATION_TOKEN is replaced: it was true while the tunnel authenticated against that token's hash, and says the wrong thing now. What is still true, and is what it warns about instead, is that without one, registration itself is unauthenticated. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
48ece89c63 |
fix(cluster): harden the worker tunnel, and stop starting a database per spec
Review follow-up. Twelve findings, none blocking, grouped here by what they protect. Panics. The handler now recovers between the WebSocket upgrade and the hand-off, the way the peer link next door already did: net/http recovers the panic but leaves the hijacked socket open, so without this a worker keeps a session this replica has no entry for and will never detach. The claim gate in Attach and reclaimOne is now released with defer, so a panic under Claim cannot wedge one node's gate for the life of the process. SetTunnels gained the nil-receiver guard its sibling Stop has. Operability. A deployment with no registration token stores an empty token_hash on every worker, so every tunnel dial 401s forever on a frontend that looks correctly configured. That now warns at startup, logs its own line rather than sharing the "wrong token" one, and is stated in the docs together with the fact that setting the token later needs the workers to register again. Authorization. A node still awaiting admin approval is refused with 403. The rest of /api/node/ gates on nothing, but the two places that hand a node something durable, its API key and its NATS credential, both refuse a pending one, and a tunnel is that kind of grant. Draining and unhealthy nodes keep their tunnels on purpose. Comments that claimed more than the code. The global auth middleware does run on this path and then declines to reject; the future per-node secret only lands without a change here if it lands in TokenHash; the empty-hash guard is defensive rather than deciding; ClusterPathPrefix is no longer only replica-to-replica; the docs no longer say a reaped replica re-claims unconditionally. And the test harness. SetupTestDB started a PostgreSQL container per BeforeEach with a readiness deadline it asserted on, which is one chance per spec to fail one spec inside its setup, anywhere, never twice in the same place: the shape of the flake seen twice here and never reproduced. It now starts one container per process and creates a database per call, which is the pattern tests/e2e already proved. Isolation is unchanged and is now asserted for the first time. All 69 call sites are untouched; the eleven consumer packages run 1404 specs green, and jobs went from 34.3s to 3.3s, agents from 13.8s to 1.9s, cluster from 97.4s to 37.5s. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] |
||
|
Ettore Di Giacinto
|
6e55092a4b |
feat(cluster): open the door a worker dials its tunnel through
A worker needs no inbound port: it dials GET /api/cluster/connect, the connection becomes one multiplexed yamux session, and the frontend opens a stream on it per request. This adds the endpoint that accepts that dial and attaches it to the tunnel registry. The dial is authenticated against the NODE's own stored token hash rather than the deployment's registration token. That is the mechanism, not yet the isolation, since a worker still registers by presenting the shared token; what it rules out is the shortcut of comparing against the configured value, which would have to be unpicked the day workers get their own secrets. Every refusal happens BEFORE the WebSocket upgrade, so a dialer reads an HTTP status rather than a handshake error. The route is registered in every deployment, single-binary ones included, which is what puts it in front of the route-coverage test that holds that rule in place; with no node registry it refuses every dial, and tells a credentialed one the frontend has no cluster rather than that its token is wrong. A lookup that FAILED is answered as a failure. Reporting a database that could not be read as "unauthorized" would send a worker re-registering, throwing away the identity its tunnel and loaded models are keyed by. Wires the tunnel registry in core/application/distributed.go and hands it to the membership loop. Without that call the re-claim after a replica is reaped had no production caller and could never run. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] |
||
|
Ettore Di Giacinto
|
aca383d263 |
feat(cluster): give phase 1 a call site, and prove it against real replicas
Tasks 1 to 5 built an instances table, a splice, both halves of a peer link and an epoch fence, and nothing in the tree called any of it: no replica registered, no route was mounted, no sweeper ran. Proving phase 1 end to end therefore had to start by wiring it. A frontend in distributed mode now publishes the address its peers dial, heartbeats it, and sweeps replicas that stopped answering along with the connection rows they owned, in one pass so the two can never disagree about who is alive. It serves the peer link and owns the sessions peers dial in, refusing streams on them until phase 2 installs a relay: a session nobody accepts on does not fail a peer's Open, it hangs it. The address is the one peers use, not the one the process binds, and it is derived from the route to PostgreSQL. That derivation only holds while the database is remote, so LOCALAI_DISTRIBUTED_ADVERTISE_ADDR sets it explicitly and a replica that can determine neither warns and keeps serving rather than failing to start. Three e2e scenarios run against real local-ai processes, real PostgreSQL and real dials: replicas publish addresses that can actually be connected to; a sibling opens a stream over the peer link and is refused without the cluster token; and a killed replica is reported unreachable, never absent, loses the claim it held, and takes no worker with it. Each was verified by mutation: eight injected defects, each failing the scenario that claims to catch it. Also moves RegisterClusterRoutes to core/http/routes beside every other registrar, folds AutoMigrate and the epoch sequence into one cluster.Migrate, and turns the peer route's auth-coverage spec into a real assertion: it drives the request through the actual auth middleware instead of comparing two string constants, which the old spec would have passed even with the exemption deleted. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
7aeb47cbf3 |
fix(launcher): auto-start the server so launching the app actually serves
Fixes #11673: on macOS the DMG launcher appeared to launch nothing. After installing, the app sat in the menu bar with no window, nothing listening on localhost:8080, and empty log files, because nothing ever started the server unless the unrelated 'start on system boot' option was enabled. - Start the LocalAI server automatically when the launcher opens and right after a fresh install. The new auto_start_server config key defaults to enabled and gets a settings checkbox; the legacy auto_start key was never honored nor exposed, so every existing launcher.json carries an unintentional false and is deliberately left behind. - Fix the welcome window suppressing itself: its 'don't show this again' checkbox was initialized with the inverted value, and SetChecked fired the change callback which persisted ShowWelcome=false on the very first showing. - Surface auto-start failures through the systray startup-error dialog, since there is no visible window during auto-start. - Pass --app-version to fyne package so the app stops reporting itself as version 0.0.0 in the About box. - Document the first-launch flow (menu bar app, auto-start, WebUI URL) in the macOS getting-started page. - Repair two launcher specs that never ran in CI: a *bool matched against BeTrue and a /tmp assertion that trips on Linux where the test tempdir itself lives under /tmp. Assisted-by: Claude Code:claude-fable-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
893a45141c |
fix(realtime): accept GA WebRTC signaling (#11778)
OpenAI GA clients send multipart or raw SDP requests. They expect a bare SDP answer. LocalAI only accepted its legacy JSON envelope, so signaling failed before media setup. Keep the JSON contract for existing clients. Accept both GA request shapes and choose the matching response format. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
80e3240f2d |
feat(distributed): key scheduling rules by a model alias (#11771)
Node placement and replica rules could only name a model, so an operator who pinned "llama3" to the GPU tier had to rewrite the rule whenever a different model took over that job. An alias already gives a stable name for whichever model serves it, and a rule on that name makes it a deployment slot: repoint the alias and the placement follows. A rule keeps the name the operator chose. Reads resolve that name through the config loader to the model the rule governs, so the reconciler counts, schedules and trims replicas of the target, and the router finds an alias-keyed rule from the target it is already routing. An alias that resolves to nothing governs nothing loadable, so the reconciler skips it and the write paths refuse it. A replica is shared by every name that resolves to it, so only one rule can decide where it runs. The REST and MCP write paths reject a rule whose target another rule already governs. A pair that arrives some other way, such as a seed file or an alias repointed onto a model that already has a rule, resolves in favour of the rule named after the model itself and then the oldest, and the rest are listed as shadowed. The eviction guard is the exception: it matches rules to replicas in raw SQL inside a locking transaction and cannot resolve an alias. It reads a stored target that the reconciler refreshes each tick, and falls back to the rule's own name when that target is empty. Assisted-by: Claude:claude-opus-5 golangci-lint eslint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
29899cd1e0 |
fix(ui): size model fit against the cluster and move node labels into the selector (#11765)
* fix(ui): move node labels into the scheduling selector field The scheduling page kept a node-label browser open above the rules whether or not anyone was writing one, while the field that actually needs labels, the rule's node selector, was two bare text inputs with no hint of what the cluster reports. The browser is gone. The selector's key input now completes against the label keys the cluster uses, and the value input offers only the values that key takes. The roster already loads for the page, so the suggestions cost no request, and a roster that fails to load costs the admin the hints and nothing else. Suggestions stay suggestions: a key no node reports yet still commits as typed, which is how an admin writes a rule before labelling the nodes for it. Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): size model fit against the cluster, not the frontend The models page asked the frontend how much memory a model may occupy. In distributed mode the frontend is usually a GPU-less pod while every model runs on a worker, so a fleet of GPU nodes was told it could only run the smallest CPU build. The variant picker's fits flag and its auto-selection came from the same place, as did the hardware recommendations. The registry now reports the largest single healthy backend node. The largest node, not the fleet total: a model loads into one node, so four 16GB workers are not a home for a 40GB model. An operator-set VRAM budget caps a node's contribution, because the scheduler refuses a load above that ceiling anyway, and a GPU node beats a CPU node holding more system RAM. GET /api/resources and GET /api/models carry this as an additional cluster object. Their aggregate and ram fields keep reporting the frontend's own hardware, which is what the resource monitor shows. Variant selection judges backends against the union of the capabilities present in the cluster, the way backend discovery already did. Every path degrades to the local host: no cluster object in single-node mode, and none when the registry cannot be read, so a hiccup narrows the answer back to single-node behaviour rather than marking the whole catalog too large. The verdicts now name the node they belong to, since a model fits somewhere or nowhere. Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
d85577ff5c |
docs: add Apache APISIX reverse proxy example (#11294)
docs: add APISIX reverse proxy example Document the route settings needed for forwarded headers, streaming responses, and long-running inference behind Apache APISIX. Closes #11215 Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Ettore Di Giacinto
|
3953448f60 |
fix(distributed): resync stored config revisions at startup
The controller pins a model's replicas to a stored revision and rejects any request carrying a different one. Nothing ever re-derived that value from the configuration on disk: it moved only on an edit, a gallery install, or a peer's change broadcast. An inference request may only establish a revision, never replace one. So any other way for the two to diverge left the model permanently unroutable. A configuration edited while a frontend was down lands there, and so does a change in what the revision is computed over: an upgrade that alters the hashed form leaves every stored revision describing a configuration that no longer exists. The only recovery was deleting the row by hand, which is not something a cluster should need. Each frontend now reconciles the stored revisions against the loaded configurations at startup and republishes the ones that disagree. Only those: republishing quarantines every replica loaded under the old revision, so doing it for a model that did not drift would unload a healthy replica for nothing. A model with no stored revision has never been served and is left for its first request to establish. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
Ettore Di Giacinto
|
e6269e3cdd |
fix(distributed): reclaim replica slots held by abandoned loads
A replica row in staging or loading holds its slot, because slot allocation counts every state except unloading. Nothing ever reclaimed such a row: every reconciler pass and the router's eviction query filter state = "loaded", and the per-model probe skips rows without an address, which is exactly what a row that never finished loading has. So a worker that dropped out mid-transfer left a row that pinned the only replica slot for that model on that node. Scheduling then found no free slot and eviction found nothing it was allowed to evict, and the request failed with "no replica slot on <node> and eviction failed: all models busy". The state persisted until an operator intervened. The reconciler now reclaims a row stuck before serving when no load job is driving it. Ownership is decided by the job's LastProgress heartbeat, not by elapsed time: staging a large checkpoint legitimately runs for a long while without touching the replica row, so a deadline would either be a model-size cliff or reclaim a healthy transfer. That heartbeat is the same signal job takeover already trusts. Any error reading the job leaves the slot held, because holding one for another pass costs a scheduling opportunity while a wrong reclaim restarts a multi-gigabyte transfer. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
Ettore Di Giacinto
|
c541dbeef4 |
fix(distributed): check a node answers before scheduling onto it
A node's status comes from its HTTP heartbeat. Backend installs travel over NATS. The two are independent, so a worker that dies stops answering on the bus at once but stays healthy in the database until its heartbeat ages out. Inside that window the scheduler picked a node it could not reach, and the request failed with "no responders available" rather than moving to a node that was up. The scheduler now probes the node it selected and, when nothing answers, marks it unhealthy and selects again. The demotion is what makes the retry terminate: the next selection reads only healthy nodes. It also tells the other frontends what this one learned, so the cluster does not rediscover a dead worker one failed request at a time. Only nats.ErrNoResponders counts as absent. A worker that answers slowly stays eligible, because dropping it would cost capacity that is really there. The probe reuses the models.running subject: a new subject would go unanswered by workers that have not been upgraded, and every one of them would then look dead. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
Ettore Di Giacinto
|
cee87d1608 |
fix(distributed): expire staged request files on the worker
A request that carries a file stages it to the worker, which writes it under its staging directory. Nothing removed it afterwards. The frontend expires ephemeral keys from object storage, but that sweep never covered a worker's local disk, so every image, audio clip and video a worker ever served stayed on it. One worker had accumulated 175 request directories over three months. The volume reached 100 percent, and from that point every backend start failed because the process manager could not create a state directory. The worker now sweeps its ephemeral staging directory on a timer and once at startup, so files left by a crash are reclaimed too. Staged model files live beside that directory and are not touched. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
Ettore Di Giacinto
|
04735cd1f6 |
fix(distributed): stamp config revision at load time
The request middleware merges the caller's prediction parameters into its copy of the model config. core/backend.ModelOptions then hashed that copy, so the revision identified the request body rather than the persisted configuration. EstablishModelConfigRevision stores the first revision it sees and requires an exact match afterwards. The first request after a restart therefore pinned the model to its own temperature, top_p and stop values, and every later request that sent different ones failed with "stale model config revision". No config edit was involved. The loader now stamps the revision when it materializes a config, before any request override reaches it, and ModelOptions reads that stamp. Model administration keeps hashing the same persisted config, so both paths agree on one revision per configuration. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
8f56e4e042 |
fix(vram): persist remote probe metadata (#11487)
* fix(vram): persist remote probe metadata The startup warmer repeated remote size and GGUF metadata probes after every restart because both caches lived only in memory. Store successful HTTP probes for 24 hours so frequent restarts reuse the prior results. Bound the cache, reject invalid records, and purge it when gallery data changes. Local model files continue to bypass persistence. Assisted-by: Codex:gpt-5 * fix(vram): check temporary file cleanup The lint gate rejects the unchecked cleanup call in the persistent cache writer. Assisted-by: Codex:gpt-5.6 [golangci-lint] * fix(vram): make persistent cache optional Remote metadata probes can transfer enough data that operators need control over disk reuse and startup warming. Gallery autoload now gates both behaviors, and the runtime setting applies changes immediately. Assisted-by: Codex:gpt-5 * fix(ui): expose gallery startup pre-warm The existing gallery autoload setting also gates the startup metadata warmer. Name both effects in Settings so operators can find the requested boot control. Assisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
82c191afad |
fix(distributed): keep model replicas config-consistent (#11664)
* docs: design configurable copy buffering Document the context-aware copy buffer option and its validation plan. Assisted-by: Codex:gpt-5 * docs: design durable distributed staging operations Assisted-by: Codex:gpt-5 * docs: design distributed model config revisions Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] * feat(config): add stable model revisions Hash typed model configuration and effective protobuf options deterministically for distributed revision comparisons. Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] * feat(worker): acknowledge exact model stops Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] * feat(nodes): track model config revisions Assisted-by: Codex:GPT-5 [apply_patch] * fix(distributed): retry quarantined model cleanup Stop quarantined replicas by exact process identity, retain failed cleanup as durable capped retries, and compare-and-delete only the claimed registry row. Process one sufficiently leased row at a time so multiple frontends cannot duplicate slow cleanup work. Assisted-by: Codex:gpt-5 * fix(distributed): bind loads to config revisions Assisted-by: Codex: GPT-5 [OpenAI Codex] * fix(modeladmin): apply config revisions consistently Route model edits, patches, state changes, deletion, and peer refreshes through the same revision lifecycle. Quarantine stale replicas before exact cleanup and report durable pending cleanup without failing successful config writes. Assisted-by: Codex: GPT-5 [OpenAI Codex] * feat(distributed): expose model config revision state Document replica revision observability and durable cleanup behavior. Keep pending cleanup explicit in model mutation responses and verify endpoint contracts expose revision state without serialized load options. Assisted-by: Codex:GPT-5 [OpenAI Codex] * test(distributed): cover model revision convergence Exercise cross-frontend quarantine, stale replay rejection, exact cleanup retry, worker re-registration, and current-generation replica convergence against the distributed PostgreSQL harness. Assisted-by: Codex:gpt-5 * fix(distributed): pass config revision CI checks Keep configured gallery sources out of authoritative runtime snapshots only after validating their real schema, and harden rollback snapshots against symlink races and non-regular files. Assisted-by: Codex: GPT-5 [OpenAI Codex] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
9d92139de4 |
feat(ui): edit scheduling rules in place (#11667)
* docs(ui): design scheduling rule editing Document the approved in-place rule editing flow and scalable node-label reference for the scheduling view. Assisted-by: Codex:gpt-5 * feat(ui): improve scheduling rule management Add scalable node-label discovery and editable scheduling rules with responsive, accessible controls. Assisted-by: Codex:gpt-5 * chore(ui): ratchet inline style baseline Record the static inline style removed by the scheduling view enhancement. Assisted-by: Codex:gpt-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-botandlocalai-org-bot
|
3684a534bb |
docs(website): simplify installation paths (#11631)
Keep the homepage focused on runtime capabilities and move engine details to their canonical directory. Make installation choices stable and explicit for users across supported hardware. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-bot <306113404+localai-org-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
a0252ad6a1 |
fix(distributed): keep staging operations stable (#11663)
* docs: design configurable copy buffering Document the context-aware copy buffer option and its validation plan. Assisted-by: Codex:gpt-5 * docs: design durable distributed staging operations Assisted-by: Codex:gpt-5 * fix(distributed): merge durable staging operations Use active model load jobs as the durable operations baseline and overlay replica-local staging progress without duplication. Preserve tracker-only operations when the registry cannot be read. Assisted-by: Codex:gpt-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
5072219829 |
feat(xio): make copy buffer size configurable (#11660)
* docs: design configurable copy buffering Document the context-aware copy buffer option and its validation plan. Assisted-by: Codex:gpt-5 * feat(xio): configure context copy buffer size --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andmudler
|
80dd0fd076 |
docs: ⬆️ update docs version mudler/LocalAI (#11643)
⬆️ Update docs version mudler/LocalAI Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
0ab632b6bd |
fix(auth): protect HTTP routes by default (#11602)
* fix(auth): default to protected HTTP routes Use a method-aware registry for the small anonymous bootstrap surface. Unknown routes now require credentials instead of inheriting fail-open path classification. Keep node self-service routes behind their registration-token middleware. Global auth no longer rejects valid worker credentials first. Assisted-by: Codex:gpt-5 * docs(auth): document public HTTP surface Assisted-by: Codex:gpt-5 * test(auth): align route coverage with default denial Assisted-by: Codex:gpt-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
2383726d6d |
Revert "chore(tests): Avoid network, sleep and more during tests" (#11601)
Revert "chore(tests): Avoid network, sleep and more during tests (#11050)"
This reverts commit
|
||
|
Richard Palethorpeandlocalai-org-maint-bot
|
cb3bf7af3f |
chore(tests): Avoid network, sleep and more during tests (#11050)
* test: make coverage failures observable Keep per-root logs, reject concurrent coverage runs, and avoid relying on /bin/sleep in the worker timeout test. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: parallelize coverage without remote fixtures Assisted-by: Codex:gpt-5 [apply_patch] [exec_command] Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: add offline resource infrastructure Introduce versioned resource manifests, a checksum-verified CAS preparer, offline test wrappers, and a guarded network transport. Replace live Hugging Face, GitHub, and OCI cases with deterministic fixtures and inject fixture metadata into importer discovery. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: enforce offline resource replay Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: harden offline resource refresh Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: expose slow coverage waits Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: eliminate avoidable wall-clock waits Inject a clock into Hugging Face retry handling, reuse a process-scoped PostgreSQL container with per-spec schemas in the nodes suite, and poll local import jobs promptly. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: remove repeated fixture startup waits Share PostgreSQL fixtures across parallel endpoint and agent suite workers, and make the worker Free deadline injectable so the wedged-backend test does not spend five seconds on wall-clock time. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: fix offline resource CI portability Normalize Docker archive metadata before content addressing, derive archive checksums during explicit refreshes, make network lint portable to macOS, and prepare distributed images before running their offline suite. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: cache Go modules before offline tests Warm the complete module graph before the Linux and macOS test jobs enter offline replay mode, so tool dependencies such as Ginkgo are not fetched through the guarded proxy. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: drop the static network lint in favour of real isolation The offline test suite already prevents tests from reaching the network twice over: run-test-linux-offline.sh puts the test process in a cgroup and REJECTs egress outside the private ranges, and HardenedTransport installs testnetwork.LocalGuard to refuse dials that resolve to a public address. Both fail the test with a precise error at the moment of the dial. test-network-lint.sh added neither. Its diff stage defaulted to a HEAD base, so on a clean checkout it compared the tree against itself and inspected nothing; the branch's own commits were never examined. It only produced output when an earlier job step dirtied the tree, and then it matched a bare https?:// against whatever changed. make react-ui runs npm install rather than npm ci, so CI rewrote core/http/react-ui/package-lock.json and the lint reported an npm registry URL as forbidden test network access: + "resolved": "https://registry.npmjs.org/hono/-/hono-4.12.25.tgz", Its fingerprint stage was self-defeating in a quieter way: hashing the whole tree's network-mechanism inventory meant every rebase onto a master that touched any _test.go needed a manual baseline bump, so the check mostly caught its own staleness. Remove the script, its make target and the two prerequisite edges, along with the test-network: fixture markers that existed only to suppress it. The isolation itself is untouched. Assisted-by: Claude:claude-opus-5 [go vet] Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: keep hidden files in the offline test bundle artifact Cherry-picked from |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
0761bd02c7 |
feat(chat): add end-to-end context compression (#11556)
* feat(config): add context compression policy Define the opt-in model configuration contract before the chat middleware consumes it. Document each policy field so later request handling does not invent a second schema.\n\nRefs #9534\n\nAssisted-by: Codex:gpt-5 * fix(config): register compression fields The model editor metadata gate rejects new config fields without descriptions and suitable controls. Register the compression policy so operators can edit its six fields safely. Assisted-by: Codex:gpt-5 [monitoring-prs] * feat(chat): compress long contexts Long conversations currently fail once they reach the model context window. The opt-in policy now summarizes complete older turns before primary inference and preserves the newest tool chains. Both OpenAI and MCP chat routes share the same transformation. Usage metadata and metrics expose each compression event. Refs #9534 Assisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Richard Palethorpe
|
d10374f849 |
feat(router): make KNN a first-class classifier with a persisted, curated corpus (#10652)
* feat(router): make KNN a first-class classifier with a persisted, curated corpus
Add `classifier: knn` — similarity-weighted voting over labelled
example prompts. Unlike score/colbert it needs no classifier model:
label knowledge lives in a corpus seeded and curated through the
admin API, so routing decisions are deterministic, auditable, and
grounded in graded experience rather than a model's opinion.
Epistemic gate: corpus entries below knn.similarity_threshold cannot
vote; when none clears it the classifier activates no labels and the
router uses the fallback — a prompt unlike all labelled experience is
treated as undecidable, not guessed. Decisions record
nearest_similarity (also on fallback rows) so admins can see how far
the nearest labelled experience was; the Routing tab explains
out-of-corpus fallbacks and shows per-label corpus counts.
Persistence: one JSONL file per router under
<data path>/router-corpus (text, labels, vector, embedder
fingerprint). The file is the source of truth; the local-store index
is rebuilt from it at classifier build time and stays a pure
in-memory index. Entries recorded under a different embedding model
re-embed on load. Also corrects the docs' false claim that
local-store collections persist — the embedding cache never survived
restarts (and still doesn't); the corpus does.
Corpus input is API-only by design (entries may contain example user
content): POST /api/router/{name}/corpus seeds (labels validated
against declared policies, embedded server-side, indexed
immediately), GET .../corpus/stats inspects — label counts only,
entry texts are never returned by any surface — DELETE .../corpus
wipes. Admin-gated like the sibling router endpoints, and exposed as
MCP tools (seed_router_corpus / get_router_corpus_stats /
clear_router_corpus) in both the httpapi and inproc clients with
coverage-test route mappings.
Plumbing: VectorStore gains SearchK (top-K was hardcoded to 1);
local-store gets InsertBatch/Delete as optional fast paths;
RouterConfig gains a knn block (embedding_model, k,
similarity_threshold, vote_threshold, store_name) with meta-registry
fields; the classifier dropdown now offers knn and the
previously-missing colbert; embedding_cache is ignored (with a
warning) for knn — it IS an embedding-KNN lookup; the stale
/api/instructions intelligent-routing entry is rewritten (it
described a classifier that no longer exists); swagger regenerated.
Tests: KNN vote/gate specs with hand-computed vote shares, corpus
manager suite (restart reload without re-embedding, fingerprint
re-embed, dedupe, hostile store names), middleware specs (corpus
routing, gate fallback, config validation, cache-wrap refusal),
corpus endpoint specs pinning the texts-never-returned contract, MCP
catalog + route-mapping gates, and a Playwright spec for corpus
stats and the out-of-corpus decision detail.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(router): name consulted corpus neighbours in knn decisions
Every knn decision (decision log rows and the /api/router/decide
response) now carries neighbors: the K retrieved corpus entries by
descending similarity - including ones below the epistemic gate, which
is what makes fallback decisions diagnosable - each as {id, similarity,
labels}. The id is the entry's content hash (first 8 bytes of the
SHA-256 of its text, hex): stable across reseeds and re-embeds, and
text-free, so an external platform that seeded the corpus can recompute
text->id on its own copy and bucket decisions by corpus region (per-
region reliability accounting) without corpus text ever leaving the
server. A corrupt index payload surfaces as an id-less neighbour at a
real similarity instead of disappearing.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* refactor(router): deduplicate knn plumbing and cut corpus hot-path waste
Post-review cleanup of the knn-first-class-router branch; no behaviour
changes on the API surface.
Reuse/altitude:
- RouterKNNConfig.ResolvedStoreName is now the single source of the
router-corpus-<name> default (was hand-derived in four files).
- corpus.ResolveKNNRouter + corpus.Seed carry the shared model
resolution and seed validation; the REST endpoints and the assistant
MCP client are thin transport adapters over them, with sentinel
errors mapped to HTTP statuses at the echo boundary.
- middleware.NewClassifierDeps assembles the classifier dependency set
once for all five entry points (OpenAI, Anthropic, realtime, decide,
corpus) instead of five hand-copied literals.
- router.AllClassifiers feeds both the status endpoint and the
unknown-classifier error, ending the classifier-list drift.
- Per-classifier requirements moved out of validateRouterPolicies into
their buildClassifier arms; the knn arm owns its embedding_cache
opt-out instead of a name-check in the shared wrap tail.
- adminOnly replaces four inline copies of the admin gate in the
middleware routes.
- localVectorStore.Search delegates to SearchK (identical traces).
Efficiency:
- Manager.Add embeds outside the manager mutex and appends to the
JSONL file (O(new) instead of O(corpus) rewrite); a torn tail from a
crash mid-append is tolerated on read and repaired on next write.
- Stats memoises per store keyed on the file's stat fingerprint and no
longer takes the manager mutex, so the 5s status poll stops parsing
vector-laden JSONL and stops blocking behind seeds.
- KNN Classify decodes each neighbour payload once (was twice) and
builds refs and votes in a single pass with one fallback return.
- Corpus file writes fsync before rename/close.
- The corpus manager is built eagerly in newApplication (sync.Once
dropped); test helper dead branch removed.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(router): bind knn corpus vectors to an embedder fingerprint and fail closed on mismatch
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* chore(mcp): align corpus tool prompts and the mutating-tool safety list
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(proto,backend): report embedding shape from the llama-cpp backend
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(embeddings): Go-side pooling — mean/last/decayed_mean with half-life
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(embeddings): accept chat messages[] and per-request pooling on /v1/embeddings
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* chore(middleware): name the failing fields when post-merge validation 400s
An intermittent post-merge validation failure surfaced as an opaque 400
during integration (pooling scheme mismatch that no client had sent).
Log the model, the request's pooling override, and the merged config's
pooling fields at the failure point so the next occurrence identifies
whether the request or the stored config carried the bad value.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(embeddings): scheme override must not inherit the config's half-life
A model config defaulting to decayed_mean pooling carries
pooling_half_life_tokens; a request overriding the scheme to mean/last
without its own half-life inherited that value, and post-merge
validation rejected the pair the server itself had assembled. Zero the
inherited half-life when the overridden scheme is not decayed_mean; a
request that explicitly pairs a half-life with a non-decayed scheme
still 400s.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix embedding pooling validation and router bounds
Declare backend embedding layouts and reject incompatible pooling modes. Reset local-store dimensions after a full clear, validate KNN thresholds, and add real backend and store integration coverage.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* ci: run local-store integration tests
Build and install the local-store backend in the Linux test job, then run the existing store integration suite so new specs are discovered automatically.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
|
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
a7bce6a128 |
fix(audio): reject incompatible transform streams (#11565)
The transform WebSocket accepted any model and opened its frame-based RPC. Any-to-any models use a different stream contract, so liquid-audio failed with an unimplemented RPC after the handshake. Reject incompatible model use cases before loading the backend. Direct realtime-audio callers to the OpenAI Realtime API. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Richard Palethorpe
|
799cc9f211 |
feat: bound global admission and expose running backend traces (#11560)
feat: bound backend admission and expose running traces Add process-wide backend execution admission without blocking UI or administrative HTTP work. Represent backend operations while they are in flight, surface running traces with immediate log links, and tie streaming admission leases to the gRPC receive lifecycle. Assisted-by: OpenAI Codex: GPT-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
0aaff91ebd |
feat(ui): unify model and backend lifecycle (#11548)
* feat(ui): add installed model lifecycle Models now owns catalog exploration and installed runtime controls under one canonical route. URL-owned state keeps lifecycle context recoverable through links and browser history. Assisted-by: Codex:gpt-5 Playwright * feat(ui): add installed backend lifecycle Backends split discovery from backend-binary management. The canonical page now keeps both lifecycle views under one URL-backed shell while it preserves target-node placement. Assisted-by: Codex:gpt-5 Playwright * fix(ui): repair lifecycle state updates Installed models lost distributed refreshes and kept a deleted selection. Backend searches also stopped tracking URL changes, while batch upgrades stopped after their first error. Preserve background refreshes and finish each requested batch action. Drive catalog results from URL-backed state without losing full metadata. Assisted-by: Codex:gpt-5 [Playwright] * feat(ui): make resource pages canonical Replace Host navigation with canonical Models and Backends lifecycle routes, preserve legacy management URLs, and surface shared host capacity on the Operate overview. Assisted-by: Codex:gpt-5 [Playwright] * feat(ui): complete canonical resource lifecycle Finish the responsive list-to-detail behavior, remove the retired Host implementation, and keep Explore focused on discovery while Installed owns destructive actions. Update regression coverage, localization, documentation, and development binding for the canonical resource pages. Assisted-by: Codex:gpt-5 [Playwright] * docs(ui): record the UI design context Record the approved users, brand character, and design principles so future interface work uses the same product direction. Index the context from the repository's agent instructions. Assisted-by: Codex:gpt-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
6fb9ab38aa |
feat(gallery): add vllm.cpp text-generation models (#11511)
Adds eight curated vllm-cpp entries to the model gallery. Until now the backend had gallery coverage only for MiniMax-H3 video, so serving text on it meant hand-writing engine_args. The flagship tier is what vllm.cpp gates its correctness and speed claims on: Qwen3.6-27B and Qwen3.6-35B-A3B in NVFP4, each with a speculative sibling (MTP on both, DFlash on the 27B). Qwen3-Coder-30B-A3B covers agentic tool use, and Qwen3-4B / Qwen3-0.6B in bf16 are the entries that run where NVFP4 cannot, CPU included. Three details are load-bearing rather than incidental: - The 27B entries pin revision 890bdef7. That repository was later re-quantized in place from NVFP4 to FP8 W8A8 under the same name, so an unpinned entry resolves to different weights and reports nothing. - Qwen3-Coder names tool_parser: qwen3_coder explicitly. Its dialect is byte-identical on the wire to step3p5's, so chat-template sniffing cannot separate them and auto-detection picks wrong. - enable_prefix_caching is deliberately left unset everywhere. It defaults on for dense models and off for the GDN hybrids, and that per-model default is the right answer. num_blocks is sized per model from its real KV footprint rather than copied between entries, which ranges from 20 KiB/token on the 35B to 144 KiB/token on the 4B. Docs: adds features/vllm-cpp.md covering installation, the model table, the pinning rationale and how to choose between the speculative variants, and cross-links it from the existing engine_args reference. It also records that the CUDA images are built for Blackwell only, which is narrower than vllm.cpp's own ten-architecture release and makes an otherwise cryptic "no kernel image is available" failure legible. Verified: gallery suite green; all eight decode and validate as a ModelConfig. qwen3-0.6b-vllm-cpp confirmed end to end on a real cluster, chat plus engine-parsed tool_calls. The NVFP4 entries are not yet runtime-verified: no available node has kernels for them. Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Bash] [Edit] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
88edd7fc7f |
fix(distributed): run cold model loads as durable jobs instead of holding the advisory lock (#11514)
* fix(advisorylock): set statement_timeout alongside lock_timeout
WithLockCtx already overrides a deployment-wide lock_timeout on its
dedicated connection so a blocking pg_advisory_lock() waits its turn
instead of failing with 55P03. statement_timeout aborts that exact same
statement independently, with SQLSTATE 57014, and was not overridden.
Production roles commonly carry statement_timeout=60s. Any guarded
section longer than that (a cold model load stages for tens of minutes)
therefore killed every concurrent waiter:
advisorylock: acquiring lock 9003261067483446873: ERROR: canceling
statement due to statement timeout (SQLSTATE 57014)
Derive it from the same context budget as lock_timeout, with a matching
RESET so the pooled connection is returned clean.
Assisted-by: Claude Opus 5 [claude-code]
* feat(distributed): add ModelLoadJob, the durable cold-load record
A cold load in distributed mode is a long-running background job, but it
was modelled as a synchronous side effect of an inference request: the
whole of it (backend install, multi-GB staging, checkpoint load) ran
inside the per-model advisory lock. Loading a 35.7 GB GGUF held that lock
for ~20 minutes, so every concurrent request for the same model blocked
on pg_advisory_lock and died at the role's 60s statement_timeout.
Introduce the row that lets the lock shrink to a decision. Exactly one
ModelLoadJob may be active per tracking key; that uniqueness — not the
lifetime of a lock — is what de-duplicates concurrent loaders across
replicas. ClaimLoadJob does its read-then-write under the advisory lock
and nothing else: no network, file or gRPC I/O inside the guarded
section, so a claim costs milliseconds no matter how long the resulting
load takes.
LastProgress is a heartbeat rather than a byte counter. A checkpoint load
legitimately moves zero bytes for many minutes, so a reaper keyed on byte
movement would reclaim a healthy job mid-load; byte progress stays the
concern of load_deadline.go. A job whose heartbeat stops for longer than
the orphan window is reclaimable, so a replica killed mid-load cannot
wedge a model permanently.
Failed jobs keep their row for a short grace so an immediately-following
request reports the real cause instead of silently starting a fresh load
of a model that just failed.
No caller yet — the router moves onto this in the next commit.
Assisted-by: Claude Opus 5 [claude-code]
* refactor(distributed): run cold loads as jobs, outside the advisory lock
Route wrapped the entire cold load — node selection, backend install,
multi-GB staging and the remote LoadModel — in the per-model advisory
lock. The lock's job is to de-duplicate concurrent loaders, a decision
that takes milliseconds; holding it for the tens of minutes the resulting
work takes is what turned a dedup mechanism into a cluster-wide outage
for that model.
Split it into a claim and a run. The claim is the only thing left inside
the lock. The run is a background job owned by the claiming replica and
bounded by the same progress-extended deadline as before; every other
request for that model — local or on another replica — attaches as a
waiter and is served the moment the model is ready, with no duplicate
load and no lock contention.
Waiters share one broadcast rather than an ordered queue: they all want
the identical outcome, so ordering them would add fairness machinery that
changes no result. The local channel wakes same-replica waiters instantly
and a 2s DB poll is the authority, because a waiter on another replica
has no channel to close. On wake a waiter re-runs the warm path rather
than trusting the signal — the model may have been evicted in between.
A waiter whose client disconnects returns immediately and the job keeps
running; it belongs to the job record, not to the request. A failure is
recorded on the row so every waiter reports the real cause, and the row
survives briefly so the next request does not read "no job" as "not
loading" and start a duplicate load of a model that just failed.
The runner heartbeats the row on a fixed interval whether or not bytes
are moving, which is what keeps a legitimately silent checkpoint load
from being reclaimed as an orphan. Phase (installing/staging/loading) and
placement ride to the heartbeat on the context, the same seam
load_deadline.go already uses, so single-host paths are untouched.
Non-distributed mode (no DB) keeps the inline load exactly as it was.
Assisted-by: Claude Opus 5 [claude-code]
* feat(distributed): bound the wait for a loading model and answer with progress
A request whose model is cold-loading now attaches to the running job and
is served the moment the model is ready. That wait has to be bounded: a
held HTTP request cannot survive real infrastructure, and an ingress or LB
idle timeout kills a twenty-minute request regardless of what LocalAI
does.
New LOCALAI_MODEL_LOAD_WAIT (default 60s) bounds the CALLER, never the
load — the job keeps running either way. On expiry the request gets 503
with Retry-After and a structured body naming the model, the node, the
phase, byte progress and an ETA. The `error` envelope keeps OpenAI
clients working; `loading` is additive so they ignore it.
The ETA comes from the job's own observed rate and is omitted rather than
guessed until enough bytes have moved for that rate to mean anything: a
confidently wrong ETA on a twenty-minute wait is worse than none.
Retry-After is that ETA when known, clamped to [5s, 300s], and the wait
budget otherwise.
LOCALAI_MODEL_LOAD_WAIT=0 waits unbounded, for deployments with no proxy
in front. Zero in the config struct still means "unset, use the default",
so the CLI records the operator's zero as ModelLoadWaitUnbounded rather
than losing the distinction.
The distributed branch of ModelLoader.loadModel wrapped the router's
error with %s, which flattened it to a string. Use %w: the typed error is
what the HTTP layer keys the 503 off.
Assisted-by: Claude Opus 5 [claude-code]
* feat(api): add GET /api/models/{id}/load-status
A client that receives 503 while a model stages onto a worker needs
somewhere to poll. This returns the same `loading` object the 503 carries
— phase, node, byte progress and ETA — or 404 when no load is running.
Read-only and observability-shaped, so it is deliberately neither
admin-gated nor feature-gated: it explains a 503 the caller just
received, and hiding that behind a per-modality feature would make the
explanation for a failed image request depend on chat permissions. It
also gets no MCP tool, since there is nothing here an admin would manage
conversationally.
Registered on the surfaces from .agents/api-endpoints-and-auth.md: the
swagger block (existing `models` tag, so /api/instructions needs no new
area), the endpoint discovery maps in RegisterLocalAIRoutes, regenerated
swagger, and the distributed-mode docs page. No FLAG_* usecase is
involved, so capabilities.js is unchanged.
Assisted-by: Claude Opus 5 [claude-code]
* feat(ui): show cold-load progress in Chat and retry when the model is ready
A chat request for a model that is still staging onto a worker now gets a
503 carrying live progress instead of an error. Render it: the composer
shows the phase (installing / staging / loading), the node, the percent
and the ETA, then polls load-status and re-sends the request the moment
the model is ready.
Reuses the staging progress idiom the page already had rather than
inventing a second one — the two sources are folded into one
loadProgress, with the load job winning because it is authoritative
across frontend replicas and knows the phase, where the staging operation
only knows about a byte transfer this replica happens to be performing.
Waiting is bounded (three send attempts, ~30 min of polling each), so a
load that never finishes still surfaces as an error rather than as a
spinner nobody questions. An aborted generation stops the polling too.
Assisted-by: Claude Opus 5 [claude-code]
* fix(distributed): check warm-path cleanup errors
The router moved legacy cleanup calls onto newly linted lines. Report
cleanup failures while preserving the fallback to a cold load.
Assisted-by: Codex:gpt-5 [golangci-lint]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
|
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
0c9d4bf9cc |
fix(vllm-cpp): build every CUDA architecture the platform can host (#11512)
The vllm-cpp CUDA images were built for Blackwell only: 120a;121a on amd64 and 121a alone on arm64. vllm.cpp's own release archive builds ten architectures, so LocalAI shipped one or two of them. The failure mode is the problem. An unlisted card is not slower, it dies at the first request with "no kernel image is available for execution on the device", long after `backends install` reported success. That covers A100, A10/3090, L4/4090/RTX 6000 Ada, H100/H200, B200, B300, Jetson Orin and Jetson Thor, and it is how a Jetson Thor node was found serving nothing at all. amd64 now builds 80;86;89;90a;100a;103a;120a;121a and arm64 builds 87;90a;100a;110;121a, split by where the silicon exists: Jetson is arm64-only, desktop 120a is amd64-only, and 90a/100a are on both because of GH200/GB200. Triton-AOT stays ON for both, which the old comment said was impossible. It is not, at the version we pin: only maintainer REGEN needs a single arch, while the BUILDER path embeds every vendored cubin tree and selects by exact SM, so 87/103a/110/120a take the portable CUDA kernels and can never load a neighbouring cubin. Upstream ships its ten-SM archive that way. The CUDA 13 guard now covers both branches rather than amd64 alone. arm64 needs compute_121a just as much, and CI already builds it with 13. Cost is smaller than the arch count suggests, because gencode is per-source: fp4-mma still resolves to 120a;121a, and the CUTLASS scaled-mm kernels to one arch each, so the added architectures do not multiply the expensive translation units. Verified: flag generation checked for both branches, CUDA 12 still refused, CPU build untouched; both arch lists expanded through vllm.cpp's own vt_cuda_gencode_options and per-feature arch gating, and all six vendored Triton trees confirmed intact, at the exact pinned commit. A real compile is CI-only: there is no CUDA toolchain on the dev box. Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Bash] [Edit] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
5c63969760 |
fix: Show MCP connection errors in the UI (#11495)
* fix(mcp): surface configured server failures Keep model-configured MCP servers visible when discovery or connection setup fails, propagate status through distributed discovery, and let the Chat UI show actionable errors while retrying unavailable servers. Add model-editor metadata for remote and stdio configuration and document the expected format, deployment networking boundary, and alternate MCP scopes. Assisted-by: Codex:gpt-5 Ordino golangci-lint Signed-off-by: Richard Palethorpe <io@richiejp.com> * build(compose): match CUDA development image Configure the API image with the cublas, CUDA 13, auth-tagged build settings used by the local development Makefile invocation, including the 24-way Docker build. Assisted-by: Codex:gpt-5 Ordino Signed-off-by: Richard Palethorpe <io@richiejp.com> * revert: keep host build settings out of compose The CUDA development deployment is managed from ~/docker/localai, not the repository example Compose file. Restore the generic example and keep machine-specific build settings in the host deployment. Assisted-by: Codex:gpt-5 Ordino Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(docker): exclude local agent artifacts Keep Claude worktrees and locally installed verification tools out of the Docker build context. These host-only directories added roughly 1.9 GB to every root image build. Assisted-by: Codex:gpt-5 Ordino Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
7a7fb00730 |
feat(ui): rebuild the import form on the restyled design language (#11461)
The import page took the new palette in #11305 but kept its old layout, so it stayed a 760px column with the primary action detached from the form it submits. Two of the problems were outright bugs. The Import button carried no className at all, so the page's single most important control fell through to the user-agent button: system chrome, wrong radius, no design-system focus ring. The YAML button carried `fas fa-save fa-upload`, which sets Font Awesome as the button's own font family (its label text inherits it) and points two glyph classes at one ::before. On the layout: `page--narrow` is documented for "forms / single-record edit views", and in Advanced mode this page held a URI field, a six-section format guide, ten modality chips, nine preference fields, a key-value repeater and a YAML editor at `calc(100vh - 400px)`. The width was the symptom; one column was the disease. - `page--medium` with a work column and a format reference beside it. The reference answers the only question a first-time admin has and used to sit behind a chevron, closed by default. Below 1024px it becomes a disclosure rather than disappearing. - The source field is the hero: monospace, because it holds something you paste, and it carries its own Import button. That removes the hidden aria-hidden submit button that existed only because the real action sat outside the form. - Simple and Advanced are gone. They were ~80% the same surface, and the overlap cost a mode switch, a localStorage key and a three-button Keep/Discard/Cancel dialog whose only job was protecting state that switching modes would hide. One form with a collapsible options panel hides nothing, so none of it is needed. What genuinely differs is the kind of input, which is now the two tabs: a source, or YAML. - The size/VRAM estimate reports under the field that produced it instead of as a banner above the page header, and an import in flight gets the progress, phase and byte counts the poller already returned and the old status card threw away. - ModalityChips resolves its labels through the same `modality.*` keys as the dropdown it filters. It hardcoded English shorthand, so one modality carried two names on one screen ("Speech" on the chip, "Speech recognition" on the group it scrolled to) and seven locales had neither. Its inline styles and its pill radius move onto the design system. - Three inline styles go, including both conditional-padding hacks; the only one left is the progress bar's runtime width. Baseline 538 -> 535. Docs updated in the same change: the WebUI section described a Simple and an Advanced mode and told the reader to "Toggle to Advanced Mode". e2e: 426 passed. The mode-switch suite is replaced by one covering the tabs and the disclosure, and a new layout suite pins the width, the styled primary action, the absence of an icon-font button, the reference column at both widths, and the estimate's position. Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] [Playwright] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
f2cdc06781 |
fix(model): surface backend startup exits (#11447)
* fix(model): surface backend startup exits Preserve the local backend process exit code and bounded stderr diagnostic when the process dies before its gRPC service becomes ready. Fixes #9050 Assisted-by: Codex:gpt-5 * fix(model): satisfy startup diagnostic checks Assisted-by: Codex:gpt-5.6 [Codex] --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
16193e1982 |
feat(gallery): add Higgs Audio v3 TTS (#11456)
Expose the existing audio.cpp Higgs support as an installable Q8 gallery model and document voice cloning and licensing constraints. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
f7db51bdf5 |
feat(realtime): add shared WebRTC UDP port (#11436)
* feat(realtime): add shared WebRTC UDP port Allow realtime WebRTC peer connections to reuse one configurable UDP mux, and surface listener bind failures through signaling. Assisted-by: Codex:gpt-5 * test(realtime): keep UDP mux alive during bind check The returned SettingEngine owns the UDP listener. Retain it through the duplicate-bind assertion so macOS cannot finalize the listener early and make the exclusivity check spuriously pass. Assisted-by: Codex:gpt-5 [systematic-debugging] * test(realtime): use IPv4 for UDP mux checks Match the socket family used by the WebRTC UDP mux so macOS does not allocate an IPv6 probe that can coexist with the IPv4 listener.\n\nAssisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
7b9167eaad |
feat(llama-cpp): serve Qwen3-TTS through the llama.cpp backend (#11392)
* fix(config): do not read a TTS speaker-encoder mmproj as vision support Qwen3-TTS on llama-cpp ships an mmproj holding the speaker encoder and code predictor. VisionSupported() treated any non-empty MMProj as proof of image input, so every such model would be advertised as vision-capable. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): add TTS request option parsing helper Validates text and speaker reference presence and strictly parses the top_k / top_p per-request params, in a header with no llama.cpp or gRPC dependencies so the standalone C++ unit test gate picks it up. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): range-check the TTS top_k and top_p request params Format validation alone let NaN, infinity and out-of-range values through. The consumer copies both values into the audio generation input unconditionally and only guards its separate sampler assignment with "> 0", a test NaN also fails, so a NaN reached llama.cpp with the guard never firing. top_k must now be >= 0 and top_p must fall within 0.0 to 1.0 inclusive, with the bound written as a negated in-range test so NaN is rejected rather than silently accepted. Also cover the two checks the suite could not previously kill: the whole-string check in the float parser and the int32 range check. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): bump pin to f9e832c10 and carry the TTS server task Picks up ggml-org/llama.cpp#26254 (Qwen3-TTS via mtmd) and #26536 (the short-input audio chunk fix). Adds 0002-add-server-task-type-tts.patch, the server-side half of the still-draft #26603, so TTS runs through the slot scheduler instead of racing it. Remove that patch when #26603 merges. The patch is rebased on top of the score patch: its tokenize-switch hunk collided with the SERVER_TASK_TYPE_SCORE case, and its lone SRV_WRN call passes no variadic argument, which the macro cannot expand. The score patch itself needed no refresh. Also fixes fallout from the bump in grpc-server.cpp: upstream dropped the per-slot n_ctx argument from server_schema::eval_llama_cmpl_schema. Only the schema branch loses it, since forks predating the server-schema split still expect the old argument list. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): implement the TTS and TTSStream RPCs Both were declared in backend.proto but unimplemented. They now submit a SERVER_TASK_TYPE_TTS task and drain the response reader, the same shape PredictStream uses. The streaming path emits a leading sample_rate message and then raw PCM, because ModelTTSStream builds the WAV header itself; the non-streaming path emits a complete WAV to the requested dst. The streamed samples are converted from the pipeline's float32 to signed 16-bit first. MTMD_HELPER_GEN_AUDIO_OUTTYPE_PCM hands back floats, while the header ModelTTSStream writes announces 16-bit samples, so shipping the floats verbatim would decode as noise. prepare.sh and CMakeLists.txt now stage tts_request_options.h alongside the other grpc-server helpers, and register its standalone test with ctest the way passthrough_options_test is registered. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): mask non-codec tokens for Qwen3-TTS generation The Qwen3-TTS gen-audio pipeline maps a sampled backbone token to a codebook row with an unchecked subtraction, in mtmd-helper-gen.cpp: inp.code0 = sampled - codec_0; For ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF the vocab is 155008 tokens, <|codec_0|> is 151936 and the codec codes end at 153983. The model's own tokenizer.ggml.suppress_tokens holds 1023 ids covering 153984..155007, every special above the codec range except <|codec_eos_token|> (154086) which stays reachable as the stop token. Nothing masks the text range 0..151935, so the backbone can sample a text token at any step, the subtraction goes negative, and ggml_compute_forward_get_rows aborts the whole backend process on GGML_ASSERT(i01 >= 0 && i01 < ne01). Complete the mask upstream started: bias every token below <|codec_0|> to -INFINITY for TTS tasks so only codec codes and the codec EOS remain reachable. The biases are appended to task.params.sampling.logit_bias, which common_sampler_init already merges with the model's suppress tokens into one llama_sampler_init_logit_bias, so no sampler is added to the chain. Measured cost is 0.082 ms per sampled token and 1.16 MB, set against a forward pass in the multi-millisecond range. It lands in launch_slot_with_task rather than in a route handler so that llama.cpp's own POST /tts and LocalAI's TTS/TTSStream RPCs are both covered, and <|codec_0|> is resolved from the vocab rather than hardcoded so a model without it is left alone. This is reproducible with upstream's own llama-tts and no LocalAI code loaded, aborting at frame 55 on Q4_K_M and frame 71 on Q8_0, so it is neither a quantization artifact nor an artifact of the gRPC adapter. Two further defects in the same draft pipeline still prevent end-to-end audio; they are independent of this one and are recorded in the task report for an upstream bug report. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): bump pin to 9de0fcf2b and drop the TTS codec mask Upstream fixed the Qwen3-TTS abort in ggml-org/llama.cpp c8e03ce81 ("mtmd/ggml: add ggml_build_forward_order", #26649), landed one hour after the previous pin. ggml_build_forward_expand marks a tensor and all its ancestors for compute, so using it as a pure ordering hint defeated ggml_build_forward_select and made GEN_WAV calls execute the GEN_CODE branch against a stale inp_code0, hitting the get_rows bound assert in ggml_compute_forward_get_rows. That single defect accounts for every abort seen on this model, so 0003-mask-non-codec-tokens-for-tts.patch is removed rather than rebased. The mask changed the observed behavior, but it was perturbing a graph ordering bug rather than fixing a sampling one: at the new pin the whole path works without it. Keeping it would have meant carrying a 152k-entry logit bias, and rebasing it on every pin bump, for no benefit. Verified at 9de0fcf2b with only 0001 and 0002 applied, which both apply clean with no fuzz and needed no rebase: non-streaming HTTP 200, 410924 bytes, 8.56 s RIFF (little-endian) data, WAVE audio, Microsoft PCM, 16 bit, mono 24000 Hz streaming HTTP 200, 560684 bytes, 11.68 s, exactly one RIFF at byte 0, same format, which also exercises the float32-to-s16 conversion at runtime for the first time Pristine unpatched llama-tts at the same pin now also completes, 130 frames to a valid WAV, where it aborted at frame 55 before. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): clear the TTS slot sequence between requests Only the first TTS request in a backend process succeeded. Every later one failed instantly, in about 0.13 s, with "TTS prompt processing failed" from step_prompt, regardless of streaming or non-streaming and regardless of the text. With LOCALAI_SINGLE_ACTIVE_BACKEND=true the process is kept alive between requests, so a deployment would have served exactly one utterance per backend start. The cause is missing KV hygiene, not anything in the gRPC adapter. TTS slots never enter the shared batch: pre_decode() returns early for them and process_tts_slots() drives them instead, so they skip the prompt-cache bookkeeping that clears a slot's sequence between requests. Nothing in the gen-audio path makes up for it: mtmd_helper_gen_audio_reset only clears host-side buffers, and the pipeline always decodes from position 0 into the sequence identified by slot.id. So the second task on a slot writes positions 0..N over the first task's tokens and llama_decode fails. Fix is one call to slot.prompt_clear(), the same helper the normal path uses, in the SERVER_TASK_TYPE_TTS branch of launch_slot_with_task before set_input. It goes into 0002 rather than a new patch file because it is a defect in the code that patch introduces, and the header now records it as ours so we know whether it still needs carrying if #26603 merges without it. Verified in one backend process, different text on every request: three consecutive non-streaming requests, three consecutive streaming requests, and an interleaved non-streaming, streaming, non-streaming, streaming run. All ten returned HTTP 200 with RIFF ... WAVE audio, Microsoft PCM, 16 bit, mono 24000 Hz, the streamed ones carrying exactly one RIFF header at byte 0, and every output measured as real speech rather than silence or a truncated fragment. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): expose max_frames for TTS requests The Qwen3-TTS backbone does not always emit <|codec_eos_token|>, and when it does not, generation runs to upstream's 512-frame n_predict default. At the model's 12.5 Hz frame rate that is 40.96 s of audio, which a short input can trigger: one request in this session produced 40.96 s for a ten-word sentence. prepareTTSTask hardcoded n_predict to -1, so callers had no way to bound it. Add a max_frames key alongside top_k and top_p, parsed with the same strict whole-string parsing so a typo is an error rather than a silently truncated value, and rejected with a field-naming message when negative. 0 keeps the existing sentinel convention and means unset, so a request that omits it behaves exactly as before. Named max_frames rather than n_predict because frames are what the parameter means at a TTS endpoint: one frame is 0.08 s of audio. The 512-frame default is deliberately unchanged. Lowering it would truncate legitimately long inputs, which is a worse failure than an occasionally overlong one. Verified end to end on one text of thirty words: max_frames=25 HTTP 200, 96044 bytes, 2.00 s, exactly 25 frames max_frames=50 HTTP 200, 192044 bytes, 4.00 s, exactly 50 frames no max_frames HTTP 200, 572204 bytes, 11.92 s, stopped at its own codec EOS after 149 frames, unchanged behavior max_frames=-1 InvalidArgument "max_frames must be >= 0, got \"-1\"" max_frames=many InvalidArgument "max_frames must be an integer, got \"many\"" Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): send the TTS sample rate up front, and tidy three review items Four items from the Task 4 review. Streaming first-byte latency. TTSStream sent the sample-rate reply only once the first audio result arrived, and a chunk needs a whole 72-frame window, roughly 5.8 s of audio and far longer in wall time on CPU. The Go side blocks on that reply before it can emit the WAV header, so a streaming client sat at zero bytes for the whole stretch. The rate is a property of the loaded model and is available synchronously from mtmd_gen_audio_get_info, so it now goes out immediately after post_task and the rate_sent bookkeeping is gone. Measured on a warm model, first byte drops from 30.48 s to 0.014 s, and the output is still a valid WAV with exactly one RIFF header at byte 0. Unchecked close. The non-streaming path ignored ofstream::close(), so a failure that only surfaces on flush was reported as success while leaving a truncated file at dst. It now returns INTERNAL like the other write failures. Wrong comment on set_lang. gen_audio::inp::get() already maps a stored blank to nullptr, so our guard is behavior-preserving, not behavior-fixing. The comment claimed otherwise; the code was right. Repetition penalty. penalty_last_n = -1 is inert at this pin, because llama_sampler_init_penalties clamps it with std::max(penalty_last_n, 0) and then builds a disabled sampler, so the 1.05 penalty never applies. Upstream's README attributes looping to a missing repeat_penalty, so it was worth testing as a root-cause fix for the model running to the frame cap. Dropping the line lets the sampling default of 64 apply, which was confirmed in the sampler chain trace as penalty_last_n = 64 with repeat_penalty = 1.050. Over 15 uncapped short requests each way it did not help: 0 of 15 ran to the cap with the penalty inert, 1 of 15 with it active. Both lines are therefore kept for parity with upstream's draft, and a comment now records that the pair is inert and why, so the next reader does not believe a penalty is applied. max_frames remains the way to bound output. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * build(llama-cpp): let unpatched forks opt out of the TTS task turboquant and bonsai copy grpc-server.cpp into llama.cpp forks that do not carry our patches. disable-tts-task.sh injects the same kind of preprocessor switch disable-score-task.sh already uses, so those builds answer UNIMPLEMENTED rather than failing to compile. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(config): keep a TTS speaker-encoder projector out of vision detection Task 1 exempted a declared-TTS model's mmproj from VisionSupported, but the first real gallery entry with an mmproj still came back vision-capable through two paths the earlier fix did not close. GuessUsecases has no FLAG_VISION branch, so it falls through to true for any chat-ish model. That is not just a wrong answer at the call site: syncKnownUsecasesFromString rewrites KnownUsecaseStrings from HasUsecases, and the loader calls it more than once per config file, so the guessed FLAG_VISION is written out and parsed back into KnownUsecases as if the operator had declared it. Give GuessUsecases a FLAG_VISION branch that defers to the same explicit signals VisionSupported uses. Second, llama.cpp builds an mtmd context for the speaker-encoder projector and reports its media marker on the first chat probe, which resurrected vision after the model had been used once. Apply the same declared-TTS exemption to MediaMarker that the mmproj check already had. Verified against the qwen3-tts-llamacpp-q4 gallery entry: no vision capability and no image input modality, before load, after a TTS request, and after a chat probe. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add Qwen3-TTS entries for the llama-cpp backend Two entries over upstream's own GGUF conversion, Q8_0 and Q4_K_M, each pairing a backbone with the Q8_0 projector. Named to sit alongside the existing qwen3-tts-cpp entries rather than replace them. Also tags the llama-cpp backend text-to-speech / TTS so the backend browser surfaces the capability. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: cover Qwen3-TTS on the llama-cpp backend Adds the gallery variants, the two-file mmproj configuration, the required voice reference, and the language and sampling knobs. Also corrects the streaming-support list, which named only voxcpm. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(config): register llama-cpp as a TTS and voice-cloning backend The branch taught the llama-cpp backend to serve Qwen3-TTS and shipped two gallery entries for it, but never told the capability table. llama-cpp still declared only the text RPCs and usecases, so: - VoiceCloningForModel returned nil at the capability check, before it ever reached the model's own tts.voice_cloning override, and /tts answered 400 "selected model does not support reference-audio voice cloning" for any localai://voice-profiles/... voice. No model YAML could opt back in. - GET /api/backends/usecases did not list tts for llama-cpp, so the gallery greyed out the TTS filter for the entries this branch adds. - The React TTS page saw voice_cloning: null and kept both models out of the Voice Library. Add the TTS RPCs and usecase, and the reference-audio contract. The contract needs narrowing, because the per-backend switch in VoiceCloningForModel ends in a permissive default: an unnarrowed entry would have advertised reference-audio cloning on every GGUF chat model in the gallery. Narrow on the declared TTS usecase rather than the model name. The TTS checkpoints are the only llama-cpp models carrying known_usecases: [tts]; name matching would have to guess at third-party repacks, and "base", the substring the neighbouring Qwen and vLLM cases key on, is a routine word in text-model names. The check reads the declared bit directly instead of going through HasUsecases, which falls through to GuessUsecases and would hand the decision to a heuristic that never had a llama.cpp TTS model in mind. DefaultUsecases stays [chat]: a bare GGUF served by llama.cpp is a chat model, and both the gallery filter and the importer read that field. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): declare what nemotron-3-nano-omni actually accepts The entry is backend: vllm-omni with known_usecases: [chat, completion], no mmproj and no media marker, so it used to report vision only through the blanket GuessUsecases fallthrough that the vision branch in this branch removed. Nemotron 3 Nano Omni is a multimodal understanding model: image, video and audio in, text out. Declaring that is what the sibling vllm-omni-qwen3-omni-30b already does. known_usecases gains vision only. FLAG_VIDEO is video GENERATION, an output modality, and this model generates none; video and audio input belong in known_input_modalities, which is where AudioInputSupported and VideoInputSupported read them from. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(importers): import a Qwen3-TTS GGUF repo as TTS, not chat The llama-cpp importer hardcodes known_usecases: [chat] and assigns any mmproj-matching file as a vision projector, so ggml-org/Qwen3-TTS-12Hz-1.7B- Base-GGUF imported as a chat model with vision. Both fields were wrong, and the model was unreachable from /tts and from the Voice Library. Filenames cannot fix this. A Qwen3-TTS repo has the exact shape of a vision repo, one backbone GGUF plus one mmproj-*.gguf, so the projector's own header is the only honest signal: mtmd writes clip.has_gen_audio_encoder for the projectors it can drive as a speech pipeline and refuses to build one without it. Probe the selected mmproj for that flag, reusing the range-fetch the MTP detection already does, and declare tts when it is set. The mmproj assignment then stops reading as vision on its own, since a declared-TTS model already exempts its projector from vision detection. The probe is best-effort like the MTP one: a network blip leaves the chat default in place rather than failing the import. Verified against the real artifacts on disk: the Qwen3-TTS projector reports gen-audio, its backbone does not. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): stop non-TTS models crashing on the new pin Two regressions, both hit every ordinary llama-cpp model and neither was caught locally because every test on this branch loaded a TTS model. The first is a null dereference. server_slot::tts_ctx::reset() called mtmd_helper_gen_audio_reset() unconditionally, but the gen-audio pipeline is only allocated for models carrying a gen-audio mmproj, and upstream's implementation reads ctx->pipeline before null-checking anything. Since server_slot::reset() runs during slot initialization for every model, any non-TTS model segfaulted the backend the moment it loaded. Guard the call on the is_supported() predicate already defined beside it, and keep the plain field resets unconditional. The second is unrelated to TTS and came in with the pin bump. PredictOptions.Penalty is a bare proto float, so a caller that names no repetition penalty sends 0 rather than omitting the field. Since 9de0fcf2b, common_sampler_init() rejects a non-positive penalty_repeat outright because it would divide logits by zero, turning every such request into "Failed to initialize samplers". Treat 0 as unset and leave llama.cpp's own neutral default in place. Verified with the same suite CI runs, which is what caught both: tests/e2e-backends passes 6 of 6 including the load and predict specs that were red. Qwen3-TTS still synthesises on both paths, 24 kHz mono 16-bit WAV with exactly one RIFF header on the streamed output. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
daa8d2adbd |
fix(gallery): identify invalid preload JSON (#11434)
Wrap PRELOAD_MODELS decoding failures with the setting name and expected top-level shape so startup errors point directly to the invalid configuration. Document the required array format and cover scalar and empty-array inputs. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
06ff56e674 |
feat(pii): restore request-scoped pseudonyms (#11272)
* feat(pii): restore request-scoped pseudonyms Replace masked request values with unique per-request tokens when response restoration is enabled, then restore them across JSON and SSE write boundaries. Document the opt-in model setting and expose it in config metadata.\n\nAssisted-by: Codex:gpt-5 * fix(pii): wrap reversible redaction tokens Use configurable token delimiters to avoid restoring ordinary model text that happens to match an internal identifier. Rename the option and document the confidentiality tradeoff. Assisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |