mirror of
https://github.com/mudler/LocalAI.git
synced 2026-09-13 14:56:11 -04:00
0dc6ebd5259133f9b594a67cd5d8884483b0a779
218
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Ettore Di Giacinto
|
0683fb1579 |
test(distributed): prove the worker tunnel end to end, under real inference
Everything this phase built was proven by unit and integration specs. This is the first run of it against the real binaries: a frontend replica per process, a worker that binds nothing routable, real inference over the result. Four scenarios, each with the question "what would make this pass if the tunnel were doing nothing" answered rather than left open. A worker with no advertised address is reached through its tunnel. The roster is asserted to report it advertising nothing, so there is no address a frontend could have dialled instead, and node_connections is asserted to name the replica that serves the request. A request landing on the replica that does NOT own the worker is relayed to the one that does. With N replicas behind round robin that is (N-1)/N of production traffic, so it gets the FIRST request for its model: the backend install, the file staging on the http tag, and the gRPC load and predict all cross the relay. Which replica owns the tunnel is read from the ownership table through the production Owner query and mapped to a frontend index through the address the harness pins per replica; the non-owner is derived from that reading and asserted to be a non-owner immediately before the request, rather than assumed from the harness default. Sending the same request to the owner reddens it. Killing the owning replica re-homes the worker onto the survivor. The worker dials a balancer rather than a replica, because LOCALAI_REGISTER_TO is resolved once at boot and is the tunnel endpoint as well as the registration one: aimed at a single replica, a worker has nowhere to reconnect to when that replica dies, and the re-home cannot happen at all. Removing the kill reddens it. And the negative control for the whole suite, which is why the other three mean anything. Frontend and worker share a host here, so every backend port the frontend names in a stream target is one it could have dialled directly; if it did, the first three would pass with the tunnel inert. LOCALAI_WORKER_TUNNEL is no longer usable for this, because it is a fatal startup error and a worker that never started says nothing about a worker reachable some other way. The balancer answers the tunnel connect path itself instead, leaving a worker that registers, heartbeats, reports healthy and holds no tunnel. It is asserted to have dialled and been refused, asserted to be held by nobody, and then asserted unreachable with the refusal naming the missing route. Then the block is lifted, nothing else changes, and the same request succeeds: that is what attributes the refusal to the tunnel rather than to any of the ordinary reasons an e2e inference fails. The fifth spec measures the head-of-line blocking this phase deferred three times. 128 MiB crosses the session while a warm model is probed back to back, direct and relayed. Median latency is unchanged, the worst probe is about 3x the baseline median and about a seventeenth of the transfer window, and the transfer runs at 415-490 MB/s direct and 222-268 MB/s relayed. A session that head-of-line blocked would park a probe for the length of the window. Leave the yamux windows untuned; and note this is loopback, so it says the multiplexing does not serialise and says nothing about a link with a bandwidth-delay product. The load spec is measured against a control that the first version did not have. It passed with the bulk artifact cut to 4 KiB, because the window it read probes against was mostly cold-load overhead: it would have reported a clean bill on a session carrying no large message. The same cold load now runs twice, once empty and once bulk, and the difference between the windows is asserted to be real before any latency is read from it. Two defects on the base commit came out of this. cluster_peerlink_test.go has been red since the relay landed, deterministically, in isolation and in the suite. It asserted that an accepted peer stream is refused at once, on the premise that phase 1 installs no relay. The relay correctly waits fifteen seconds for a frame naming the worker, and the spec's budget was five. It now writes a relay request for a node no replica holds and asserts the refusal is ErrNotOwner and specifically not ErrNoConnection, which is a stronger spec than the one it replaces and the only thing in the e2e suite that exercises the relay's refusal path. The harness handed a worker's own HTTP port to a backend process. It took two ports from freeport and used one as the gRPC base and the other for the file transfer server; freeport returns adjacent ports often, and the backend allocator hands out base, base+1, base+2, so the second backend started on a worker was regularly given the HTTP server's port and died with EADDRINUSE. No spec had started two backends on one worker before, so it had never fired; the load spec starts five and it failed about one run in three. Each worker now reserves a contiguous bind-probed block laid out the way production lays it out, below the kernel's ephemeral range, with LOCALAI_GRPC_MAX_PORT bounding the allocator to it. The underlying production defect is not fixed here and is recorded in the report: allocatePort never checks that a port is free, and its default range overlaps the ephemeral range on every Linux box. Constraint 6, whether distributed mode should now refuse to start without an advertised address, is DEFERRED, and the comment and the docs that described the cost were understating it. A replica with no advertised address writes no instances row, and Owner joins a connection against a live instance, so a worker whose tunnel lands there is unroutable from every OTHER replica while being registered and healthy. Refusing to start would still be wrong, because the deployments it would break are single-host ones with no peers to be unreachable by, and telling those apart at startup is a design with its own specs. Both places now say what actually happens. Suite wall clock 592s for 15 specs, up from 502s for 10 of which 2 were red. The CI budget of 20 minutes does not move. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
3338d7bc56 |
fix(distributed): refuse a worker that cannot tunnel, and say why it was refused
Review round 1 on the change that stopped workers listening. One blocking item
and seven notes.
LOCALAI_WORKER_TUNNEL=false was the blocking one, and the ruling was to make it
fatal rather than to correct the comment that still promised it fell back to the
advertised address. There is no fallback left: a worker on this branch
advertises nothing and binds only loopback, so turning the tunnel off leaves it
reachable by nothing while it registers, heartbeats and reports healthy, and the
scheduler keeps placing models on it. That is the worst available failure shape,
so a new Config.validateStartup refuses it before prefetch, registration and
NATS, while the worker is still invisible to the cluster. It absorbs the
pre-existing empty-registration-token check, which had the same shape and no
spec. The flag is kept rather than deleted so an operator who set it is told the
promise is gone instead of having the setting ignored, and the guard around
StartTunnel is removed, because a branch nothing can take reads as a supported
no-tunnel mode that does not exist.
The justification for erroring on an install that names no address was wrong,
and the review is right that this is the dangerous form of overclaiming, because
the conclusion holds and the mechanism does not. It said the resulting empty
target would be refused as an invalid stream and that the refusal would read as
the worker answering about its backend. Nothing in this repo branches on
cluster.ErrNoRoute, and nodes.unroutable treats any recorded dial error as
unroutable, so that refusal reaches every reap guard as ProbeUnknown and deletes
nothing. The site now stands on what holds, that an install naming no port
produced nothing routable and the failure belongs to the install rather than to
a later probe, and records the retracted claim so nobody re-derives it. This
retracts the same paragraph in the body of
|
||
|
Ettore Di Giacinto
|
1cf847f29e |
feat(distributed): stop workers listening, and stop them advertising
A worker now opens no listener on a routable interface and states no endpoint at registration. Backend processes and the file-transfer server bind loopback, and the frontend reaches both through the tunnel the worker dials. The bind address is built from loopbackHost, the same constant the tunnel's grpc tag dials, so "the worker binds where its tunnel dials" is one fact in one place rather than two literals that can drift. All three advertisement sites are closed, not one: the registration body, RegisterNodeRequest, and the per-backend address in the install reply. That third one was hiding a live bug. stopModelExact refuses a stop whose ExpectedAddress does not match what the worker recorded for the process. The worker recorded 127.0.0.1:port; handleBackendInstall reported advertiseHost:port; the router stored the reported one and sent it straight back. On any worker whose advertise host was not 127.0.0.1, every acknowledged model stop failed with an address mismatch. Nothing caught it because the e2e harness set LOCALAI_ADVERTISE_ADDR=127.0.0.1, which made the rewrite a no-op. Removing the rewrite makes the two strings the same by construction. The brief was wrong about two of the four functions it called dead. effectiveBasePort is the base of the backend port allocator and resolveHTTPAddr is the file server's bind address; deleting them would have deleted the port allocator and the file server. Only the two advertise* helpers were dead, and addr_test.go is rewritten rather than deleted, because the port arithmetic it pinned still needs pinning. NodeModel.Address survives with a narrowed meaning and is renamed WorkerLocalAddress, along with the install reply field that feeds it. The frontend still has to say WHICH backend process on a worker it means, and the port in this string is how it says it: it travels as a stream target and the worker dials its own loopback. The gorm column and the json key stay "address", so neither a migration nor an API break rides along. Every fall-back to the node's address is gone. installBackendOnNode now errors when a worker reports success without naming one, because substituting the now-always-empty node address would name an empty target, and the worker refuses that as an invalid stream, which is classified as the worker answering about its backend. That is the "a present worker reads as something it is not" class this phase forbids. DistributedModelStore.Range had the same shape and was already wrong: it built each remote model's client from the node's base gRPC port, never the port a backend process listens on, so Free and Status went to the wrong place. It uses the replica's address now. BackendNode.Address and HTTPAddress are kept but made provably inert: no writer, no reader that acts on them, and Register force-clears both on re-registration so an upgraded worker's stale advertisement does not outlive its own upgrade in the API and the Nodes page. Dropping the columns is a ~90-site edit across the specs, the e2e suite, the MCP dto and the UI; it is recorded as a follow-up rather than folded in here. A persistent tunnel 401 still does not trigger re-registration, and now for a reason rather than a deferral. Register CLEARS the node's replica rows, so re-registering on a 401 would delete a live worker's rows on every retry, and under the name collision that causes the 401 the two workers would take turns doing it forever: a credential failure causing model reclamation. It also cannot fix the named cause, since a collision is indistinguishable from a restart. The 401 log now names both causes and says nothing can reach this worker, which is true only now that it has no listener. The container healthcheck did not break the way the brief expected, since the listener still exists on loopback and the probe runs inside the container. It did have a real #10987 defect that this change makes the common case: it read LOCALAI_SERVE_ADDR only, while effectiveBasePort reads LOCALAI_ADDR first, so a worker on a non-default base port was probed on 50050 and reported unhealthy while working. It follows the same precedence now. Docs, the compose file and the e2e harness are updated in step: no inbound rule or published port is needed for a worker, the two advertise variables are gone, the remaining address variables are read for their port only, the firewall-the-file-transfer-port warning is narrowed to the LOCALAI_HTTP_ADDR opt-out, and the upgrade-order note no longer claims the worker still listens. The Nodes page showed node.address, which is now always blank, so it shows the node id instead. Eight mutations, all red on a named spec, including reverting the loopback bind, re-adding the address to the registration body, restoring both node-address fall-backs, dropping the force-clear, storing the endpoint's address again, and un-fixing the healthcheck. One of them caught a defect in a spec I had just written: it asserted 200 where the endpoint returns 201, which went unnoticed because core/http/endpoints/localai is not on the task's verify list. It is run here. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
b4d8e23abb |
fix(grpc): let the transport answer through the wrappers, not only past gRPC
Re-review round 2. One blocking defect, and it was the concern I filed myself last round and mis-scoped as a future trap. It was live, and it sat on the most destructive reaping path of the five. RouteResult.Client is an InFlightTrackingClient, over a FileStagingClient when a stager is configured. model_router puts that on the cached remote model and pkg/model's checkIsLoaded asks IT whether the transport failed. Both wrappers embed grpc.Backend, which does not declare LastDialError, so the type assertion read nil and the guard added last round fell straight through to the old eviction. That eviction sends backend.stop over NATS to every node holding the model and deletes every replica row, where the other sites delete one. The spec covering it built a bare client by hand, which is why it passed while production did not. This is the third time in this task a correct fix was disarmed one layer out, so the fix is a mechanism rather than two methods. BackendUnwrapper is one line per decorator, LastDialErrorOf walks the chain, and both consumers now call it instead of each keeping its own assertion. One implementation, no per-caller policy to get wrong. Sweeping every type that embeds or holds a grpc.Backend found a third decorator the review had not named, and it is itself a reaping consumer of the same collapsed signal. ConnectionEvictingClient is built for remote models in initializers.go and its evict callback runs ShutdownModel; it fires during INFERENCE rather than on a health check, so a tunnel blip mid-request was enough to stop a model that was loaded and serving. It consults the transport first now. A locally spawned backend has no custom transport, so that path is unchanged byte for byte. Everything else touching a Backend is a consumer rather than a decorator; there is no fourth. The probe cache joiner shape is pinned. It was the right design last round with nothing holding it: the mutation back to a closed-over variable passed all 602 specs in the package. Eight goroutines coalesced on a probe that blocks on a channel now assert every joiner gets the leader's REASON and not just its answer, which is the difference between a leader declining to reap and its seven joiners reaping on the leader's own observation. The LastDialError scope note claimed an exactness it does not have at checkIsLoaded, which reads a shared long-lived client after releasing opMutex. It now says which caller is not exact, why the imprecision is accepted there, and what making it exact would cost. The four-outcome table in the docs still said a worker with no live owner is treated as absent and rescheduled, contradicting the code and the paragraph nine lines below it. None of those outcomes is absence any more, and the table says so, names the fifth, and points at the heartbeat as the thing that does decide presence. Five mutations, each reddening named specs, including the two the reviewer found surviving. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
a8ac2af167 |
fix(cluster): make "no route" a condition of its own, and let it out of the package
Review round 1 on task 6. Five blocking findings, all with the same root: the conditions the dialer kept apart were erased one layer out, because every one of them arrived at core/services/nodes as a gRPC codes.Unavailable, which is also what a backend process that died produces. Four call sites acted on that by deleting a replica row, one of them after a single failed probe. The fifth condition is ErrNoRoute: this replica could not get a request to a worker's backend, and no claim at all about the worker. A worker's presence is its HEARTBEAT, which nodes owns; a route is a separate fact that cluster owns, and the two now differ. They differ in normal operation, not exotically: a worker that has not dialled its tunnel yet after a frontend-first upgrade is unroutable on every request while it heartbeats and serves. Two properties, both mutation-tested. Every failure to resolve or open a route carries ErrNoRoute, so a consumer has one check to make. No failure carries an absence sentinel: routeFailure is the single place that rule lives, and it keeps ErrNoConnection and ErrInstanceNotFound in the message and out of the unwrap chain, the guarantee unreachableError already made for peers. Everything else stays matchable, so ErrNotOwner and ErrPeerUnreachable are unchanged for anyone who can act on them. A worker's own refusal carries no umbrella, because a worker that answers has demonstrated it is there and that is the only real evidence on the path. Crossing the boundary needed a value, not a code. NewClientWithDialer wraps the dialer and records each outcome; LastDialError hands it back behind a narrow interface, and nodes.unroutable turns it into ErrWorkerUnroutable with the cluster sentinels still in the chain. A spec asserts a dial failing with ErrNoRoute plus ErrPeerUnreachable arrives matching all three and matching neither absence sentinel. The sweep found a fourth site the review had not named: pkg/model checkIsLoaded evicts a remote model on a connection error, and a tunnel dial failure is one. Four other reap sites were cleared with reasons - inflight and the worker authoritative pass reap only on semantic answers, scale-down is driven by last_used, abandoned loads decide on the node's heartbeat. Every fixed site also grew the opposite spec, so the new check cannot pass by never reaping. probeCache carries the reason through singleflight rather than a closed-over variable. A variable is only written by the goroutine that runs the probe, so the leader would correctly decline to reap while every joiner reaped on the leader's own observation; a mutation reproduces exactly that. The docs sentence promising LOCALAI_WORKER_TUNNEL=false restores direct dialling is gone. There is no such path, so it said the operator could take a worker dark and call it a rollback. Replaced with the upgrade order that is actually safe. The deadline spec the reviewer found vacuous now waits on the dial context's own Done channel before touching the stream, so the armed deadline has really expired; the mutation that survived for the reviewer reddens it. Nine mutations, each reddening a named spec, including both halves of isAbsenceClaim independently. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
75953d9f63 |
feat(cluster): reach every worker through its tunnel, never its address
The tunnel, the fence, the registry and the relay were all built and none of them carried a byte: every dial from the frontend still went to the address a worker registered. This is where that stops. One WorkerDialer resolves where a worker's tunnel is held, opens a stream on it locally or relays through the owning replica, and hands back a conn past both handshakes; gRPC, the file stager's HTTP client and the log-streaming WebSocket are all pointed at it. A worker's address stops being somewhere to connect to and becomes the name of which backend process a stream is for. It still appears in URLs, logs and errors, because that is what identifies the process; what it no longer decides is where the bytes go. Nothing falls back to dialling it. BackendClientFactory now has exactly one method, NewClientForNode, and returns an error where there is no way to reach the worker. The direct-dial constructor was removed rather than kept beside it, because leaving one on the interface keeps the bypass one word away from every call site that holds an address, which is all of them. The second construction path is closed too. DistributedModelStore built remote models with a nil client, and pkg/model.Model.GRPC then dialled the raw address lazily on first use - reached in production by ShutdownModel's Free and by the backend monitor's Status. Those models now carry the tunnel-backed client, and a model that cannot be given one is logged and not listed. Four conditions stay unmixable, and one path produces absence: the dialer answers ErrNoConnection only where Owner's liveness join did. A peer that will not answer, a stale ownership row, a worker's own refusal and a missing relay path are each reported as themselves. This matters because nodes ACTS on absence, and the collapse would have it reclaim the models of a worker that is connected and busy. That is not hypothetical. Writing the mutation for it exposed the bug in this change's own first draft: probeHealth returned bare false when it could not build a client, and tryWarmPath deletes the replica row on a false probe. A frontend whose dialer broke would have emptied node_models for the whole deployment while every model kept running. probeHealth now returns alive and probed separately, the reconciler gets a ProbeUnknown outcome that neither advances nor clears a failure streak, and the health monitor skips rather than counting a miss. Task 5 left the relay's open timeout at a fixed 15s and said so: no operator has the information to set it, because the number that matters is the original client's remaining budget, which is invisible on the relay side. The dialer has that budget, so it now states it in the relay request frame and the owner takes the smaller of the two. It can only shorten - a patient client must not be able to park a relay goroutine and a stream slot on a worker that stopped accepting. Zero is written as no budget at all, since on the far side the number zero is a caller with nothing left and would refuse healthy traffic. Seven mutations, each reddening a named spec: peer-unreachable as absence; the local-failure guard dropped; max instead of min on the budget; the nil-client model restored; ProbeUnknown falling through to the reaper; OwnerRow instead of Owner; probed collapsed into alive. The first budget spec passed for the wrong reason - a handshake deadline, not the relay - and was replaced by three that each assert one link, including one where the spec plays the owning replica and reads the budget out of the frame instead of inferring it from a clock. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
5108be222d |
fix(worker): spec the tunnel's routing table, which was the SSRF boundary
Review follow-up. One blocking finding and seven others. The blocking one first, and it is this project's recurring shape: the untested path. loopbackService is the function whose comment calls the discarded host "the security property this function exists for", and nothing tested it. The reviewer replaced its body with a dial of whatever the frontend named, no port range, and all 131 specs passed. Every spec installed the permissive test dialler, so the real routing table was exercised nowhere. It now has specs, and the property is stated as reachability rather than as a property of the code: a listener on 127.0.0.2 that only the frontend's target names must NOT be reached. Plus the port-range table, fixedService, loopbackAddr, tunnelEndpoint, and the table itself, which moved out of Run into tunnelServices so it can be built without starting a worker. One spec drives a real stream through that table over the wire, so the routing rules are exercised end to end at least once rather than only in isolation. The reviewer's mutation now reddens ten specs, and six narrower ones redden between two and four each, so no spec is riding on another. The shape changed too, not only the coverage. The dial address is built from a loopbackHost constant and strconv.Itoa of a validated int, so nothing derived from the wire reaches DialContext at all: restoring the hole takes ADDING a data flow, not deleting a check. And a taxonomy fix found while specifying it. A port outside this worker's allocator range was reported as unavailable, which tells a frontend to retry something that can never work. It is a bad request now, and a backend that is merely not listening yet stays unavailable, which is the retryable one. Agent nodes no longer get a tunnel credential. Nothing dials into an agent worker, so a tunnel replaces nothing for it and no client would open one, and the gate is at the mint site rather than in the handler: with no credential minted the hash stays empty and the existing empty-hash refusal covers it, so enforcement is structural. Two comments and one doc paragraph said an anonymous registrant gets a "working" credential. With auto-approve off the node is pending and the credential is inert, which is the distinction this same change argues three files away to justify minting for pending nodes at all. A refusal reason over the frame limit was cut on a byte boundary and could split a rune. It cuts on a rune boundary now, and the code survives truncation, which is what keeps a refusal classifiable. Also: the pending-node spec asserted only that a credential was non-empty, so a credential derived from the shared token passed it; it now pins per-node-ness the way the headline spec does. The tunnel handler's citations into nodes.go were stale before this branch landed, having been written against a file the same commit was editing, and are by function name now. The static-NATS path says plainly that an externally forced rotation locks it out until restart, and where that gets fixed. tunnelproto gained direct specs, including that a read failure is never reported as a refusal. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] |
||
|
Ettore Di Giacinto
|
29a2020f3d |
feat(worker): dial, hold and serve the tunnel, on a credential of its own
The worker end of the tunnel. It dials wss://<register-to>/api/cluster/connect, holds one yamux session as the CLIENT, and serves every stream the frontend opens on it. Nothing dials into the worker, which is the point: no inbound port, no reachable address. Each stream opens with a length-prefixed frame naming a tag and a target, and the worker answers before either side speaks the tunnelled protocol. The reply is sent on every stream, not only on refusal, because the protocols carried here are client-speaks-first and a reply sent only sometimes would arrive interleaved with a response body. Two tags today: grpc reaches a backend process, and only on 127.0.0.1 within this worker's own backend port range, because a tunnel terminates inside the worker and letting the frontend name a host would make every worker a proxy into its own LAN; http reaches the worker's file-transfer server, whose address the frontend is not asked about. An unknown tag, an unreachable local service and an unparseable request are three refusals and stay three on the wire. A frontend gives up on the first and retries the second. Each is answered AND the stream is ended: a worker that says why and leaves the stream open has parked the caller on a request nobody will answer, and a deadline on the far side cannot tell that from a slow worker. The specs assert the stream ends rather than that an error occurred, which is what phase 1 shipped in three places and held in none. Reconnects double from 500ms to a 30s ceiling, each wait drawn between half the interval and all of it, and the interval returns to its floor only after a session that LASTED. Resetting on connect is how a rolling restart, where every dial succeeds and dies moments later, becomes a retry storm against the first replica back up. Nothing is assumed to survive a reconnect: the credential is read at dial time, never captured. And the credential is now real. The tunnel endpoint advertised authenticating a worker against its own secret, but registration stored the hash of the shared registration token, so a leak plus a known node ID still opened a tunnel. Registration now mints a per-node secret, returns the plaintext once as tunnel_token, and stores only its SHA-256 in a new column; the endpoint compares against that and does not fall back to the old one. Rotating on every registration follows from storing only the hash, since a re-registering worker cannot be told the secret it already holds; its live tunnel is unaffected, because the credential is checked when a tunnel is dialled and never again. Unlike the agent API key and the NATS JWT next to it, the credential IS issued to a node awaiting approval: the tunnel route re-reads the node's status on every dial and refuses a pending one, so it is inert until an admin acts, and withholding it would strand every worker that registers exactly once. A node that has not registered since this change cannot tunnel, and the column cannot be back-filled because the plaintext only ever existed in the response that minted it. The boot warning that said tunnels need LOCALAI_REGISTRATION_TOKEN is replaced: it was true while the tunnel authenticated against that token's hash, and says the wrong thing now. What is still true, and is what it warns about instead, is that without one, registration itself is unauthenticated. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
48ece89c63 |
fix(cluster): harden the worker tunnel, and stop starting a database per spec
Review follow-up. Twelve findings, none blocking, grouped here by what they protect. Panics. The handler now recovers between the WebSocket upgrade and the hand-off, the way the peer link next door already did: net/http recovers the panic but leaves the hijacked socket open, so without this a worker keeps a session this replica has no entry for and will never detach. The claim gate in Attach and reclaimOne is now released with defer, so a panic under Claim cannot wedge one node's gate for the life of the process. SetTunnels gained the nil-receiver guard its sibling Stop has. Operability. A deployment with no registration token stores an empty token_hash on every worker, so every tunnel dial 401s forever on a frontend that looks correctly configured. That now warns at startup, logs its own line rather than sharing the "wrong token" one, and is stated in the docs together with the fact that setting the token later needs the workers to register again. Authorization. A node still awaiting admin approval is refused with 403. The rest of /api/node/ gates on nothing, but the two places that hand a node something durable, its API key and its NATS credential, both refuse a pending one, and a tunnel is that kind of grant. Draining and unhealthy nodes keep their tunnels on purpose. Comments that claimed more than the code. The global auth middleware does run on this path and then declines to reject; the future per-node secret only lands without a change here if it lands in TokenHash; the empty-hash guard is defensive rather than deciding; ClusterPathPrefix is no longer only replica-to-replica; the docs no longer say a reaped replica re-claims unconditionally. And the test harness. SetupTestDB started a PostgreSQL container per BeforeEach with a readiness deadline it asserted on, which is one chance per spec to fail one spec inside its setup, anywhere, never twice in the same place: the shape of the flake seen twice here and never reproduced. It now starts one container per process and creates a database per call, which is the pattern tests/e2e already proved. Isolation is unchanged and is now asserted for the first time. All 69 call sites are untouched; the eleven consumer packages run 1404 specs green, and jobs went from 34.3s to 3.3s, agents from 13.8s to 1.9s, cluster from 97.4s to 37.5s. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] |
||
|
Ettore Di Giacinto
|
6e55092a4b |
feat(cluster): open the door a worker dials its tunnel through
A worker needs no inbound port: it dials GET /api/cluster/connect, the connection becomes one multiplexed yamux session, and the frontend opens a stream on it per request. This adds the endpoint that accepts that dial and attaches it to the tunnel registry. The dial is authenticated against the NODE's own stored token hash rather than the deployment's registration token. That is the mechanism, not yet the isolation, since a worker still registers by presenting the shared token; what it rules out is the shortcut of comparing against the configured value, which would have to be unpicked the day workers get their own secrets. Every refusal happens BEFORE the WebSocket upgrade, so a dialer reads an HTTP status rather than a handshake error. The route is registered in every deployment, single-binary ones included, which is what puts it in front of the route-coverage test that holds that rule in place; with no node registry it refuses every dial, and tells a credentialed one the frontend has no cluster rather than that its token is wrong. A lookup that FAILED is answered as a failure. Reporting a database that could not be read as "unauthorized" would send a worker re-registering, throwing away the identity its tunnel and loaded models are keyed by. Wires the tunnel registry in core/application/distributed.go and hands it to the membership loop. Without that call the re-claim after a replica is reaped had no production caller and could never run. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Opus 5 [claude-code] |
||
|
Ettore Di Giacinto
|
aca383d263 |
feat(cluster): give phase 1 a call site, and prove it against real replicas
Tasks 1 to 5 built an instances table, a splice, both halves of a peer link and an epoch fence, and nothing in the tree called any of it: no replica registered, no route was mounted, no sweeper ran. Proving phase 1 end to end therefore had to start by wiring it. A frontend in distributed mode now publishes the address its peers dial, heartbeats it, and sweeps replicas that stopped answering along with the connection rows they owned, in one pass so the two can never disagree about who is alive. It serves the peer link and owns the sessions peers dial in, refusing streams on them until phase 2 installs a relay: a session nobody accepts on does not fail a peer's Open, it hangs it. The address is the one peers use, not the one the process binds, and it is derived from the route to PostgreSQL. That derivation only holds while the database is remote, so LOCALAI_DISTRIBUTED_ADVERTISE_ADDR sets it explicitly and a replica that can determine neither warns and keeps serving rather than failing to start. Three e2e scenarios run against real local-ai processes, real PostgreSQL and real dials: replicas publish addresses that can actually be connected to; a sibling opens a stream over the peer link and is refused without the cluster token; and a killed replica is reported unreachable, never absent, loses the claim it held, and takes no worker with it. Each was verified by mutation: eight injected defects, each failing the scenario that claims to catch it. Also moves RegisterClusterRoutes to core/http/routes beside every other registrar, folds AutoMigrate and the epoch sequence into one cluster.Migrate, and turns the peer route's auth-coverage spec into a real assertion: it drives the request through the actual auth middleware instead of comparing two string constants, which the old spec would have passed even with the exemption deleted. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
893a45141c |
fix(realtime): accept GA WebRTC signaling (#11778)
OpenAI GA clients send multipart or raw SDP requests. They expect a bare SDP answer. LocalAI only accepted its legacy JSON envelope, so signaling failed before media setup. Keep the JSON contract for existing clients. Accept both GA request shapes and choose the matching response format. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
80e3240f2d |
feat(distributed): key scheduling rules by a model alias (#11771)
Node placement and replica rules could only name a model, so an operator who pinned "llama3" to the GPU tier had to rewrite the rule whenever a different model took over that job. An alias already gives a stable name for whichever model serves it, and a rule on that name makes it a deployment slot: repoint the alias and the placement follows. A rule keeps the name the operator chose. Reads resolve that name through the config loader to the model the rule governs, so the reconciler counts, schedules and trims replicas of the target, and the router finds an alias-keyed rule from the target it is already routing. An alias that resolves to nothing governs nothing loadable, so the reconciler skips it and the write paths refuse it. A replica is shared by every name that resolves to it, so only one rule can decide where it runs. The REST and MCP write paths reject a rule whose target another rule already governs. A pair that arrives some other way, such as a seed file or an alias repointed onto a model that already has a rule, resolves in favour of the rule named after the model itself and then the oldest, and the rest are listed as shadowed. The eviction guard is the exception: it matches rules to replicas in raw SQL inside a locking transaction and cannot resolve an alias. It reads a stored target that the reconciler refreshes each tick, and falls back to the rule's own name when that target is empty. Assisted-by: Claude:claude-opus-5 golangci-lint eslint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
29899cd1e0 |
fix(ui): size model fit against the cluster and move node labels into the selector (#11765)
* fix(ui): move node labels into the scheduling selector field The scheduling page kept a node-label browser open above the rules whether or not anyone was writing one, while the field that actually needs labels, the rule's node selector, was two bare text inputs with no hint of what the cluster reports. The browser is gone. The selector's key input now completes against the label keys the cluster uses, and the value input offers only the values that key takes. The roster already loads for the page, so the suggestions cost no request, and a roster that fails to load costs the admin the hints and nothing else. Suggestions stay suggestions: a key no node reports yet still commits as typed, which is how an admin writes a rule before labelling the nodes for it. Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): size model fit against the cluster, not the frontend The models page asked the frontend how much memory a model may occupy. In distributed mode the frontend is usually a GPU-less pod while every model runs on a worker, so a fleet of GPU nodes was told it could only run the smallest CPU build. The variant picker's fits flag and its auto-selection came from the same place, as did the hardware recommendations. The registry now reports the largest single healthy backend node. The largest node, not the fleet total: a model loads into one node, so four 16GB workers are not a home for a 40GB model. An operator-set VRAM budget caps a node's contribution, because the scheduler refuses a load above that ceiling anyway, and a GPU node beats a CPU node holding more system RAM. GET /api/resources and GET /api/models carry this as an additional cluster object. Their aggregate and ram fields keep reporting the frontend's own hardware, which is what the resource monitor shows. Variant selection judges backends against the union of the capabilities present in the cluster, the way backend discovery already did. Every path degrades to the local host: no cluster object in single-node mode, and none when the registry cannot be read, so a hiccup narrows the answer back to single-node behaviour rather than marking the whole catalog too large. The verdicts now name the node they belong to, since a model fits somewhere or nowhere. Assisted-by: Claude:claude-opus-5 golangci-lint eslint playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
3953448f60 |
fix(distributed): resync stored config revisions at startup
The controller pins a model's replicas to a stored revision and rejects any request carrying a different one. Nothing ever re-derived that value from the configuration on disk: it moved only on an edit, a gallery install, or a peer's change broadcast. An inference request may only establish a revision, never replace one. So any other way for the two to diverge left the model permanently unroutable. A configuration edited while a frontend was down lands there, and so does a change in what the revision is computed over: an upgrade that alters the hashed form leaves every stored revision describing a configuration that no longer exists. The only recovery was deleting the row by hand, which is not something a cluster should need. Each frontend now reconciles the stored revisions against the loaded configurations at startup and republishes the ones that disagree. Only those: republishing quarantines every replica loaded under the old revision, so doing it for a model that did not drift would unload a healthy replica for nothing. A model with no stored revision has never been served and is left for its first request to establish. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
Ettore Di Giacinto
|
e6269e3cdd |
fix(distributed): reclaim replica slots held by abandoned loads
A replica row in staging or loading holds its slot, because slot allocation counts every state except unloading. Nothing ever reclaimed such a row: every reconciler pass and the router's eviction query filter state = "loaded", and the per-model probe skips rows without an address, which is exactly what a row that never finished loading has. So a worker that dropped out mid-transfer left a row that pinned the only replica slot for that model on that node. Scheduling then found no free slot and eviction found nothing it was allowed to evict, and the request failed with "no replica slot on <node> and eviction failed: all models busy". The state persisted until an operator intervened. The reconciler now reclaims a row stuck before serving when no load job is driving it. Ownership is decided by the job's LastProgress heartbeat, not by elapsed time: staging a large checkpoint legitimately runs for a long while without touching the replica row, so a deadline would either be a model-size cliff or reclaim a healthy transfer. That heartbeat is the same signal job takeover already trusts. Any error reading the job leaves the slot held, because holding one for another pass costs a scheduling opportunity while a wrong reclaim restarts a multi-gigabyte transfer. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
Ettore Di Giacinto
|
c541dbeef4 |
fix(distributed): check a node answers before scheduling onto it
A node's status comes from its HTTP heartbeat. Backend installs travel over NATS. The two are independent, so a worker that dies stops answering on the bus at once but stays healthy in the database until its heartbeat ages out. Inside that window the scheduler picked a node it could not reach, and the request failed with "no responders available" rather than moving to a node that was up. The scheduler now probes the node it selected and, when nothing answers, marks it unhealthy and selects again. The demotion is what makes the retry terminate: the next selection reads only healthy nodes. It also tells the other frontends what this one learned, so the cluster does not rediscover a dead worker one failed request at a time. Only nats.ErrNoResponders counts as absent. A worker that answers slowly stays eligible, because dropping it would cost capacity that is really there. The probe reuses the models.running subject: a new subject would go unanswered by workers that have not been upgraded, and every one of them would then look dead. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
Ettore Di Giacinto
|
cee87d1608 |
fix(distributed): expire staged request files on the worker
A request that carries a file stages it to the worker, which writes it under its staging directory. Nothing removed it afterwards. The frontend expires ephemeral keys from object storage, but that sweep never covered a worker's local disk, so every image, audio clip and video a worker ever served stayed on it. One worker had accumulated 175 request directories over three months. The volume reached 100 percent, and from that point every backend start failed because the process manager could not create a state directory. The worker now sweeps its ephemeral staging directory on a timer and once at startup, so files left by a crash are reclaimed too. Staged model files live beside that directory and are not touched. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
Ettore Di Giacinto
|
04735cd1f6 |
fix(distributed): stamp config revision at load time
The request middleware merges the caller's prediction parameters into its copy of the model config. core/backend.ModelOptions then hashed that copy, so the revision identified the request body rather than the persisted configuration. EstablishModelConfigRevision stores the first revision it sees and requires an exact match afterwards. The first request after a restart therefore pinned the model to its own temperature, top_p and stop values, and every later request that sent different ones failed with "stale model config revision". No config edit was involved. The loader now stamps the revision when it materializes a config, before any request override reaches it, and ModelOptions reads that stamp. Model administration keeps hashing the same persisted config, so both paths agree on one revision per configuration. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
8f56e4e042 |
fix(vram): persist remote probe metadata (#11487)
* fix(vram): persist remote probe metadata The startup warmer repeated remote size and GGUF metadata probes after every restart because both caches lived only in memory. Store successful HTTP probes for 24 hours so frequent restarts reuse the prior results. Bound the cache, reject invalid records, and purge it when gallery data changes. Local model files continue to bypass persistence. Assisted-by: Codex:gpt-5 * fix(vram): check temporary file cleanup The lint gate rejects the unchecked cleanup call in the persistent cache writer. Assisted-by: Codex:gpt-5.6 [golangci-lint] * fix(vram): make persistent cache optional Remote metadata probes can transfer enough data that operators need control over disk reuse and startup warming. Gallery autoload now gates both behaviors, and the runtime setting applies changes immediately. Assisted-by: Codex:gpt-5 * fix(ui): expose gallery startup pre-warm The existing gallery autoload setting also gates the startup metadata warmer. Name both effects in Settings so operators can find the requested boot control. Assisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
82c191afad |
fix(distributed): keep model replicas config-consistent (#11664)
* docs: design configurable copy buffering Document the context-aware copy buffer option and its validation plan. Assisted-by: Codex:gpt-5 * docs: design durable distributed staging operations Assisted-by: Codex:gpt-5 * docs: design distributed model config revisions Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] * feat(config): add stable model revisions Hash typed model configuration and effective protobuf options deterministically for distributed revision comparisons. Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] * feat(worker): acknowledge exact model stops Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] * feat(nodes): track model config revisions Assisted-by: Codex:GPT-5 [apply_patch] * fix(distributed): retry quarantined model cleanup Stop quarantined replicas by exact process identity, retain failed cleanup as durable capped retries, and compare-and-delete only the claimed registry row. Process one sufficiently leased row at a time so multiple frontends cannot duplicate slow cleanup work. Assisted-by: Codex:gpt-5 * fix(distributed): bind loads to config revisions Assisted-by: Codex: GPT-5 [OpenAI Codex] * fix(modeladmin): apply config revisions consistently Route model edits, patches, state changes, deletion, and peer refreshes through the same revision lifecycle. Quarantine stale replicas before exact cleanup and report durable pending cleanup without failing successful config writes. Assisted-by: Codex: GPT-5 [OpenAI Codex] * feat(distributed): expose model config revision state Document replica revision observability and durable cleanup behavior. Keep pending cleanup explicit in model mutation responses and verify endpoint contracts expose revision state without serialized load options. Assisted-by: Codex:GPT-5 [OpenAI Codex] * test(distributed): cover model revision convergence Exercise cross-frontend quarantine, stale replay rejection, exact cleanup retry, worker re-registration, and current-generation replica convergence against the distributed PostgreSQL harness. Assisted-by: Codex:gpt-5 * fix(distributed): pass config revision CI checks Keep configured gallery sources out of authoritative runtime snapshots only after validating their real schema, and harden rollback snapshots against symlink races and non-regular files. Assisted-by: Codex: GPT-5 [OpenAI Codex] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
0ab632b6bd |
fix(auth): protect HTTP routes by default (#11602)
* fix(auth): default to protected HTTP routes Use a method-aware registry for the small anonymous bootstrap surface. Unknown routes now require credentials instead of inheriting fail-open path classification. Keep node self-service routes behind their registration-token middleware. Global auth no longer rejects valid worker credentials first. Assisted-by: Codex:gpt-5 * docs(auth): document public HTTP surface Assisted-by: Codex:gpt-5 * test(auth): align route coverage with default denial Assisted-by: Codex:gpt-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
0761bd02c7 |
feat(chat): add end-to-end context compression (#11556)
* feat(config): add context compression policy Define the opt-in model configuration contract before the chat middleware consumes it. Document each policy field so later request handling does not invent a second schema.\n\nRefs #9534\n\nAssisted-by: Codex:gpt-5 * fix(config): register compression fields The model editor metadata gate rejects new config fields without descriptions and suitable controls. Register the compression policy so operators can edit its six fields safely. Assisted-by: Codex:gpt-5 [monitoring-prs] * feat(chat): compress long contexts Long conversations currently fail once they reach the model context window. The opt-in policy now summarizes complete older turns before primary inference and preserves the newest tool chains. Both OpenAI and MCP chat routes share the same transformation. Usage metadata and metrics expose each compression event. Refs #9534 Assisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Richard Palethorpe
|
d10374f849 |
feat(router): make KNN a first-class classifier with a persisted, curated corpus (#10652)
* feat(router): make KNN a first-class classifier with a persisted, curated corpus
Add `classifier: knn` — similarity-weighted voting over labelled
example prompts. Unlike score/colbert it needs no classifier model:
label knowledge lives in a corpus seeded and curated through the
admin API, so routing decisions are deterministic, auditable, and
grounded in graded experience rather than a model's opinion.
Epistemic gate: corpus entries below knn.similarity_threshold cannot
vote; when none clears it the classifier activates no labels and the
router uses the fallback — a prompt unlike all labelled experience is
treated as undecidable, not guessed. Decisions record
nearest_similarity (also on fallback rows) so admins can see how far
the nearest labelled experience was; the Routing tab explains
out-of-corpus fallbacks and shows per-label corpus counts.
Persistence: one JSONL file per router under
<data path>/router-corpus (text, labels, vector, embedder
fingerprint). The file is the source of truth; the local-store index
is rebuilt from it at classifier build time and stays a pure
in-memory index. Entries recorded under a different embedding model
re-embed on load. Also corrects the docs' false claim that
local-store collections persist — the embedding cache never survived
restarts (and still doesn't); the corpus does.
Corpus input is API-only by design (entries may contain example user
content): POST /api/router/{name}/corpus seeds (labels validated
against declared policies, embedded server-side, indexed
immediately), GET .../corpus/stats inspects — label counts only,
entry texts are never returned by any surface — DELETE .../corpus
wipes. Admin-gated like the sibling router endpoints, and exposed as
MCP tools (seed_router_corpus / get_router_corpus_stats /
clear_router_corpus) in both the httpapi and inproc clients with
coverage-test route mappings.
Plumbing: VectorStore gains SearchK (top-K was hardcoded to 1);
local-store gets InsertBatch/Delete as optional fast paths;
RouterConfig gains a knn block (embedding_model, k,
similarity_threshold, vote_threshold, store_name) with meta-registry
fields; the classifier dropdown now offers knn and the
previously-missing colbert; embedding_cache is ignored (with a
warning) for knn — it IS an embedding-KNN lookup; the stale
/api/instructions intelligent-routing entry is rewritten (it
described a classifier that no longer exists); swagger regenerated.
Tests: KNN vote/gate specs with hand-computed vote shares, corpus
manager suite (restart reload without re-embedding, fingerprint
re-embed, dedupe, hostile store names), middleware specs (corpus
routing, gate fallback, config validation, cache-wrap refusal),
corpus endpoint specs pinning the texts-never-returned contract, MCP
catalog + route-mapping gates, and a Playwright spec for corpus
stats and the out-of-corpus decision detail.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(router): name consulted corpus neighbours in knn decisions
Every knn decision (decision log rows and the /api/router/decide
response) now carries neighbors: the K retrieved corpus entries by
descending similarity - including ones below the epistemic gate, which
is what makes fallback decisions diagnosable - each as {id, similarity,
labels}. The id is the entry's content hash (first 8 bytes of the
SHA-256 of its text, hex): stable across reseeds and re-embeds, and
text-free, so an external platform that seeded the corpus can recompute
text->id on its own copy and bucket decisions by corpus region (per-
region reliability accounting) without corpus text ever leaving the
server. A corrupt index payload surfaces as an id-less neighbour at a
real similarity instead of disappearing.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* refactor(router): deduplicate knn plumbing and cut corpus hot-path waste
Post-review cleanup of the knn-first-class-router branch; no behaviour
changes on the API surface.
Reuse/altitude:
- RouterKNNConfig.ResolvedStoreName is now the single source of the
router-corpus-<name> default (was hand-derived in four files).
- corpus.ResolveKNNRouter + corpus.Seed carry the shared model
resolution and seed validation; the REST endpoints and the assistant
MCP client are thin transport adapters over them, with sentinel
errors mapped to HTTP statuses at the echo boundary.
- middleware.NewClassifierDeps assembles the classifier dependency set
once for all five entry points (OpenAI, Anthropic, realtime, decide,
corpus) instead of five hand-copied literals.
- router.AllClassifiers feeds both the status endpoint and the
unknown-classifier error, ending the classifier-list drift.
- Per-classifier requirements moved out of validateRouterPolicies into
their buildClassifier arms; the knn arm owns its embedding_cache
opt-out instead of a name-check in the shared wrap tail.
- adminOnly replaces four inline copies of the admin gate in the
middleware routes.
- localVectorStore.Search delegates to SearchK (identical traces).
Efficiency:
- Manager.Add embeds outside the manager mutex and appends to the
JSONL file (O(new) instead of O(corpus) rewrite); a torn tail from a
crash mid-append is tolerated on read and repaired on next write.
- Stats memoises per store keyed on the file's stat fingerprint and no
longer takes the manager mutex, so the 5s status poll stops parsing
vector-laden JSONL and stops blocking behind seeds.
- KNN Classify decodes each neighbour payload once (was twice) and
builds refs and votes in a single pass with one fallback return.
- Corpus file writes fsync before rename/close.
- The corpus manager is built eagerly in newApplication (sync.Once
dropped); test helper dead branch removed.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(router): bind knn corpus vectors to an embedder fingerprint and fail closed on mismatch
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* chore(mcp): align corpus tool prompts and the mutating-tool safety list
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(proto,backend): report embedding shape from the llama-cpp backend
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(embeddings): Go-side pooling — mean/last/decayed_mean with half-life
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* feat(embeddings): accept chat messages[] and per-request pooling on /v1/embeddings
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* chore(middleware): name the failing fields when post-merge validation 400s
An intermittent post-merge validation failure surfaced as an opaque 400
during integration (pooling scheme mismatch that no client had sent).
Log the model, the request's pooling override, and the merged config's
pooling fields at the failure point so the next occurrence identifies
whether the request or the stored config carried the bad value.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix(embeddings): scheme override must not inherit the config's half-life
A model config defaulting to decayed_mean pooling carries
pooling_half_life_tokens; a request overriding the scheme to mean/last
without its own half-life inherited that value, and post-merge
validation rejected the pair the server itself had assembled. Zero the
inherited half-life when the overridden scheme is not decayed_mean; a
request that explicitly pairs a half-life with a non-decayed scheme
still 400s.
Assisted-by: Claude:claude-fable-5 [Claude Code]
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* fix embedding pooling validation and router bounds
Declare backend embedding layouts and reject incompatible pooling modes. Reset local-store dimensions after a full clear, validate KNN thresholds, and add real backend and store integration coverage.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
* ci: run local-store integration tests
Build and install the local-store backend in the Linux test job, then run the existing store integration suite so new specs are discovered automatically.
Assisted-by: Codex:gpt-5
Signed-off-by: Richard Palethorpe <io@richiejp.com>
---------
Signed-off-by: Richard Palethorpe <io@richiejp.com>
|
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
a7bce6a128 |
fix(audio): reject incompatible transform streams (#11565)
The transform WebSocket accepted any model and opened its frame-based RPC. Any-to-any models use a different stream contract, so liquid-audio failed with an unimplemented RPC after the handshake. Reject incompatible model use cases before loading the backend. Direct realtime-audio callers to the OpenAI Realtime API. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
0aaff91ebd |
feat(ui): unify model and backend lifecycle (#11548)
* feat(ui): add installed model lifecycle Models now owns catalog exploration and installed runtime controls under one canonical route. URL-owned state keeps lifecycle context recoverable through links and browser history. Assisted-by: Codex:gpt-5 Playwright * feat(ui): add installed backend lifecycle Backends split discovery from backend-binary management. The canonical page now keeps both lifecycle views under one URL-backed shell while it preserves target-node placement. Assisted-by: Codex:gpt-5 Playwright * fix(ui): repair lifecycle state updates Installed models lost distributed refreshes and kept a deleted selection. Backend searches also stopped tracking URL changes, while batch upgrades stopped after their first error. Preserve background refreshes and finish each requested batch action. Drive catalog results from URL-backed state without losing full metadata. Assisted-by: Codex:gpt-5 [Playwright] * feat(ui): make resource pages canonical Replace Host navigation with canonical Models and Backends lifecycle routes, preserve legacy management URLs, and surface shared host capacity on the Operate overview. Assisted-by: Codex:gpt-5 [Playwright] * feat(ui): complete canonical resource lifecycle Finish the responsive list-to-detail behavior, remove the retired Host implementation, and keep Explore focused on discovery while Installed owns destructive actions. Update regression coverage, localization, documentation, and development binding for the canonical resource pages. Assisted-by: Codex:gpt-5 [Playwright] * docs(ui): record the UI design context Record the approved users, brand character, and design principles so future interface work uses the same product direction. Index the context from the repository's agent instructions. Assisted-by: Codex:gpt-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
6fb9ab38aa |
feat(gallery): add vllm.cpp text-generation models (#11511)
Adds eight curated vllm-cpp entries to the model gallery. Until now the backend had gallery coverage only for MiniMax-H3 video, so serving text on it meant hand-writing engine_args. The flagship tier is what vllm.cpp gates its correctness and speed claims on: Qwen3.6-27B and Qwen3.6-35B-A3B in NVFP4, each with a speculative sibling (MTP on both, DFlash on the 27B). Qwen3-Coder-30B-A3B covers agentic tool use, and Qwen3-4B / Qwen3-0.6B in bf16 are the entries that run where NVFP4 cannot, CPU included. Three details are load-bearing rather than incidental: - The 27B entries pin revision 890bdef7. That repository was later re-quantized in place from NVFP4 to FP8 W8A8 under the same name, so an unpinned entry resolves to different weights and reports nothing. - Qwen3-Coder names tool_parser: qwen3_coder explicitly. Its dialect is byte-identical on the wire to step3p5's, so chat-template sniffing cannot separate them and auto-detection picks wrong. - enable_prefix_caching is deliberately left unset everywhere. It defaults on for dense models and off for the GDN hybrids, and that per-model default is the right answer. num_blocks is sized per model from its real KV footprint rather than copied between entries, which ranges from 20 KiB/token on the 35B to 144 KiB/token on the 4B. Docs: adds features/vllm-cpp.md covering installation, the model table, the pinning rationale and how to choose between the speculative variants, and cross-links it from the existing engine_args reference. It also records that the CUDA images are built for Blackwell only, which is narrower than vllm.cpp's own ten-architecture release and makes an otherwise cryptic "no kernel image is available" failure legible. Verified: gallery suite green; all eight decode and validate as a ModelConfig. qwen3-0.6b-vllm-cpp confirmed end to end on a real cluster, chat plus engine-parsed tool_calls. The NVFP4 entries are not yet runtime-verified: no available node has kernels for them. Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Bash] [Edit] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
88edd7fc7f |
fix(distributed): run cold model loads as durable jobs instead of holding the advisory lock (#11514)
* fix(advisorylock): set statement_timeout alongside lock_timeout
WithLockCtx already overrides a deployment-wide lock_timeout on its
dedicated connection so a blocking pg_advisory_lock() waits its turn
instead of failing with 55P03. statement_timeout aborts that exact same
statement independently, with SQLSTATE 57014, and was not overridden.
Production roles commonly carry statement_timeout=60s. Any guarded
section longer than that (a cold model load stages for tens of minutes)
therefore killed every concurrent waiter:
advisorylock: acquiring lock 9003261067483446873: ERROR: canceling
statement due to statement timeout (SQLSTATE 57014)
Derive it from the same context budget as lock_timeout, with a matching
RESET so the pooled connection is returned clean.
Assisted-by: Claude Opus 5 [claude-code]
* feat(distributed): add ModelLoadJob, the durable cold-load record
A cold load in distributed mode is a long-running background job, but it
was modelled as a synchronous side effect of an inference request: the
whole of it (backend install, multi-GB staging, checkpoint load) ran
inside the per-model advisory lock. Loading a 35.7 GB GGUF held that lock
for ~20 minutes, so every concurrent request for the same model blocked
on pg_advisory_lock and died at the role's 60s statement_timeout.
Introduce the row that lets the lock shrink to a decision. Exactly one
ModelLoadJob may be active per tracking key; that uniqueness — not the
lifetime of a lock — is what de-duplicates concurrent loaders across
replicas. ClaimLoadJob does its read-then-write under the advisory lock
and nothing else: no network, file or gRPC I/O inside the guarded
section, so a claim costs milliseconds no matter how long the resulting
load takes.
LastProgress is a heartbeat rather than a byte counter. A checkpoint load
legitimately moves zero bytes for many minutes, so a reaper keyed on byte
movement would reclaim a healthy job mid-load; byte progress stays the
concern of load_deadline.go. A job whose heartbeat stops for longer than
the orphan window is reclaimable, so a replica killed mid-load cannot
wedge a model permanently.
Failed jobs keep their row for a short grace so an immediately-following
request reports the real cause instead of silently starting a fresh load
of a model that just failed.
No caller yet — the router moves onto this in the next commit.
Assisted-by: Claude Opus 5 [claude-code]
* refactor(distributed): run cold loads as jobs, outside the advisory lock
Route wrapped the entire cold load — node selection, backend install,
multi-GB staging and the remote LoadModel — in the per-model advisory
lock. The lock's job is to de-duplicate concurrent loaders, a decision
that takes milliseconds; holding it for the tens of minutes the resulting
work takes is what turned a dedup mechanism into a cluster-wide outage
for that model.
Split it into a claim and a run. The claim is the only thing left inside
the lock. The run is a background job owned by the claiming replica and
bounded by the same progress-extended deadline as before; every other
request for that model — local or on another replica — attaches as a
waiter and is served the moment the model is ready, with no duplicate
load and no lock contention.
Waiters share one broadcast rather than an ordered queue: they all want
the identical outcome, so ordering them would add fairness machinery that
changes no result. The local channel wakes same-replica waiters instantly
and a 2s DB poll is the authority, because a waiter on another replica
has no channel to close. On wake a waiter re-runs the warm path rather
than trusting the signal — the model may have been evicted in between.
A waiter whose client disconnects returns immediately and the job keeps
running; it belongs to the job record, not to the request. A failure is
recorded on the row so every waiter reports the real cause, and the row
survives briefly so the next request does not read "no job" as "not
loading" and start a duplicate load of a model that just failed.
The runner heartbeats the row on a fixed interval whether or not bytes
are moving, which is what keeps a legitimately silent checkpoint load
from being reclaimed as an orphan. Phase (installing/staging/loading) and
placement ride to the heartbeat on the context, the same seam
load_deadline.go already uses, so single-host paths are untouched.
Non-distributed mode (no DB) keeps the inline load exactly as it was.
Assisted-by: Claude Opus 5 [claude-code]
* feat(distributed): bound the wait for a loading model and answer with progress
A request whose model is cold-loading now attaches to the running job and
is served the moment the model is ready. That wait has to be bounded: a
held HTTP request cannot survive real infrastructure, and an ingress or LB
idle timeout kills a twenty-minute request regardless of what LocalAI
does.
New LOCALAI_MODEL_LOAD_WAIT (default 60s) bounds the CALLER, never the
load — the job keeps running either way. On expiry the request gets 503
with Retry-After and a structured body naming the model, the node, the
phase, byte progress and an ETA. The `error` envelope keeps OpenAI
clients working; `loading` is additive so they ignore it.
The ETA comes from the job's own observed rate and is omitted rather than
guessed until enough bytes have moved for that rate to mean anything: a
confidently wrong ETA on a twenty-minute wait is worse than none.
Retry-After is that ETA when known, clamped to [5s, 300s], and the wait
budget otherwise.
LOCALAI_MODEL_LOAD_WAIT=0 waits unbounded, for deployments with no proxy
in front. Zero in the config struct still means "unset, use the default",
so the CLI records the operator's zero as ModelLoadWaitUnbounded rather
than losing the distinction.
The distributed branch of ModelLoader.loadModel wrapped the router's
error with %s, which flattened it to a string. Use %w: the typed error is
what the HTTP layer keys the 503 off.
Assisted-by: Claude Opus 5 [claude-code]
* feat(api): add GET /api/models/{id}/load-status
A client that receives 503 while a model stages onto a worker needs
somewhere to poll. This returns the same `loading` object the 503 carries
— phase, node, byte progress and ETA — or 404 when no load is running.
Read-only and observability-shaped, so it is deliberately neither
admin-gated nor feature-gated: it explains a 503 the caller just
received, and hiding that behind a per-modality feature would make the
explanation for a failed image request depend on chat permissions. It
also gets no MCP tool, since there is nothing here an admin would manage
conversationally.
Registered on the surfaces from .agents/api-endpoints-and-auth.md: the
swagger block (existing `models` tag, so /api/instructions needs no new
area), the endpoint discovery maps in RegisterLocalAIRoutes, regenerated
swagger, and the distributed-mode docs page. No FLAG_* usecase is
involved, so capabilities.js is unchanged.
Assisted-by: Claude Opus 5 [claude-code]
* feat(ui): show cold-load progress in Chat and retry when the model is ready
A chat request for a model that is still staging onto a worker now gets a
503 carrying live progress instead of an error. Render it: the composer
shows the phase (installing / staging / loading), the node, the percent
and the ETA, then polls load-status and re-sends the request the moment
the model is ready.
Reuses the staging progress idiom the page already had rather than
inventing a second one — the two sources are folded into one
loadProgress, with the load job winning because it is authoritative
across frontend replicas and knows the phase, where the staging operation
only knows about a byte transfer this replica happens to be performing.
Waiting is bounded (three send attempts, ~30 min of polling each), so a
load that never finishes still surfaces as an error rather than as a
spinner nobody questions. An aborted generation stops the polling too.
Assisted-by: Claude Opus 5 [claude-code]
* fix(distributed): check warm-path cleanup errors
The router moved legacy cleanup calls onto newly linted lines. Report
cleanup failures while preserving the fallback to a cold load.
Assisted-by: Codex:gpt-5 [golangci-lint]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>
|
||
|
Richard Palethorpe
|
5c63969760 |
fix: Show MCP connection errors in the UI (#11495)
* fix(mcp): surface configured server failures Keep model-configured MCP servers visible when discovery or connection setup fails, propagate status through distributed discovery, and let the Chat UI show actionable errors while retrying unavailable servers. Add model-editor metadata for remote and stdio configuration and document the expected format, deployment networking boundary, and alternate MCP scopes. Assisted-by: Codex:gpt-5 Ordino golangci-lint Signed-off-by: Richard Palethorpe <io@richiejp.com> * build(compose): match CUDA development image Configure the API image with the cublas, CUDA 13, auth-tagged build settings used by the local development Makefile invocation, including the 24-way Docker build. Assisted-by: Codex:gpt-5 Ordino Signed-off-by: Richard Palethorpe <io@richiejp.com> * revert: keep host build settings out of compose The CUDA development deployment is managed from ~/docker/localai, not the repository example Compose file. Restore the generic example and keep machine-specific build settings in the host deployment. Assisted-by: Codex:gpt-5 Ordino Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(docker): exclude local agent artifacts Keep Claude worktrees and locally installed verification tools out of the Docker build context. These host-only directories added roughly 1.9 GB to every root image build. Assisted-by: Codex:gpt-5 Ordino Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
16193e1982 |
feat(gallery): add Higgs Audio v3 TTS (#11456)
Expose the existing audio.cpp Higgs support as an installable Q8 gallery model and document voice cloning and licensing constraints. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
f7db51bdf5 |
feat(realtime): add shared WebRTC UDP port (#11436)
* feat(realtime): add shared WebRTC UDP port Allow realtime WebRTC peer connections to reuse one configurable UDP mux, and surface listener bind failures through signaling. Assisted-by: Codex:gpt-5 * test(realtime): keep UDP mux alive during bind check The returned SettingEngine owns the UDP listener. Retain it through the duplicate-bind assertion so macOS cannot finalize the listener early and make the exclusivity check spuriously pass. Assisted-by: Codex:gpt-5 [systematic-debugging] * test(realtime): use IPv4 for UDP mux checks Match the socket family used by the WebRTC UDP mux so macOS does not allocate an IPv6 probe that can coexist with the IPv4 listener.\n\nAssisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
7b9167eaad |
feat(llama-cpp): serve Qwen3-TTS through the llama.cpp backend (#11392)
* fix(config): do not read a TTS speaker-encoder mmproj as vision support Qwen3-TTS on llama-cpp ships an mmproj holding the speaker encoder and code predictor. VisionSupported() treated any non-empty MMProj as proof of image input, so every such model would be advertised as vision-capable. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): add TTS request option parsing helper Validates text and speaker reference presence and strictly parses the top_k / top_p per-request params, in a header with no llama.cpp or gRPC dependencies so the standalone C++ unit test gate picks it up. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): range-check the TTS top_k and top_p request params Format validation alone let NaN, infinity and out-of-range values through. The consumer copies both values into the audio generation input unconditionally and only guards its separate sampler assignment with "> 0", a test NaN also fails, so a NaN reached llama.cpp with the guard never firing. top_k must now be >= 0 and top_p must fall within 0.0 to 1.0 inclusive, with the bound written as a negated in-range test so NaN is rejected rather than silently accepted. Also cover the two checks the suite could not previously kill: the whole-string check in the float parser and the int32 range check. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): bump pin to f9e832c10 and carry the TTS server task Picks up ggml-org/llama.cpp#26254 (Qwen3-TTS via mtmd) and #26536 (the short-input audio chunk fix). Adds 0002-add-server-task-type-tts.patch, the server-side half of the still-draft #26603, so TTS runs through the slot scheduler instead of racing it. Remove that patch when #26603 merges. The patch is rebased on top of the score patch: its tokenize-switch hunk collided with the SERVER_TASK_TYPE_SCORE case, and its lone SRV_WRN call passes no variadic argument, which the macro cannot expand. The score patch itself needed no refresh. Also fixes fallout from the bump in grpc-server.cpp: upstream dropped the per-slot n_ctx argument from server_schema::eval_llama_cmpl_schema. Only the schema branch loses it, since forks predating the server-schema split still expect the old argument list. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): implement the TTS and TTSStream RPCs Both were declared in backend.proto but unimplemented. They now submit a SERVER_TASK_TYPE_TTS task and drain the response reader, the same shape PredictStream uses. The streaming path emits a leading sample_rate message and then raw PCM, because ModelTTSStream builds the WAV header itself; the non-streaming path emits a complete WAV to the requested dst. The streamed samples are converted from the pipeline's float32 to signed 16-bit first. MTMD_HELPER_GEN_AUDIO_OUTTYPE_PCM hands back floats, while the header ModelTTSStream writes announces 16-bit samples, so shipping the floats verbatim would decode as noise. prepare.sh and CMakeLists.txt now stage tts_request_options.h alongside the other grpc-server helpers, and register its standalone test with ctest the way passthrough_options_test is registered. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): mask non-codec tokens for Qwen3-TTS generation The Qwen3-TTS gen-audio pipeline maps a sampled backbone token to a codebook row with an unchecked subtraction, in mtmd-helper-gen.cpp: inp.code0 = sampled - codec_0; For ggml-org/Qwen3-TTS-12Hz-1.7B-Base-GGUF the vocab is 155008 tokens, <|codec_0|> is 151936 and the codec codes end at 153983. The model's own tokenizer.ggml.suppress_tokens holds 1023 ids covering 153984..155007, every special above the codec range except <|codec_eos_token|> (154086) which stays reachable as the stop token. Nothing masks the text range 0..151935, so the backbone can sample a text token at any step, the subtraction goes negative, and ggml_compute_forward_get_rows aborts the whole backend process on GGML_ASSERT(i01 >= 0 && i01 < ne01). Complete the mask upstream started: bias every token below <|codec_0|> to -INFINITY for TTS tasks so only codec codes and the codec EOS remain reachable. The biases are appended to task.params.sampling.logit_bias, which common_sampler_init already merges with the model's suppress tokens into one llama_sampler_init_logit_bias, so no sampler is added to the chain. Measured cost is 0.082 ms per sampled token and 1.16 MB, set against a forward pass in the multi-millisecond range. It lands in launch_slot_with_task rather than in a route handler so that llama.cpp's own POST /tts and LocalAI's TTS/TTSStream RPCs are both covered, and <|codec_0|> is resolved from the vocab rather than hardcoded so a model without it is left alone. This is reproducible with upstream's own llama-tts and no LocalAI code loaded, aborting at frame 55 on Q4_K_M and frame 71 on Q8_0, so it is neither a quantization artifact nor an artifact of the gRPC adapter. Two further defects in the same draft pipeline still prevent end-to-end audio; they are independent of this one and are recorded in the task report for an upstream bug report. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): bump pin to 9de0fcf2b and drop the TTS codec mask Upstream fixed the Qwen3-TTS abort in ggml-org/llama.cpp c8e03ce81 ("mtmd/ggml: add ggml_build_forward_order", #26649), landed one hour after the previous pin. ggml_build_forward_expand marks a tensor and all its ancestors for compute, so using it as a pure ordering hint defeated ggml_build_forward_select and made GEN_WAV calls execute the GEN_CODE branch against a stale inp_code0, hitting the get_rows bound assert in ggml_compute_forward_get_rows. That single defect accounts for every abort seen on this model, so 0003-mask-non-codec-tokens-for-tts.patch is removed rather than rebased. The mask changed the observed behavior, but it was perturbing a graph ordering bug rather than fixing a sampling one: at the new pin the whole path works without it. Keeping it would have meant carrying a 152k-entry logit bias, and rebasing it on every pin bump, for no benefit. Verified at 9de0fcf2b with only 0001 and 0002 applied, which both apply clean with no fuzz and needed no rebase: non-streaming HTTP 200, 410924 bytes, 8.56 s RIFF (little-endian) data, WAVE audio, Microsoft PCM, 16 bit, mono 24000 Hz streaming HTTP 200, 560684 bytes, 11.68 s, exactly one RIFF at byte 0, same format, which also exercises the float32-to-s16 conversion at runtime for the first time Pristine unpatched llama-tts at the same pin now also completes, 130 frames to a valid WAV, where it aborted at frame 55 before. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): clear the TTS slot sequence between requests Only the first TTS request in a backend process succeeded. Every later one failed instantly, in about 0.13 s, with "TTS prompt processing failed" from step_prompt, regardless of streaming or non-streaming and regardless of the text. With LOCALAI_SINGLE_ACTIVE_BACKEND=true the process is kept alive between requests, so a deployment would have served exactly one utterance per backend start. The cause is missing KV hygiene, not anything in the gRPC adapter. TTS slots never enter the shared batch: pre_decode() returns early for them and process_tts_slots() drives them instead, so they skip the prompt-cache bookkeeping that clears a slot's sequence between requests. Nothing in the gen-audio path makes up for it: mtmd_helper_gen_audio_reset only clears host-side buffers, and the pipeline always decodes from position 0 into the sequence identified by slot.id. So the second task on a slot writes positions 0..N over the first task's tokens and llama_decode fails. Fix is one call to slot.prompt_clear(), the same helper the normal path uses, in the SERVER_TASK_TYPE_TTS branch of launch_slot_with_task before set_input. It goes into 0002 rather than a new patch file because it is a defect in the code that patch introduces, and the header now records it as ours so we know whether it still needs carrying if #26603 merges without it. Verified in one backend process, different text on every request: three consecutive non-streaming requests, three consecutive streaming requests, and an interleaved non-streaming, streaming, non-streaming, streaming run. All ten returned HTTP 200 with RIFF ... WAVE audio, Microsoft PCM, 16 bit, mono 24000 Hz, the streamed ones carrying exactly one RIFF header at byte 0, and every output measured as real speech rather than silence or a truncated fragment. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): expose max_frames for TTS requests The Qwen3-TTS backbone does not always emit <|codec_eos_token|>, and when it does not, generation runs to upstream's 512-frame n_predict default. At the model's 12.5 Hz frame rate that is 40.96 s of audio, which a short input can trigger: one request in this session produced 40.96 s for a ten-word sentence. prepareTTSTask hardcoded n_predict to -1, so callers had no way to bound it. Add a max_frames key alongside top_k and top_p, parsed with the same strict whole-string parsing so a typo is an error rather than a silently truncated value, and rejected with a field-naming message when negative. 0 keeps the existing sentinel convention and means unset, so a request that omits it behaves exactly as before. Named max_frames rather than n_predict because frames are what the parameter means at a TTS endpoint: one frame is 0.08 s of audio. The 512-frame default is deliberately unchanged. Lowering it would truncate legitimately long inputs, which is a worse failure than an occasionally overlong one. Verified end to end on one text of thirty words: max_frames=25 HTTP 200, 96044 bytes, 2.00 s, exactly 25 frames max_frames=50 HTTP 200, 192044 bytes, 4.00 s, exactly 50 frames no max_frames HTTP 200, 572204 bytes, 11.92 s, stopped at its own codec EOS after 149 frames, unchanged behavior max_frames=-1 InvalidArgument "max_frames must be >= 0, got \"-1\"" max_frames=many InvalidArgument "max_frames must be an integer, got \"many\"" Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): send the TTS sample rate up front, and tidy three review items Four items from the Task 4 review. Streaming first-byte latency. TTSStream sent the sample-rate reply only once the first audio result arrived, and a chunk needs a whole 72-frame window, roughly 5.8 s of audio and far longer in wall time on CPU. The Go side blocks on that reply before it can emit the WAV header, so a streaming client sat at zero bytes for the whole stretch. The rate is a property of the loaded model and is available synchronously from mtmd_gen_audio_get_info, so it now goes out immediately after post_task and the rate_sent bookkeeping is gone. Measured on a warm model, first byte drops from 30.48 s to 0.014 s, and the output is still a valid WAV with exactly one RIFF header at byte 0. Unchecked close. The non-streaming path ignored ofstream::close(), so a failure that only surfaces on flush was reported as success while leaving a truncated file at dst. It now returns INTERNAL like the other write failures. Wrong comment on set_lang. gen_audio::inp::get() already maps a stored blank to nullptr, so our guard is behavior-preserving, not behavior-fixing. The comment claimed otherwise; the code was right. Repetition penalty. penalty_last_n = -1 is inert at this pin, because llama_sampler_init_penalties clamps it with std::max(penalty_last_n, 0) and then builds a disabled sampler, so the 1.05 penalty never applies. Upstream's README attributes looping to a missing repeat_penalty, so it was worth testing as a root-cause fix for the model running to the frame cap. Dropping the line lets the sampling default of 64 apply, which was confirmed in the sampler chain trace as penalty_last_n = 64 with repeat_penalty = 1.050. Over 15 uncapped short requests each way it did not help: 0 of 15 ran to the cap with the penalty inert, 1 of 15 with it active. Both lines are therefore kept for parity with upstream's draft, and a comment now records that the pair is inert and why, so the next reader does not believe a penalty is applied. max_frames remains the way to bound output. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * build(llama-cpp): let unpatched forks opt out of the TTS task turboquant and bonsai copy grpc-server.cpp into llama.cpp forks that do not carry our patches. disable-tts-task.sh injects the same kind of preprocessor switch disable-score-task.sh already uses, so those builds answer UNIMPLEMENTED rather than failing to compile. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(config): keep a TTS speaker-encoder projector out of vision detection Task 1 exempted a declared-TTS model's mmproj from VisionSupported, but the first real gallery entry with an mmproj still came back vision-capable through two paths the earlier fix did not close. GuessUsecases has no FLAG_VISION branch, so it falls through to true for any chat-ish model. That is not just a wrong answer at the call site: syncKnownUsecasesFromString rewrites KnownUsecaseStrings from HasUsecases, and the loader calls it more than once per config file, so the guessed FLAG_VISION is written out and parsed back into KnownUsecases as if the operator had declared it. Give GuessUsecases a FLAG_VISION branch that defers to the same explicit signals VisionSupported uses. Second, llama.cpp builds an mtmd context for the speaker-encoder projector and reports its media marker on the first chat probe, which resurrected vision after the model had been used once. Apply the same declared-TTS exemption to MediaMarker that the mmproj check already had. Verified against the qwen3-tts-llamacpp-q4 gallery entry: no vision capability and no image input modality, before load, after a TTS request, and after a chat probe. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add Qwen3-TTS entries for the llama-cpp backend Two entries over upstream's own GGUF conversion, Q8_0 and Q4_K_M, each pairing a backbone with the Q8_0 projector. Named to sit alongside the existing qwen3-tts-cpp entries rather than replace them. Also tags the llama-cpp backend text-to-speech / TTS so the backend browser surfaces the capability. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: cover Qwen3-TTS on the llama-cpp backend Adds the gallery variants, the two-file mmproj configuration, the required voice reference, and the language and sampling knobs. Also corrects the streaming-support list, which named only voxcpm. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(config): register llama-cpp as a TTS and voice-cloning backend The branch taught the llama-cpp backend to serve Qwen3-TTS and shipped two gallery entries for it, but never told the capability table. llama-cpp still declared only the text RPCs and usecases, so: - VoiceCloningForModel returned nil at the capability check, before it ever reached the model's own tts.voice_cloning override, and /tts answered 400 "selected model does not support reference-audio voice cloning" for any localai://voice-profiles/... voice. No model YAML could opt back in. - GET /api/backends/usecases did not list tts for llama-cpp, so the gallery greyed out the TTS filter for the entries this branch adds. - The React TTS page saw voice_cloning: null and kept both models out of the Voice Library. Add the TTS RPCs and usecase, and the reference-audio contract. The contract needs narrowing, because the per-backend switch in VoiceCloningForModel ends in a permissive default: an unnarrowed entry would have advertised reference-audio cloning on every GGUF chat model in the gallery. Narrow on the declared TTS usecase rather than the model name. The TTS checkpoints are the only llama-cpp models carrying known_usecases: [tts]; name matching would have to guess at third-party repacks, and "base", the substring the neighbouring Qwen and vLLM cases key on, is a routine word in text-model names. The check reads the declared bit directly instead of going through HasUsecases, which falls through to GuessUsecases and would hand the decision to a heuristic that never had a llama.cpp TTS model in mind. DefaultUsecases stays [chat]: a bare GGUF served by llama.cpp is a chat model, and both the gallery filter and the importer read that field. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): declare what nemotron-3-nano-omni actually accepts The entry is backend: vllm-omni with known_usecases: [chat, completion], no mmproj and no media marker, so it used to report vision only through the blanket GuessUsecases fallthrough that the vision branch in this branch removed. Nemotron 3 Nano Omni is a multimodal understanding model: image, video and audio in, text out. Declaring that is what the sibling vllm-omni-qwen3-omni-30b already does. known_usecases gains vision only. FLAG_VIDEO is video GENERATION, an output modality, and this model generates none; video and audio input belong in known_input_modalities, which is where AudioInputSupported and VideoInputSupported read them from. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(importers): import a Qwen3-TTS GGUF repo as TTS, not chat The llama-cpp importer hardcodes known_usecases: [chat] and assigns any mmproj-matching file as a vision projector, so ggml-org/Qwen3-TTS-12Hz-1.7B- Base-GGUF imported as a chat model with vision. Both fields were wrong, and the model was unreachable from /tts and from the Voice Library. Filenames cannot fix this. A Qwen3-TTS repo has the exact shape of a vision repo, one backbone GGUF plus one mmproj-*.gguf, so the projector's own header is the only honest signal: mtmd writes clip.has_gen_audio_encoder for the projectors it can drive as a speech pipeline and refuses to build one without it. Probe the selected mmproj for that flag, reusing the range-fetch the MTP detection already does, and declare tts when it is set. The mmproj assignment then stops reading as vision on its own, since a declared-TTS model already exempts its projector from vision detection. The probe is best-effort like the MTP one: a network blip leaves the chat default in place rather than failing the import. Verified against the real artifacts on disk: the Qwen3-TTS projector reports gen-audio, its backbone does not. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): stop non-TTS models crashing on the new pin Two regressions, both hit every ordinary llama-cpp model and neither was caught locally because every test on this branch loaded a TTS model. The first is a null dereference. server_slot::tts_ctx::reset() called mtmd_helper_gen_audio_reset() unconditionally, but the gen-audio pipeline is only allocated for models carrying a gen-audio mmproj, and upstream's implementation reads ctx->pipeline before null-checking anything. Since server_slot::reset() runs during slot initialization for every model, any non-TTS model segfaulted the backend the moment it loaded. Guard the call on the is_supported() predicate already defined beside it, and keep the plain field resets unconditional. The second is unrelated to TTS and came in with the pin bump. PredictOptions.Penalty is a bare proto float, so a caller that names no repetition penalty sends 0 rather than omitting the field. Since 9de0fcf2b, common_sampler_init() rejects a non-positive penalty_repeat outright because it would divide logits by zero, turning every such request into "Failed to initialize samplers". Treat 0 as unset and leave llama.cpp's own neutral default in place. Verified with the same suite CI runs, which is what caught both: tests/e2e-backends passes 6 of 6 including the load and predict specs that were red. Qwen3-TTS still synthesises on both paths, 24 kHz mono 16-bit WAV with exactly one RIFF header on the streamed output. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
daa8d2adbd |
fix(gallery): identify invalid preload JSON (#11434)
Wrap PRELOAD_MODELS decoding failures with the setting name and expected top-level shape so startup errors point directly to the invalid configuration. Document the required array format and cover scalar and empty-array inputs. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
a0f50b2af2 |
feat(vllm-cpp): serve MiniMax-H3 video+audio generation (#11424)
* feat(vllm-cpp): serve MiniMax-H3 video+audio generation
vllm.cpp's C ABI grew a video slice (ABI v12): a second engine handle
loaded from the MiniMax-H3 checkpoint SET, one blocking generate, and a
composed ffmpeg argv the caller execs. This wires that into LocalAI's
existing /video endpoint, so `vllm-cpp` now serves both text and video
and a clip comes back as an MP4 with a real audio track rather than a
silent render.
The video engine is a separate handle rather than a mode of the text
one because H3 is not a model directory: the DiT, the text encoder and
two VAEs are separate artifacts, and vllm.cpp has the two loaders refuse
each other's checkpoints. `Load` takes the video branch when the config
declares any of the video options; `parameters.model` is the DiT and the
rest of the set is named in `options:`.
Three details are worth calling out because getting them wrong is
expensive:
- The partition is DECLARED, not detected. The community quantisations
strip the release metadata and the FL2VA and Ref2VA DiTs are
byte-structurally identical, so the engine refuses to generate until
it is told which it has. Worse, a mismatch does not fail cleanly: a
reference passed to an FL2VA DiT renders for hours and returns a
coloured lattice over the frame. The backend refuses that combination
up front instead.
- ffmpeg comes from the host. libvllm writes frames plus a WAV and
composes the mux argv, then spawns nothing - that process boundary is
upstream's decision. The backend execs it, the same arrangement
vibevoice-cpp uses for transcoding, and ffmpeg also converts a
start_image upload into the binary PPM at the exact output canvas the
engine requires.
- It is slow. Roughly 176 s per denoise step at the default 1344x768
canvas on a 20-SM device, so the 50-step default is a multi-hour job.
Nothing on this path imposes a deadline.
The /video endpoint no longer forces 512x512 when the request omits the
geometry. Every video backend already supplies its own default for a
zero (512x512 for stablediffusion-ggml, 1280x720 for diffusers, 832x480
for longcat-video, 1344x768 for H3), so the hardcoded value only ever
overrode the model's trained canvas with one three of the four were
never trained at.
Moving the engine pin from ABI v10 to v16 also grows the text
vllm_model_params mirror by the v14 device field and the v16 KV-sizing
knobs. LocalAI sets none of them - 0 is the pre-v14 engine byte for byte
- but the struct SIZE is part of the layout contract, so leaving them
out would have vllm_engine_load read past the allocation.
Gallery: `minimax-h3-fl2va-q4` installs the Q4_K_M FL2VA set (~40 GB
across five weight files plus the two VAE configs that carry the latent
statistics).
Assisted-by: Claude:claude-opus-5 golangci-lint yamllint go-vet
* fix(vllm-cpp): unbreak the Darwin build at the new engine pin
src/capi/vllm_c.cpp opens one `extern "C" {` for the whole ABI surface,
so file-local helpers declared inside it inherit C linkage. The video
slice added one that returns std::string, which Apple Clang reports as
-Wreturn-type-c-linkage and vllm.cpp's target-local -Werror turns into a
build failure. GCC and upstream Clang do not diagnose it, so only the
metal-darwin-arm64 job saw it.
Suppress it the same way this Makefile already suppresses Apple Clang's
-Wgnu-folding-constant on the Metal build. The helper is never called
across the boundary so the warning describes no hazard here, but it is a
real upstream wart: the fix belongs in vllm.cpp, hoisting the helper
above the extern "C" block, and this flag should go when a pin carrying
that fix lands.
Assisted-by: Claude:claude-opus-5
* fix(vllm-cpp): patch the engine clone instead of the warning flag
The -Wno-return-type-c-linkage added in the previous commit does nothing.
vllm_cpp_set_warnings adds `-Wall -Wextra -Werror` as PRIVATE target
options, so they land after anything CMAKE_CXX_FLAGS contributes, and
-Wall re-enables the -Wreturn-type group that -Wreturn-type-c-linkage
belongs to. The darwin job failed again on the same line, which is the
evidence: a consumer cannot wave this off from outside the engine.
Position is the only fix, so carry it as a patch against the pinned SHA,
the way longcat-video patches its own upstream. It hoists the helper
above the `extern "C" {` that gives it C linkage; it is file-local and
never called across the boundary, so nothing else moves.
`git apply` is unguarded on purpose: a patch that stops applying must
fail the clone loudly, because the alternative is a pin that silently
ships without a fix it is documented to carry. The patch header names
what retires it - a pin carrying the fix upstream, where it belongs.
Verified by applying the patch with `git apply` to the exact blob at the
pinned SHA and diffing the result against the intended file.
Assisted-by: Claude:claude-opus-5
* chore(vllm-cpp): bump the engine pin to ABI v17 and drop the vendored OrEmpty patch
The OrEmpty linkage fix this backend carried as patches/0001-* landed upstream
(mudler/vllm.cpp#195, 7534da65), so the patch has done its job. It is deleted
rather than left in place: the Makefile applies patches/*.patch unguarded and
documents that "a patch that no longer applies must FAIL the clone", so keeping
it against fixed source would break the build the moment the pin moved. Bumping
the pin and deleting the patch therefore have to be the SAME change.
Pin f921062b -> 776c56f1 (current vllm.cpp main).
That range also carries the engine's ABI v17 (vllm_server_main: the OpenAI server
published on the public surface). registerLib compares the library's
vllm_abi_version against `abiVersion` for EXACT equality, so the constant moves
16 -> 17 in the same commit or every load fails with an ABI mismatch.
The bump is safe for the layout assertions in video_test.go: diffing include/vllm.h
across the two pins shows zero struct-field changes -- v17 adds one function
declaration, the version macro and a doc comment, nothing else -- so every
unsafe.Offsetof in the video params test still holds.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
* chore(vllm-cpp): re-pin to pick up the VLLM_CPP_SERVER=OFF link fix
The previous pin carried vllm.cpp's ABI v17 (vllm_server_main) but not the guard
that makes it link when the server is compiled out. This backend builds libvllm
with VLLM_CPP_SERVER off, so the darwin lane failed at the dylib link with
vllm::entrypoints::openai::VllmServerMain undefined.
Fixed upstream in mudler/vllm.cpp#202: the C entry point is now guarded, so the
symbol is still exported (ABI v17 stays resolvable for dlopen) while the
no-server arm reports the missing capability instead of dragging in a translation
unit that was never compiled.
Verified upstream in BOTH arms before re-pinning: SERVER=ON builds and runs, and
SERVER=OFF configures, links, produces libvllm.so, and `nm -D` shows
vllm_server_main exported next to vllm_video_generate and vllm_transcribe.
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Adiraandlocalai-org-maint-bot
|
ab52813342 |
feat(modelartifacts): support bounded parallel Hugging Face file downloads (#11162)
* feat(modelartifacts): support bounded parallel Hugging Face file downloads Closes #11114. Snapshot materialization fetched every file through the sequential executor in DownloadFilesWithContext, so a repository split into many shards spent most of its wall clock in per-file request latency rather than moving bytes. Add DownloadFilesWithConcurrency, an errgroup with SetLimit, and keep DownloadFilesWithContext as a wrapper that passes a limit of 1. That leaves the two non-artifact callers (core/gallery and the model config loader) on exactly the path they had: tasks still run in slice order, and the first failure still returns before any later task starts. Only whole files run in parallel. A single file is never split, so the .partial resume machinery and the per-file SHA check in downloadTaskWithRetry are untouched. Two details the parallel path forced: - completedBytes becomes an atomic.Int64. Several AfterDownload hooks add to it while other files' progress callbacks read it; without this the race detector reports three races on the new specs. - The caller's status callback is serialized. The sequential path gave it an implicit guarantee of never being entered twice at once, and it belongs to the caller, so the executor keeps that promise rather than pushing locking onto every caller. AfterDownload is deliberately not serialized -- it does the verify-and-promote work that parallelism exists to overlap. Manifest order needed no work: each hook already writes its own manifest.Files slot by snapshot index, so entries stay in snapshot order whatever the completion order. A spec now pins that. The default is 1, unchanged behaviour. A shared models volume is often the bottleneck rather than the link, so raising it is a deployment decision; --artifact-download-concurrency and LOCALAI_ARTIFACT_DOWNLOAD_CONCURRENCY expose it on both `run` and `models install`. Not done here, per the issue: no chunk-level parallelism within a single file, and no throughput measurements across concurrency 1/2/4/8 -- that needs a representative sharded repo and a real link. Assisted-by: Claude:claude-opus-5 go-test gofmt Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> * feat(modelartifacts): expose download concurrency in settings Follow-up to review feedback on #11162: - The CLI flag and docs no longer describe the limit as Hugging Face specific. It applies to any artifact source, as @mudler pointed out. - artifact_download_concurrency is now a persisted runtime setting and is editable from the WebUI, so it can be changed without a restart. The manager's limit becomes an atomic.Int64 behind SetDownloadConcurrency, because a live runtime setting can be updated while a materialization is already in flight. Injected materializers stay compatible through an optional setter interface, so a manager that does not implement it is simply left alone. Verified before taking this on: go build, go vet and go test -race all pass for pkg/modelartifacts, pkg/downloader and core/config. The React UI builds with vite, artifact_download_concurrency is present in the built Settings chunk, and eslint reports the same 8 pre-existing warnings on Settings.jsx as it does without the change. Implementation contributed by localai-org-maint-bot on the review thread; reviewed, verified and signed off by me. Assisted-by: Codex:gpt-5 Assisted-by: Claude:claude-opus-5 go-test vite eslint Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> --------- Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
9a6156d808 |
feat(nemo-speech-cpp): add the NVIDIA NeMo-Speech.cpp backend (#11406)
* feat(nemo-speech-cpp): scaffold the backend and upstream build
Adds the backend skeleton and the NeMo-Speech.cpp build, pinned at
2e12e2def8a98ed06666f7ee3ca94e7193e04be4. The Go side is deliberately a stub:
it dlopens the runtime and starts the gRPC server, later work fills in the
symbol table and the model logic.
Three details of the upstream layout differ from what the plan assumed, and the
build reflects the real tree:
* The TTS C ABI ships as libnemo_speech_tts, not libnemo_speech_tts_c. Upstream
compiles c_api.cpp straight into the implementation library and only aliases
the nemo_speech_tts_c CMake target, so no _c object exists on disk. ASR and
NMT do build a real _c shim.
* Shared objects land in build/bin, since upstream points
CMAKE_LIBRARY_OUTPUT_DIRECTORY at ${CMAKE_BINARY_DIR}/bin.
* The ASR and NMT _c shims carry a DT_NEEDED on libnemo_speech_asr and
libnemo_speech_nmt, so those are staged and packaged alongside them.
Otherwise dlopen fails at startup.
The ggml patch step uses an order-only prerequisite. cmake writes into the
checkout and bumps its mtime past the sentinel, which would otherwise re-run
git apply over an already-patched tree and break every incremental build.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): make 'build' produce the package and bundle the ITN stack
Addresses the review of the scaffold commit.
backend/Dockerfile.golang runs 'make -C backend/go/$(BACKEND) build' and then
copies package/ into the final image, so 'build' has to end with a populated
package/. It only staged shared objects, which would have shipped an image with
no binary and no libraries at all. The old staging recipe is now stage-libs and
the chain is stage-libs, nemo-speech-cpp-grpc, package, build, matching every
sibling Go backend.
Text normalization was packaged incorrectly. nemo_speech_text_normalization is
STATIC but links sparrowhawk, fstfar and fst PUBLIC, so they land as DT_NEEDED
on libnemo_speech_asr.so, and they live in a project-local prefix that nothing
else provides. WITH_NORM stays ON by default on Linux, since normalization is a
wanted feature. Instead stage_libs now copies .deps/itn/lib when WITH_NORM=ON,
and package.sh bundles it.
Staging that prefix is still not enough on its own: Sparrowhawk drags in
protobuf, re2 and absl, which neither build_itn_deps.sh nor
package-system-libs.sh provides. Rather than hard-code another hand-maintained
list, package.sh now walks the DT_NEEDED entries of everything staged and copies
whatever is unresolved, skipping the core set and the GPU set that the shared
scripts already own. It fails at package time, not at first dlopen, when
something cannot be resolved. On a WITH_NORM=OFF build the closure is already
complete and it copies nothing.
Restore CGO_ENABLED=0 on the Go build to match whisper, parakeet-cpp and
omnivoice-cpp. Note that purego reaches dlopen through fakecgo, so the binary is
dynamically linked either way; what the flag changes is the NEEDED set, and
lib/ld.so routing in run.sh exists precisely because the binary is not static.
Replace the hand-rolled .patched sentinel with upstream's
scripts/apply-ggml-patches.sh. It applies the series in filename order, exits
non-zero when a patch does not apply, and detects "already applied" by comparing
the full-series tree hash rather than an mtime, so it is safe to run every time
and there is no sentinel left to go stale or to wedge the build when deleted. It
is wired as an order-only prerequisite so running it does not force a relink.
Also: correct the package.sh header, which claimed three shared objects when
there are five and none of the TTS ones carry a _c suffix; give 'make test' the
LD_LIBRARY_PATH the dlopen tests will need; document that a NEMO_SPEECH_VERSION
bump needs 'make purge'; and extend 'clean' to remove package/ and the ITN
libraries.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): make the closure guard fail closed and give CI its toolchain
Addresses the second review round.
The dependency-closure guard failed open. Its glob expands once per pass, so
each pass advanced the closure by exactly one level, and the fixed count of five
passes then fell out of the loop without checking whether anything remained. An
eight-deep chain packaged six libraries, exited zero and reported success. That
is the case the guard was written for: asr to sparrowhawk to protobuf to absl
already runs several levels deep, so a WITH_NORM build could ship missing its
deepest libraries and fail at first dlopen. The loop now runs until the staged
set stops growing, and exhausting the bound is a hard error rather than a silent
exit.
For the same reason, a build image with neither readelf nor objdump no longer
warns and skips. It cannot show the package is complete, so it refuses to ship
it. The guard is entered only when there is something to check, so an empty
package cannot trip the new error.
Dockerfile.golang installed ninja-build only in the Vulkan branch while this
Makefile runs cmake -G Ninja unconditionally, so the CPU, cuBLAS and L4T images
could not configure at all. ninja-build moves to the common apt list; it does
not change CMake's default generator, so it is inert for the other backends.
gcc-12 was nowhere in the tree, yet WITH_NORM defaults ON and
build_itn_deps.sh needs it, so the committed default was unbuildable in CI.
Install it, with the protobuf, absl, re2 and autotools that Sparrowhawk and
OpenFST need, gated on BACKEND so the other Go images do not carry it. The list
follows upstream's own docker/Dockerfile, trimmed of the gRPC, portaudio and
python entries a BUILD_GRPC=OFF build does not use. Text normalization stays ON:
downgrading it silently would ship a backend advertising a feature it lacks.
Also: make test depend on stage-libs, so LD_LIBRARY_PATH is not an empty
directory on a clean tree, and add an engine target so Dockerfile.golang's
cacheable prebuild layer is not skipped and a CUDA build stops recompiling all
of upstream on every Go-side change.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): pin protoc for ITN and make the norm stack its own target
Addresses the third review round.
Dockerfile.golang installs protoc 27.1 into /usr/local/bin, ahead of /usr/bin,
while libprotobuf-dev is the distro's 3.21 on noble and 3.12 on jammy.
Sparrowhawk resolves protoc from PATH at make time (configure.ac uses
AC_CHECK_PROG, so PROTOC substitutes to the bare word, and src/proto/Makefile.am
invokes it) and commits no pregenerated stubs, so the rule always runs. Code
generated by 27.1 includes google/protobuf/runtime_version.h and a
PROTOBUF_VERSION guard the older headers lack, so the WITH_NORM build could not
complete. Pin PROTOC to the apt one for that step; configure documents that a
pre-set value wins. The apt protoc and libprotobuf-dev come from one source
package at one version, which is the property that makes this correct.
The text-normalization stack is now a target keyed on a file build_itn_deps.sh
actually produces, rather than a side effect of the runtime library rule. As a
side effect make could not see whether it existed, so once the library was up to
date the script could never run again: a tree built WITH_NORM=OFF could not move
to ON, and make test hard-failed with no escape but a full 345 MB clean. It is
now built on demand and reachable on its own as 'make itn'. Staging keys on the
prefix existing rather than on WITH_NORM, so it stages what the tree actually
built, and package.sh's closure guard remains the backstop.
An already-configured build tree also now wins over the platform default, so a
tree built WITH_NORM=OFF is not silently reconfigured to ON by a bare make test,
which is what demanded gcc-12 from developers who chose not to have it. An
explicit WITH_NORM= on the command line still overrides both, and the ITN rule
preflights for gcc-12 with an error that names the alternative.
Move ninja-build out of the shared apt layer into the existing BACKEND-gated
block. Dockerfile.golang serves 225 matrix entries and only this backend
configures with -G Ninja, so the common list is byte-identical to master again
and no other image loses its cache.
Drop libabsl-dev and correct the comment that justified it. No base image here
ships protobuf 25, so nothing needs the absl split, and the cmake glob looks in
/usr/lib rather than the multiarch directory Ubuntu actually uses, so the
package could never have contributed anything.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): move the backend apt gate below the expensive layers
Addresses the fourth review round.
The nemo-speech-cpp apt block sat immediately after the shared apt layer, above
the Vulkan SDK build, the CUDA and ROCm installs, the Go toolchain and the
protoc download. Docker keys each layer on its parent, so inserting a step there
re-keys everything below it: a byte-identical shared layer is not enough, and
merging as it stood would have forced all of those to re-execute once for every
Go backend image. Move it down beside the existing opus, crispasr and
sherpa-onnx gates, which sit after those layers for the same reason.
Checked the ordering both ways before moving. Nothing between the two positions
uses these packages: the Vulkan and opus blocks install their own ninja and
pkg-config, go install protoc-gen-go needs the Go toolchain rather than protoc,
and the protoc 27.1 step is a release-binary download that needs neither
protobuf-compiler nor libprotobuf-dev. Nothing in the block needs anything those
layers provide; it uses only apt, and the mirror rewrite from the first RUN
persists in the image. It also runs no update-alternatives, so the default
compiler stays untouched for later layers. The diff against master is now a
single additive hunk with no shared layer touched.
Also preflight ITN_PROTOC. configure gates a preset PROTOC on test -n alone, so
a path that does not exist is accepted and the error surfaces much later as a
bare "No such file or directory" from inside make -C src/proto. The pin
introduced that failure on a box whose only protoc is in /usr/local/bin, which
worked before. Check it alongside the gcc-12 check and name the ITN_PROTOC=
override in the message.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): parse model options
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): guard the empty option value and warn on a bad gpu index
An empty value like "vad_model:" must stay empty, since callers read the
empty string as "unset". That branch of resolve() had no spec: dropping the
guard left every spec green while parseOptions started returning the models
directory itself. Add the spec that fails without the guard.
A known key with an unparseable value is a typo, not a config from a newer
backend, and "gpu:banna" failed expensively: the model loaded, produced
correct output, and ran on CPU with no signal anywhere. Log it. Unknown keys
stay silently ignored, which is what keeps configs forward compatible.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): detect model family and discover TTS assets
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): bind the C ABI with layout assertions
purego binds by name at runtime and the config structs are passed by pointer,
so both a renamed symbol and a mismatched struct layout would otherwise survive
a green build. registerSymbols names the failing symbol, and the layout specs
compare each Go mirror against the size the library reports for itself, against
the offsets a C compiler produces for the installed headers, and against the
default values upstream writes into the structs it returns.
Two of the bindings differ from the plan because the headers do. The plan's
nemo_speech_diar_segments signature omits the segmentation-config pointer that
diar.h declares as the second parameter, which would have shifted the output
buffer, the capacity and the count pointer one position each. And
nemo_speech_diar_stream_push_f32 was missing from the symbol table although
standalone diarization cannot work without it.
Also close the two panic and equality gaps left in family.go: ValueString panics
on a mistyped general.architecture, and the self-codec guard compared a Cleaned
candidate path against an uncleaned one, so a doubled separator let the primary
GGUF be selected as its own codec.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(nemo-speech-cpp): run the ABI specs in CI and refuse to skip them
The layout assertions were inert. TEST_PATHS does not cover this backend and
the per-backend list in test-extra had no entry for it, so nothing invoked the
package's tests. Add it next to depth-anything-cpp, supertonic and vllm-cpp,
the group whose own test target carries its build prerequisites; stage-libs
already pulls the native build chain, so no prepare-test-extra entry is needed.
The skip guard was also loader-inconsistent: librariesPresent stats bare
filenames relative to the working directory while openLibraries resolves them
through the loader search path, so any invocation other than make test skipped
every library-backed spec and still reported green. NEMO_SPEECH_REQUIRE_LIBS=1
turns that into a failure naming the directory and the remedy, and the Makefile
test target sets it. Unset, the plain skip survives so a developer without a
build can still run the pure-Go layer specs.
Trim the default-value fingerprint from roughly forty assertions to eight. It
was pinning tunables such as threads and flush_partial_chunk, so a legitimate
pin bump would have failed with a message reading like a layout error. What
survives is only header-documented contract: the lone non-zero max_alternatives,
the run of -1 sentinels and the zero that witnesses where it stops. Verified the
narrowed spec still catches a mirror and offset table corrupted in lockstep,
which is the one class only this layer sees.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): select the family at load and gate RPCs on it
Load sniffs the GGUF architecture, maps it to a family and dispatches to
the family's loader. requireFamily gates every other RPC, returning
Unimplemented naming both the loaded and the wanted family so a
misconfigured model YAML produces a message a user can act on.
The family is committed only once its loader has succeeded. A load that
fails part way through would otherwise leave the gate open on a handle
that was never created.
cstr uses runtime.Pinner rather than an ordinary Go allocation. The
address crosses the ABI as a uintptr, which the collector does not
trace, so incidental reachability through the release closure is not a
guarantee: a caller discarding that closure could have the bytes
collected before the create call reads them. Pinning is the sanctioned
mechanism, makes the release function do real work, and turns a dropped
release into a loud leaked-Pinner panic instead of silent corruption.
Free overrides the base no-op to destroy the handle and reset the
family. Every family owns C memory only its own destroy entry point can
release, so without this an unloaded model leaks an acoustic model.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): pin the load ordering, close the engineMu race
Three review items, plus a defect the race detector turned up.
The spec covering "no family selected after a failed load" wrote junk to
a .gguf, so Load returned at ggufArchitecture before a family was ever
chosen and the assertion was vacuous. Generalised the GGUF test helper
to take a string architecture, and added a spec that loads a magpietts
GGUF with no sibling codec, so familyFor succeeds and discoverTTSAssets
then fails. It self-guards on ggufArchitecture so it cannot degrade back
into the earlier path.
requireFamily read n.fam unlocked while Free wrote it under engineMu,
which the race detector confirms is a real race. pkg/grpc/server.go
calls Free without the backend lock every other RPC holds, so teardown
can land mid-request. withEngine now takes the lock, checks the family
and runs the body under one acquisition; two would leave a window for
Free to destroy the handle between check and use. The locking protocol
is stated in both directions for the RPCs still to be written.
Running -race also enables checkptr, which aborts on cstr's pointer
being read back by goString: converting a uintptr to a pointer is fatal
whenever the address lands in a Go allocation, so a pinned Go buffer can
never be dereferenced from Go. The pointer is for C alone. Both helpers
now document the one-way contract, and goString is tested against a real
C-owned string by rebinding the version symbol to return a raw char*.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): implement offline transcription
Create the ASR recognizer in loadASR and serve AudioTranscription.
Segment times are int64 nanoseconds, not seconds: the proto field is an
int64 that core/backend reads straight into a time.Duration, while the
runtime reports word offsets in milliseconds. Words are grouped into one
segment per consecutive speaker run, with the 1-based speaker tag carried
through and 0 (untagged) left unlabelled.
The whole RPC body runs inside withEngine so the family check and the C
calls happen under one acquisition of engineMu. Free runs without the
backend lock, so checking the family and then relocking would let a
teardown destroy the handle in the gap. The audio decode is inside the
closure too, which costs nothing: base.SingleThread already serialises
this backend's RPCs.
recognizeF32 guards zero-length PCM. &pcm[0] panics on an empty slice, so
Go never reaches the C side's own "empty audio" rejection, and a silent
clip or a truncated upload is ordinary input.
pkg/utils has no WAV decode helper, only the ffmpeg normalisation, so
audio.go pairs AudioToWav with go-audio the way parakeet-cpp does. It
returns the sample rate rather than a duration, since the C API resamples
off that number.
Also closes the write-side half of the race Task 5 fixed on the read
side: Load now holds engineMu across the family switch and the n.fam
commit, matching Free. The loaders still must not take it.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): implement streaming and live transcription
AudioTranscriptionStream drives a whole clip through the cache-aware
streaming API in 100 ms pushes, emitting each finalized utterance as a
delta and closing with the assembled result. AudioTranscriptionLive
serves the bidirectional RPC over the same session: config first, a ready
ack, deltas with word timings as utterances land, and a terminal result
when the caller closes its send side.
Both wrap their body in withEngine, so a stream holds engineMu for its
whole life and Free waits on it rather than destroying the recognizer
underneath a half-finished stream. That makes the way out load-bearing:
the file loop honours the request context between pushes, and the live
loop ends when the host closes the request channel, so a disconnected
client cannot pin the model against unload.
Only finals become deltas. The runtime applies punctuation and inverse
text normalization on finals only, so a final rewrites the utterance
rather than extending its interim, and delta on the wire is
newly-finalized text that consumers concatenate. Forwarding interims
would duplicate and mispunctuate every utterance.
The four streaming entry points sit behind an asrSession interface. No
NeMo GGUF is small enough to keep in the tree, so without that seam the
need-more-audio drain would have no test at all: nemo_speech_asr_stream_next
reports OK with a NULL handle when it wants more audio, which is a pause
rather than an end, and reading it either way round drops results or
spins forever.
Also folds in three items from the offline transcription review:
- empty audio is now refused before anything crosses the ABI, not
inside recognizeF32. The added integration spec caught the old
ordering panicking on an unbound entry point instead of failing;
- an undecodable sample rate is an error rather than 0, which this
runtime reads as "already at the model rate" and would have made a
wrong rate silently pitch-shift the audio;
- AudioTranscription guards its result pointer instead of relying on
an unstated invariant.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): make live deltas concatenate and fill segment words
runLive wrote the inter-utterance separator into the accumulated
transcript but emitted the delta without it, so a two-utterance turn sent
"one." and "two." while the terminal result read "one. two.". The live
consumer is the one that really concatenates: the realtime semantic-VAD
path joins the accumulated deltas with the empty string and clears them
only at a turn reset, never at an endpoint, so the running caption read
"one.two.". The separator now goes into the delta, as it already did on
the file path, and the terminal text is the verbatim concatenation rather
than a trimmed rebuild.
TranscriptSegment.Words was never populated, so a request asking for
timestamp_granularities ["word"] came back with no words at all even
though the timings were decoded. wordsToSegments now attaches them,
gated on the granularity the same way parakeet-cpp gates it, so a
transcript that did not ask for word timestamps does not pay for them.
Also: the final that comes back from the tail flush no longer claims an
end-of-utterance. It is the end of the stream, not a user yielding the
turn, and eou is what the realtime turn detector acts on.
The comment explaining why interims are suppressed led with the runtime's
postprocessing. The wire contract is the stronger reason and now comes
first: consumers concatenate deltas, so forwarding a growing hypothesis
assembles to "hehellhelloHello.". The postprocessing only explains why no
diffing trick would rescue them. It is also ITN and strip_formatting
rather than punctuation, which is off by default here.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): implement standalone diarization
loadDiarizer creates the Sortformer diarizer and Diarize serves the RPC
over a diarization stream: decode, chunked push, finish, then the
count-then-fill segments protocol.
nemo_speech_diar_segment carries start_time and end_time in SECONDS
already, not frame indices, so no conversion happens on the way to
DiarizeSegment.start/end and the model's seconds-per-frame is not
involved at all. The speaker label is the runtime's 1-based tag as a
decimal string, matching what wordsToSegments emits on the ASR path, so
the same speaker reads the same way whether a caller diarized a file or
transcribed it.
The six frame-geometry overrides are written as -1 rather than left
zero. c_api.cpp applies left_context_frames when it is >= 0 while every
other override needs > 0, so a zeroed config would silently pin the left
context to zero and change the model's streaming geometry.
nemo_speech_diar_segments writes *count before it rejects a buffer that
is too small, so a rejected fill still reports the size to retry with.
collectSegments uses that rather than truncating, bounded at four
attempts because the RPC holds engineMu for its whole body and an
unbounded retry would block an unload behind it.
Two DiarizeRequest knobs map onto the segmentation config, and the
proto and header names cross over: min_duration_on is the C
min_duration_sec and min_duration_off is the C min_gap_sec. Six fields
have no equivalent in this pipeline and are logged rather than dropped
in silence: num_speakers, min_speakers and max_speakers (Sortformer's
capacity is fixed by the checkpoint), clustering_threshold (there is no
clustering stage), include_text (no ASR here) and threads.
The empty-PCM guard fires before the stream is opened, so a silent clip
never reaches a purego entry point that would dereference &pcm[0].
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): pin the diarizer geometry sentinels and cap the segment buffer
The six frame-geometry overrides were written as -1 with nothing
asserting it. c_api.cpp applies left_context_frames at >= 0 while the
other five need > 0, so a dropped sentinel there pins the model's left
context to zero, and the struct keeps exactly the same shape, which is
all the layout assertions can see. Extracting diarModelConfig makes the
values assertable: five specs now pin all six frame fields, the device
index, the declared size and the NULL preset, each frame field on its
own line so a missing sentinel names itself.
distinctSpeakers had a spec with three segments over three distinct
labels, which len(segs) satisfies just as well as the real thing. Four
segments over three labels makes it a spec that can fail.
collectSegments sized its buffer straight from a count the C side
reported, and make() panics rather than erroring on a length it cannot
satisfy, so an uninitialised size_t coming back across the ABI killed
the backend process instead of failing one request. A ceiling of 2^22
segments, upwards of 93 hours of audio at one 80 ms frame each, turns
that into a diagnosable error.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): implement TTS and streaming TTS
The PCM callback is compiled once per process behind a sync.Once, not once per
request and not once per load. purego.NewCallback writes into a fixed table of
2000 entries (purego/syscall_sysv.go) and never releases one, so a per-request
callback panics the backend process on the 2001st synthesis, and a per-load one
reaches the same ceiling on a server that swaps models. Synthesis is routed
through that single callback plus a user_data id: engineMu is per-model, one
process holds several models, so a single current-sink pointer would be
overwritten by two TTS models synthesizing at once.
Deviations from the brief, all verified against the real headers and proto:
- TTS is TTS(*pb.TTSRequest) error and TTSStream is
TTSStream(*pb.TTSRequest, chan []byte) error, per pkg/grpc/interface.go.
The brief's context/pb.Result and server-stream forms do not implement the
interface. The channel is closed on every path, including the family
rejection, because pkg/grpc/server.go blocks on its drain goroutine and an
unclosed channel hangs the RPC with the backend lock held.
- The callback takes unsafe.Pointer, not uintptr. Converting a uintptr
parameter back to a pointer is a checkptr violation that aborts under
-race.
- resolveSpeaker refuses to turn a negative number into a speaker index. -1
is the C API's "use the default" sentinel, so the brief's rule would have
made a request naming an invalid voice synthesize in the default voice
instead of being rejected.
temperature and cfg_scale each write their override flag as well:
magpietts/runtime.cpp reads the float only when the flag is set, so a
temperature without it is silently discarded.
Also folds in Task 8's review finding on asr.go: the six bare -1 sentinels in
loadASR move to an asrDiarConfig builder reusing diarGeometryDefault, with
specs. src/asr/c_api.cpp applies left_context_frames at >= 0, so a dropped
sentinel pins the model geometry to 0 and no layout assertion can see it.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): surface NMT translation through Predict
nemo_speech_nmt_translate takes explicit source and target languages and has no
free-form generation or token-callback entry point, so there is no prompt in the
LLM sense. The pair comes from the source_language / target_language model
options, with an optional leading [src->tgt] directive as the only per-request
override, and PredictStream emits the whole translation as a single chunk
because the C API has nothing finer to give it.
Both RPCs wrap their body in withEngine so the family check and the C calls that
trust the handle share one acquisition of engineMu. PredictStream closes its
channel on every path, including the family rejection: this is the legacy
streaming contract, and pkg/grpc/server.go blocks on a drain goroutine that only
finishes when the channel closes, so leaving it open hangs the RPC rather than
failing it.
nmtTranslatorConfig is extracted so its four adjacent pointer fields can be
asserted against distinct sentinels. Transposing two of them changes neither the
struct size nor any field offset, so the layout assertions cannot see it.
Also removes goString, which had no production caller: every string-returning
symbol in abi.go is bound with a Go string return that purego converts itself.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* test(nemo-speech-cpp): pin the three-segment pair tag in an NMT directive
The directive regex allowed an unbounded run of two-letter segments per side, but
nothing tested it: narrowing that run back to a single optional segment left every
spec green. resolve_tag accepts a ready pair tag in one field with the other empty
(src/nmt/langpairs.cc), and those tags run to three segments (en-zh-cn, pt-br-en),
so a shorter pattern does not mis-split the tag, it fails to match the directive at
all and the whole bracket is handed to the model as text to translate.
The justification on the regex was also wrong and is corrected: pt-br and zh-cn are
two segments and parse either way. It is the single-field form that needs the run.
Renames the NMT handle to n.nmt so it stops sharing a name with the translator
interface, following n.synth, which is shortened for the same reason.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(nemo-speech-cpp): register the backend and give its specs a CI job
Registers nemo-speech-cpp across every surface .agents/adding-backends.md
requires, and adds the CI job its unit suite never had.
backend/index.yaml gets the meta backend (capabilities map, no uri), a
development meta and 12 image entries. No amd and no intel capability keys:
upstream NeMo-Speech.cpp builds ggml with CUDA, Vulkan or Metal only, and
SystemState.Capability falls back to "default", so those hosts get the CPU
build rather than a tag that does not exist. The nvidia-cuda-* and
nvidia-l4t-cuda-* keys are present because getSystemCapabilities() refines an
NVIDIA host to them whenever the CUDA directory exists; without them every
modern CUDA host and Jetson would miss the map and quietly run on CPU.
.github/backend-matrix.yml gets 7 include rows and 1 includeDarwin row. No
hipblas and no sycl rows, for the same upstream reason. cpu and vulkan are
per-arch pairs sharing a tag-suffix so backend-merge-jobs builds a multi-arch
manifest: an ARM host with no NVIDIA GPU reports "default" and the Jetson image
does not cover it.
The CI job is the substantive part. make test-extra is dead on master, because
prepare-test-extra depends on a protogen-python target that does not exist and
no workflow invokes it anyway, so the entry added earlier in this series ran
nowhere. abi_test.go asserts the size and field offsets of every Go mirror
struct against the C ABI it is dlopened into, and those assertions are the only
defence against silent memory corruption after a purego symbol rename or an
upstream header change. tests-nemo-speech-cpp in test-extra.yml now executes
them on pull_request and on master, gated on the backend's own path filter.
The recipe sets NEMO_SPEECH_REQUIRE_LIBS=1, so a missing library fails rather
than skips. WITH_NORM=OFF skips the OpenFST leg and costs no coverage: nothing
in the four C ABI headers is conditional on it, so the layouts are identical.
Also registers the upstream pin with the bump bot, which the backend Makefile
already claimed but was never wired up, and adds the BackendCapabilities entry
so a hand-written model config gets a real usecase surface. PossibleUsecases is
the union of the four families and DefaultUsecases is transcript alone, the
audio-cpp pattern. No VoiceCloning key: MagpieTTS synthesizes from baked
speaker ids, not a reference clip.
No gallery entries: publishing converted GGUFs is a follow-up.
ModelIdentity needs no work in this backend. main.go serves through
grpc.StartServer, so every RPC lands on pkg/grpc's shared server wrapper first,
and checkModelIdentity is the first statement of all seven handlers this
backend implements. A second check inside NemoSpeech would be unreachable and
would risk diverging from the cross-language sentinel the router matches on.
AudioTranscriptionLive stays unguarded because TranscriptLiveRequest carries no
ModelIdentity field at all, which is a proto-level gap affecting every backend
and needs its own change.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): build the CUDA-13 Jetson image the l4t-cuda-13 key needs
The nvidia-l4t-cuda-13 capability pointed at nvidia-l4t-arm64-nemo-speech-cpp,
which is built on nvcr.io/nvidia/l4t-jetpack:r36.4.0 and therefore links ggml
against CUDA 12. A Jetson whose CUDA 13 runtime is present reports that
capability and would have pulled an image with no libcudart.so.12 to dlopen,
failing hard at load. That is worse than omitting the key: with no key
Capability() falls back to "default" and the host gets a working CPU build.
Fixed the way parakeet-cpp and moss-transcribe-cpp already do it, by shipping
the second L4T image rather than dropping the key. Nothing prevents building it
here: those peers use plain ubuntu:24.04 on ubuntu-24.04-arm with the same
Dockerfile.golang as this backend's other rows, and every package in the
nemo-speech-cpp apt gate exists on noble arm64.
Adds the -nvidia-l4t-cuda-13-arm64-nemo-speech-cpp matrix row and its two index
entries, repoints the key on both metas, and rewrites the capability-map comment,
which had the reasoning backwards.
Also adds the documentary inferBackendPath branch, matching all six sibling
*-cpp Go backends. Behaviour is unchanged; the generic golang fallthrough
already resolved this backend correctly.
The previous commit message said "all seven handlers" of the shared gRPC
wrapper. There are eight RPC entry points: seven are guarded by
checkModelIdentity and AudioTranscriptionLive is the unguarded eighth, which
that message already called out separately. Wording only.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(nemo-speech-cpp): document the backend and list it in the importer
Adds docs/content/features/nemo-speech-cpp.md, alongside the audio.cpp page
that is its closest sibling, and cross-links it from the speech-to-text,
diarization, text-to-speech, backend-type and compatibility-table pages so the
backend is reachable from every surface that lists its modalities.
The page covers the architecture-to-family table, every option key with a model
YAML per family, the translation prefix directive, the acceleration matrix, and
the four limitations this backend ships with: Linux-only inverse text
normalization, suppressed interim streaming results, the library's default
translation context and generation limits, and the absence of gallery entries.
knownPrefOnlyBackends gains the backend so it appears in the /import-model
dropdown. It stays preference-only and AutoDetect=false: general.architecture
lives inside the GGUF where no remote-repo probe can read it, and a translation
model carries an ordinary LLM architecture with no NeMo-specific marker.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* docs(nemo-speech-cpp): correct the translation limits, the macOS gap and the TTS conversion
Three factual errors found in review, all of them the kind a user would act on.
The translation limits were described backwards. Input longer than the 1024-token
context is rejected, not truncated: translator.cpp throws "nmt: prompt too long
(N tokens) for context 1024", which reaches the caller as a failed request. What
is silently cut is the output, by the max_new_tokens loop at 256. The bullet now
separates the two and says which one fails quietly.
The macOS gap covers TTS text normalization as well. Both directions sit behind
the single NEMO_SPEECH_WITH_NORM flag, which the Makefile forces off on Darwin,
so tn_dir is as inert there as itn_dir. Neither fails the load: both warn and
carry on. pnc_model really is unaffected, since punctuation is compiled in
unconditionally. The tn_dir row in the option reference gained the caveat the
itn_dir row already had.
The TTS conversion procedure produced a model that could not load. It converted
MagpieTTS and stopped, leaving no NanoCodec, which the same page lists as
required; following it gave "no NanoCodec GGUF found next to ...". Both halves
are now there, each with the download that feeds it, so the block runs top to
bottom on a clean machine.
Also: any negative gpu value pins TTS to the CPU, not only -1, and FLAG_CHAT
additionally surfaces the model in the web UI chat picker.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): map every C status, not just the NMT one
INVALID_ARGUMENT was translated to codes.InvalidArgument at exactly one of
sixteen C call sites. Everywhere else a non-zero status collapsed to
codes.Internal, so the same backend answered an unsupported language pair with
HTTP 400 and an unknown TTS voice, which is the same class of caller mistake
against the same process, with HTTP 500. Status 4 is CANCELLED on the ASR and
TTS surfaces and was reported as a backend failure rather than as the consumer
having stopped listening.
asr.h, tts.h and nmt.h each declare their own status enum and diar.h reuses the
ASR one; the values they share agree, and the single divergence is that NMT
declares no CANCELLED because nemo_speech_nmt_translate has no callback for a
consumer to stop with. That is an absence, not a disagreement, so one table
serves all three. status.go carries it, with the header line numbers and a note
that a pin bump has to recheck it: purego binds by name and the status crosses
as a bare int32, so nothing in the build or the linker can see a drift.
New specs cover the whole enum, unknown values, and one real INVALID_ARGUMENT
per family driven through the shared objects rather than through the Go mapping
asserting against itself.
Also add UsecaseChat to this backend's capability entry, which the docs already
told operators to set for translation models. chat is a gallery filter key and
completion is not, so GET /api/backends/usecases would have greyed the Chat
filter out and hidden a Riva-Translate gallery entry from the one filter that
fits it. The flag gates no endpoint; it makes the model eligible as the default
chat model and puts it in the web UI chat picker, both of which Predict and
PredictStream already serve.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): audit the gosec unsafe and file-inclusion sites
gosec flags 13 alerts on this backend: one G304 and twelve G103. Each was
checked individually rather than blanket-suppressed, and each annotation
states what makes that particular site safe.
The G304 at audio.go is a false positive. The opened path is
filepath.Join of a directory the function just created with os.MkdirTemp
and a constant basename; the request-controlled path is the input to
AudioToWav and never reaches the open.
The twelve G103 sites are the package's three established shapes, and
every one was verified against them: cstr and pinPtr take the address of
something pinned on the line above and return it one-way (nothing in the
package converts either result back, which is what keeps checkptr out of
it under -race), and each *Create hands C a stack-local POD config whose
uintptr members are cstr allocations or pinPtr addresses held by a pinner
the loader unpins only after the call. The two slice-building sites are
bounded by construction: DiarSegments is handed exactly len(buf) with the
buffer sized under maxDiarSegments and a reported count larger than it
rejected rather than sliced to, and the TTS callback copies out a slice
whose length is the length the runtime declared for that buffer.
Separately, sampleRateOf gets a real fix rather than an annotation.
go-audio reads the WAV header's sample rate from an unsigned 32-bit field
into an int, so a header claiming more than 2^31-1 passed the "> 0" test
and then narrowed to a NEGATIVE rate, which the runtime would take as a
resampling ratio. AudioToWav cannot produce one today, but that is a
property of another package and this function exists precisely because
the rate is read back rather than assumed, so the bound is enforced here
and pinned by a spec.
The four remaining integer narrowings are annotated with the bound that
makes each safe: the WAV payload length is already checked against
maxWAVDataBytes, the speaker count is bounded by maxDiarSegments, and the
two segment ids are the proto's own int32 wire type.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): skip the CUDA-only ggml patch series on darwin
The macOS backend build died in patch-ggml:
scripts/apply-ggml-patches.sh: line 56: mapfile: command not found
make[1]: *** [patch-ggml] Error 127
mapfile is a bash 4 builtin (and its -d flag needs 4.4). macOS ships bash
3.2.57 as /bin/bash and GitHub's runner images add no newer one, so the
bare `bash` the recipe resolves from PATH cannot run upstream's script.
Rather than hunt for a capable bash that the runner does not have, drop
the step where it does nothing. ggml-patches/ is a CUDA series: every
kernel it adds is under src/ggml-cuda/, and its whole footprint outside
that directory is an op enum plus prototype in include/ggml.h, the
constructor and a name-table entry in src/ggml.c, and two ggml-cpu lines
that make the CUDA-only op report unsupported and abort. Nothing it
touches is compiled into a Metal kernel or changes a CPU one.
The project's own references to patch-only ggml symbols sit behind
NEMO_SPEECH_FUSED_RELPOS_ATTN and NEMO_SPEECH_FASTCONFORMER_CUDA_FUSIONS,
which cmake already forces OFF without GGML_CUDA, or behind
NEMO_SPEECH_GGML_PATCHED itself, which guards a GGML_TENSOR_FLAG_Q8_PLANAR
write that a non-CUDA buffer throws before reaching. So passing
NEMO_SPEECH_GGML_PATCHED=OFF costs the Metal build nothing, and it is
required once the series is skipped: that flag is what stops the ASR
sources referencing a tensor flag stock ggml does not define.
This is upstream's own Metal configuration. Its metal-* and vulkan-*
CMake presets inherit the cpu-* ones, which set NEMO_SPEECH_GGML_PATCHED
to OFF; docker/Dockerfile and scripts/windows/build.ps1 do the same for
their non-CUDA targets. LocalAI's Makefile never passed the flag at all
and so inherited the CUDA default everywhere.
Linux is untouched and keeps applying the series, including its
idempotency and its hard failure on a patch that does not apply. The gate
is the same uname test the WITH_NORM block above already uses, and both
branches keep the order-only clone prerequisite, which on a WITH_NORM=OFF
tree is the only thing that pulls sources/ in.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): restore std::binary_function for MeCab on libc++
NEMO_SPEECH_TTS_WITH_JA=ON compiles Open JTalk's bundled MeCab, and
mecab/src/dictionary.cpp derives a comparator from std::binary_function,
which C++17 removed. libstdc++ still ships it as deprecated-but-present
under -std=gnu++17, so Linux never notices. libc++ compiles it out and
the macOS arm64 build dies with "no template named 'binary_function' in
namespace 'std'".
This is ours, not an upstream regression: upstream defaults both
NEMO_SPEECH_TTS_WITH_JA and NEMO_SPEECH_TTS_WITH_ZH to OFF and the OSS
drop carries no CI at all, so that target is never built there. Upstream
does already carry the equivalent workaround for MSVC's STL
(_HAS_AUTO_PTR_ETC plus /FIfunctional) but has no libc++ branch.
libc++ gates the two templates on
_LIBCPP_ENABLE_CXX17_REMOVED_UNARY_BINARY_FUNCTION, and has since LLVM
16, older than any clang Xcode still ships. The name is the whole
problem: _LIBCPP_ENABLE_CXX17_REMOVED_BINDERS covers bind1st, bind2nd,
ptr_fun and mem_fun and not unary_function or binary_function, and the
umbrella _LIBCPP_ENABLE_CXX17_REMOVED_FEATURES no longer exists in
libcxx at all. A wrong name preprocesses fine and fixes nothing.
Applied through CMAKE_CXX_FLAGS rather than to the one target, because
the tokenizer CMakeLists is upstream's and sources/ is a pinned
checkout. Project-wide is also the safer scope: the macro decides
whether libc++'s internal __binary_function alias resolves to
std::binary_function or to __binary_function_keep_layout_base, a base
class of std::less and friends, so defining it for a subset of
translation units would give those class templates two spellings in one
binary. Both bases are empty and, at C++17, carry identical members, so
the define changes no layout and no ABI.
Darwin only. On Linux the branch is unreachable and the macro is not a
name libstdc++ knows, so it would be inert even if taken; a Linux
configure with the flag forced on puts it on all 23 C++ TUs of
nemo_speech_openjtalk_frontend including dictionary.cpp at -std=gnu++17,
and on none of the 16 C TUs.
Mandarin needs nothing: cppjieba v5.6.7 and limonp have no removed C++17
constructs left (limonp replaced std::not1 and std::bind2nd with
lambdas) and cppjieba's own CI builds macos-14 and macos-latest at C++11
through C++20.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): repair OpenFST's FstImpl::operator= for gcc-14
The first WITH_NORM=ON build failed compiling fst_normalizer.cpp against the
installed OpenFST 1.8.3 headers:
fst.h:690:59: error: no match for 'operator=' (operand types are
'std::unique_ptr<fst::SymbolTable, ...>' and 'fst::SymbolTable*')
FstImpl's copy-assignment operator assigns the raw pointer returned by
SymbolTable::Copy() straight to a std::unique_ptr member. No C++ standard
allows that, so the line is ill-formed everywhere; it survived because nothing
instantiates FstImpl::operator= and gcc up to 13 only checks a template
member's body when it is instantiated. gcc 14 resolves non-dependent operator
expressions at template definition time, so it rejects the line in any
translation unit that includes <fst/fst.h>. The CI diagnostic confirms the
phase: it reads "In member function", not "In instantiation of", and carries
no instantiation backtrace.
That is why this surfaces only here. build_itn_deps.sh compiles OpenFST with
gcc-12 and upstream's own images build the runtime with gcc-13, so neither
compiler reaches the check; backend/Dockerfile.golang installs gcc-14 and
promotes it with update-alternatives, and fst_normalizer.cpp is the one
translation unit in this backend that includes OpenFST.
Fix it in the installed ITN prefix, which is the only copy the cmake build
compiles against, using the same .reset() spelling FstImpl::SetInputSymbols
already uses for the identical operation. libfst.so is linked before this runs
and cannot contain the function, since no compiler could ever have emitted it,
so there is no ABI or ODR consequence. The rule is guarded on both sides so a
pin bump to a fixed OpenFST fails loudly rather than silently no-opping.
Verified with a real gcc 14.2: the CI error reproduces byte for byte from a
file whose entire content is '#include <fst/fst.h>', and gcc 14 reports
exactly two errors over the whole OpenFST include closure this backend uses,
both of them these two lines. After the patch that closure compiles clean
under gcc-14 with the target's own flags. The step is reachable only under
WITH_NORM=ON, so 'make -n stage-libs WITH_NORM=OFF' mentions neither it nor
the ITN build, and darwin, which defaults WITH_NORM to OFF, never evaluates it.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* fix(nemo-speech-cpp): install cmake 3.31 on bases that ship less than 3.26
The JetPack r36.4.0 row dies on the first line of NeMo-Speech.cpp's
CMakeLists.txt:
CMake Error at CMakeLists.txt:3 (cmake_minimum_required):
-- Configuring incomplete, errors occurred!
Upstream opens with cmake_minimum_required(VERSION 3.26). That base image is
Ubuntu 22.04 jammy, whose apt cmake is 3.22.1, so configure aborts before it
reads a single one of the backend's -D flags. Every other Linux row in this
block is noble, which ships 3.28 and clears the bar, so the failure is one
base image wide rather than a code problem. Everything before it on that row
had already worked, including the OpenFST and Sparrowhawk ITN build.
No other Go backend needs this. parakeet-cpp and moss-transcribe-cpp share
the same JetPack base and both declare cmake_minimum_required(VERSION 3.18),
and nothing in the repo installs a cmake newer than the distro's, so there is
no existing pattern to reuse. Nothing depends on jammy's cmake staying 3.22
either: build_itn_deps.sh never invokes cmake at all, since OpenFST and
Sparrowhawk are autotools builds.
Kitware's release tarball rather than their APT repo or pip. The tarball is a
pinned URL with a published checksum, so an upstream release cannot change
what lands here. The APT repo does carry jammy arm64, but it serves a moving
latest that today is CMake 4.4, and 4.x drops compatibility with
cmake_minimum_required below 3.5, which vendored third_party subprojects
still declare; pinning it there would mean tracking Kitware's Debian revision
string instead of an upstream version. pip would drag a Python toolchain into
a backend that has none. 3.31.12 is the last 3.x release, so it clears 3.26
while keeping the CMake 3 policy surface, and it stays close to the 3.28 the
green noble rows already use. The binaries need only glibc 2.17 and carry no
libstdc++ DT_NEEDED, well under jammy's 2.35. doc/, man/, ccmake and cmake-gui
are not extracted; the final image is FROM scratch, but there is no reason to
page 100 MB of Qt GUI and docs through the CI cache.
Gated on the installed cmake actually being older than 3.26, so the rows that
already build green keep configuring with exactly the cmake they use today,
and folded into the existing ${BACKEND} block rather than added as a new
instruction, so no other Go backend image gains a layer and nothing above the
Vulkan SDK, CUDA, Go and protoc layers moves.
The symlink lands in /usr/local/bin and shadows apt's cmake. Unlike the protoc
shadowing that broke Sparrowhawk earlier in this series that is inert: protoc
has to agree with the libprotobuf headers it generates against, whereas cmake
links nothing into the product and has no ABI relationship with anything in
the image, and it resolves the symlink back to /opt to find its own Modules/
tree, so a 3.31 binary can never read 3.22's modules.
The version test avoids $(...) deliberately. BuildKit delivers a RUN heredoc
through an outer shell with an unquoted delimiter, so a command substitution
runs there, too early, in a container where the files it reads do not exist
yet, and its empty output is pasted into the script; the first draft took the
install branch on every row because of it.
Verified by building the block against nvcr.io/nvidia/l4t-jetpack:r36.4.0
arm64 under qemu, the row's actual base image: cmake 3.22.1 detected, tarball
checksum verified, 3.31.12 installed, and a cmake_minimum_required(VERSION
3.26) project configures with -G Ninja and builds, with CMAKE_ROOT resolving
to /opt/cmake/share/cmake-3.31. Same on ubuntu:22.04 amd64 and arm64.
ubuntu:24.04 skips the install, gains no /opt/cmake and still configures on
/usr/share/cmake-3.28. The NeMo-Speech.cpp compile itself on JetPack CUDA 12
is not reproducible here and remains for CI.
Assisted-by: Claude Code:claude-opus-5[1m]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
8845ccebf7 |
feat(gallery): default to index.localai.io with GitHub as a mirror (#11409)
The primary is a caching mirror of the same two files, so an install resolves its gallery from infrastructure the project controls. The GitHub URI stays as a mirror, so behaviour is unchanged whenever the primary is unreachable. Docs and agent guides that either state the shipped defaults or hand out a copy-pasteable gallery list are updated to match, so following them no longer silently demotes an install off the new primary. Assisted-by: Claude:claude-opus-5 [go vet] [go test] [golangci-lint] Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
a77780ad14 |
feat(gallery): fall back to mirrors and a cached index when the primary source fails (#11389)
* feat(version): include OS and arch in the outbound User-Agent
Registries and galleries already receive LocalAI/<version>; adding the
platform follows ordinary client convention and discloses nothing a
registry cannot infer from the manifest it is asked for.
Updates the User-Agent note in docs/content/getting-started/models.md,
which documented the old format.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* feat(downloader): identify LocalAI on outbound requests
pkg/oci has always sent a User-Agent; the downloader sent none, so gallery
reads, model-file downloads, resume probes, content-length probes and the
HuggingFace safety scan all went out as a bare Go HTTP client, unattributable
to LocalAI by the hosts serving them.
HuggingFaceScan moves off the client's Get shorthand to an explicit request
for the same reason — the shorthand gives no place to hang a header.
Extends the User-Agent note in docs/content/getting-started/models.md, which
claimed the header was sent only to Ollama and OCI registries.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* feat(gallery): add a mirrors list to gallery configuration
Mirrors are an availability fallback, tried in order only after the primary
URL fails. omitempty keeps existing configurations byte-identical.
The slice makes config.Gallery non-comparable with ==, which broke the two
slices.Equal callers in the runtime settings registry. Replace them with an
explicit Gallery.Equal / GalleriesEqual so a gallery list that differs from
the baseline only by its mirrors still counts as env/CLI-set. Equal compares
the Verification block by value; == compared it by pointer identity, which
called two structurally identical policies different.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* fix(downloader): treat an HTTP error status as a failed read
ReadWithCallback handed the response body to its callback whatever the
status was, so a 404 page or a 502 from a CDN arrived as if it were a
gallery index or a model config: it parsed to nothing, got cached for an
hour, and no caller could tell the source had been down. DownloadFile has
always checked the status; this path never did.
Mirror fallback depends on it — a source that answers with an error page
has to count as unreachable, or the next candidate is never tried.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* feat(gallery): fall back to mirrors when the primary source fails
Candidates are tried primary-first with a bounded timeout each, and a
source that just failed is skipped for a cooldown so a dead host is not
re-dialled on every listing. When every candidate is in cooldown they are
all tried anyway: refusing to serve a gallery we might be able to reach is
worse than one slow request.
The one-hour index cache is untouched and stays keyed on the gallery's own
identity, so a mirror-served fetch fills the entry the primary would have.
No SSRF validation is applied to the candidates. validateGalleryConfigURL
guards GetGalleryConfigFromURL because that URL arrives in a request body;
mirrors come from the operator's gallery configuration, the same place the
primary has always come from, and the index fetch has never validated the
primary. Validating mirrors while the primary goes unchecked would buy
nothing and would break the deployment mirrors exist for — an index served
from a host on the LAN.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* fix(gallery): loosen the mirror fetch timeout and stop blaming the caller
The downloader only ever bounded response headers, never the body, so the
per-attempt deadline added with mirror fallback was the first whole-transfer
timeout this path has had. At 30s the default 2.2 MB index demanded ~75 KB/s
sustained: a rural-DSL, mobile or satellite user who used to wait 60s and
succeed would now fail, and then eat a 10-minute cooldown on a source that
was perfectly healthy. Raised to 120s (~19 KB/s), which no link that could
go on to download a model will miss, and made it a var so a test can shorten
it and prove a hanging candidate is actually abandoned.
Caller cancellation is no longer recorded as a failure of the source.
Unreachable today since getGalleryElements passes context.Background(), but
once a request context is wired through, a browser disconnect would have
blackholed every candidate for ten minutes over something the sources had
no part in.
Also document that mirrors do not cover a .ref gallery URL: the reference is
resolved before mirrors are considered, so a .ref that cannot be fetched
fails the gallery outright. Routing .ref resolution through the candidate
list needs a per-candidate resolve-and-fetch and a decision about cache
identity, which is more than this change should carry.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* feat(gallery): serve the last known good index when everything is offline
A successful fetch is cached alongside the models directory and served when
no source is reachable, so an offline or airgapped machine can still list
its gallery. Entries may be stale in that state, and the fallback is logged.
The copy is deliberately kept out of the models directory, where a <name>.yaml
file is read as an installed model's configuration, and is named after a digest
of the gallery URL so the model and backend galleries cannot collide. Writing
it is best effort: a read-only or full disk must not fail a fetch that
otherwise succeeded.
Also corrects the mirror scheme list in the docs: the HuggingFace prefixes are
huggingface://, hf:// and hf.co/, not huggingface:.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* fix(gallery): only cache a response that is really a gallery index
The last known good copy was written on any 2xx, before anything looked
at the bytes: the parse only happens later, in getGalleryElements. A
captive portal, a corporate proxy or a CDN error page all answer HTTP 200
with HTML, so any of them could overwrite a good copy. The listing fails
then and there, and the next offline start — the one case this cache
exists for — serves the interception page instead of the gallery it
already had.
Probe the body before persisting it: unmarshal into a []any and keep the
older copy unless the result is a non-empty sequence. An empty document
is rejected too. It parses fine, so a parse-only check would still let a
blank response replace a populated index with one that lists nothing,
which from the user's side is the same outage; and an empty index is
worth nothing offline, so there is no case where caching it beats keeping
what came before. The live body is still returned to the caller — the
probe gates persistence only, and getGalleryElements remains the thing
that reports a real parse failure.
Also in this pass:
- The empty-basePath guard only caught exact "". galleryCachePath(".")
and galleryCachePath("models") still resolved the cache sibling against
the process working directory, which is what the guard was written to
prevent. Reject any non-absolute base.
- The docs claimed the offline cache "applies to every gallery, with or
without mirrors". Not true for a .ref URL: the reference is resolved
before the cache is consulted, so a .ref gallery fails offline even
after a successful earlier fetch, and the cache file it writes can
never be read. Extend the .ref warning and qualify the sentence.
- pkg/oci's UserAgent comment never mentioned the platform component
added earlier on this branch.
- resetGalleryFailures and expireGalleryFailure had no non-test callers;
move them into the test file.
- The all-candidates-failed error reported len(attempt), so a three
mirror gallery with two sources in cooldown said "all 1 source(s)
failed" — which reads as a misconfiguration. Report how many were
configured and how many were skipped.
- Give the package's tests their own TMPDIR. The cache is a sibling of
the models directory, which is right in production, but specs that
build a models directory directly under /tmp made the sibling resolve
to /tmp/cache and left it behind after every run.
Assisted-by: Claude:claude-opus-5 [go vet] [go test]
* fix(gallery): convert the new tests to Ginkgo and clear the lint gate
.agents/coding-style.md requires Ginkgo v2 + Gomega for every Go test and
has forbidigo enforce it; the stdlib-style tests still in the tree are tech
debt, not a pattern. Every test file this branch added was written in the
forbidden style, which is what turned CI red.
Convert all five of them. internal had no suite bootstrap, so add one;
core/config, core/gallery and pkg/downloader already have theirs and are
reused, so no package mixes styles. pkg/downloader/useragent_test.go and
read_status_test.go were not in CI's forbidigo list but used the same
forbidden calls, so they are converted too.
The one conversion with a trap in it is core/gallery. Go's t.TempDir()
yields $TMPDIR/<TestName>NNNN/001, so the gallery cache — a sibling of the
models directory — was isolated per test. GinkgoT().TempDir() yields a flat
$TMPDIR/ginkgoNNNN, which would put every spec's cache in one shared
directory and break the specs that count files in it. tempModelsDir()
restores the original isolation.
Also make the deliberate cleanup-path ignores explicit with `_ =`, drop the
gallery cache directory to 0750 (nothing outside the server's own user and
group reads it), and justify the cache read with a #nosec G304 comment in
the form already used elsewhere in the tree: the path is a hex sha256 under
a fixed directory with a non-absolute base already rejected, so no
caller-supplied text reaches it.
Re-ran the mutations these specs were verified against — dropping the
platform suffix from UserAgent, making Gallery.Equal ignore Mirrors and
ignore Name, removing persistGalleryIndex's validity probe, removing the
!filepath.IsAbs guard, not skipping a cooled-down candidate, and dropping
the per-attempt timeout. All seven still fail the converted specs.
Assisted-by: Claude:claude-opus-5 [go vet] [go test] [golangci-lint] [gosec]
---------
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Richard Palethorpe
|
5ac445e1d4 |
fix(react-ui): restore 3D Studio results and history (#11393)
* fix(react-ui): restore 3D Studio results and history Keep large conditioning-image payloads out of the rendered request panel so the generated viewer can mount reliably. Accept clipboard images and synchronize 3D history consumers so new results appear in Studio without a reload. Cover clipboard input, bounded request rendering, result display, and cross-view history synchronization with Playwright. Assisted-by: Codex:gpt-5 Playwright * perf(react-ui): idle the 3D viewport when still Limit auto-rotate rendering to 30 FPS and stop scheduling frames when rotation is disabled. Resize, view controls, and pointer input invalidate the still frame on demand. Assisted-by: Codex:gpt-5 Playwright |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
ea438cdeaf |
feat(vllm-cpp): wire the full engine config surface through engine_args (#11159)
The backend could configure four of the engine's knobs (block size, KV block
count, max sequence length, max concurrent sequences) out of a config surface
that is considerably larger. Speculative decoding, prefix caching, the
chunked-prefill token budget, the scheduling policy and the external KV
connector were reachable from vllm.cpp's own HTTP server and from nothing
LocalAI could write in a model config.
Config now goes through `engine_args:`, the same map the vLLM and SGLang
backends take, with keys spelled as vLLM's own CLI flags so a speculative_config
or kv_transfer_config block written for vLLM works verbatim. The legacy
`options:` list keeps working and reads every key too; engine_args wins where
both set one. Unknown keys are logged and ignored rather than fatal: the field
is shared with the other engines, so a config carrying their knobs must not take
the model down.
Two details worth knowing:
`enable_prefix_caching: false` maps to the ABI tri-state force-OFF (2), not 0.
0 means "let the model capability decide" and dense architectures default the
cache on, so collapsing the two would silently enable it against an explicit
false. enable_jump_forward (ABI v10) shares the encoding, deferring to
VT_ENABLE_JUMP_FORWARD instead of to the model.
The importer probes config.json on a vllm-cpp import and writes
speculative_config: {method: mtp} when the checkpoint declares an MTP head, the
safetensors analogue of the llama-cpp importer's GGUF probe. DFlash draft repos
are refused with a warning instead, since a drafter cannot serve alone and the
pairing is not derivable from either repo. The draft path is resolved against
LocalAI's model directory, because the engine only looks in a directory holding
config.json or in the HF cache and never downloads: the repo-id spelling the
vLLM docs teach used to die deep in the load with "draft checkpoint not found".
docs/content/features/text-generation.md gains a vllm.cpp section covering the
engine_args table, all three speculative methods, LMCache and the legacy list.
The backend had no documentation page before.
This replaces a branch that had gone stale behind master and carried its own
route to ABI v10, which #11386 has since landed in minimal form. Rebased onto
that as a single commit rather than replaying the intermediate steps, whose
ABI v9 mirrors no longer make sense against master's pin. The Darwin build
fixes for Apple Clang's gnu-folding-constant diagnostic on C++, Objective-C and
Objective-C++, originally authored by localai-org-maint-bot, are folded in here.
Verified: `make abi-check` agrees at v10; unit specs, core/config and
core/gallery/importers green; and the full e2e passes in 1330s against a CPU
libvllm.so reporting ABI v10 with Qwen_Qwen3.5-0.8B-Q4_K_M.gguf (load, blocking
completion, streaming, chat and tool calls).
Assisted-by: Claude:claude-fable-5 golangci-lint
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
a05a790021 |
fix(ci): emit verifiable backend signature bundles (#11366)
Cosign v2.4.1 does not select the Sigstore bundle format by default, while LocalAI's verifier only consumes OCI bundle referrers. Request the format explicitly for both registries and guard the producer contract with a shell regression test. Document strict backend integrity configuration and release-tag identities for operators. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
9f62401fca |
feat(traces): show in-flight API requests (#11368)
Register JSON API exchanges before their handlers run so the traces dashboard can surface active work. Replace the live entry with the completed persisted record under the same ID, and clean it up if a handler panics. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
133c546c3f |
feat(api): add text moderation endpoint (#11316)
* feat(api): add text moderation endpoint Add an OpenAI-compatible /v1/moderations endpoint backed by constrained local text generation. Register its auth and discovery surfaces, document the text-only MVP, and cover response shaping and access control. Assisted-by: Codex:gpt-5 * test(mcp): update assistant client stub Keep the LocalAI Assistant holder test stub aligned with the scheduling methods added to LocalAIClient so repository-wide type checking succeeds.\n\nAssisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Peteandlocalai-org-maint-bot
|
8a68f3571c |
feat(api): add POST /v1/images/upscale endpoint (#10227)
* feat(api): add POST /v1/images/upscale endpoint Add a new image upscaling endpoint that accepts a source image and returns an upscaled version. Supports selectable upscaler models (e.g. realesrgan) and a configurable scale factor (2x or 4x). - backend.proto: add UpscaleImage RPC and UpscaleImageRequest message - pkg/grpc: implement UpscaleImage in Backend interface, client, server and embed shim - core/backend/upscale.go: new backend helper (mirrors ImageGeneration) - core/http/endpoints/openai/upscale.go: new multipart/form-data handler - core/http/routes/openai.go: register POST /v1/images/upscale - core/http/auth/features.go: gate upscale routes under FeatureImages - backend/python/diffusers/backend.py: implement UpscaleImage — uses diffusers upscale pipeline when loaded, falls back to Lanczos resize * fix(grpc): add UpscaleImage stub to Base backend All Go backends embedding Base now satisfy the AIModel interface without needing to implement UpscaleImage explicitly. * fix(images): complete upscale endpoint integration Store generated upscales under the served images directory, validate scale factors, document and advertise the endpoint, and add a functional Stable Diffusion x4 gallery model. Assisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
fd4ec083b9 |
feat(downloads): add resume-safe pause action (#11222)
Give gallery operations distinct pause and cancel paths. Pause preserves partial download data so reinstalling the same model or backend resumes through HTTP Range, while cancel keeps its destructive semantics. Surface the action in the Activity UI and document the API behavior. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
58ea2f5d79 |
feat(ui): give Operate and Studio a front door, and fix two layout regressions (#11305)
* feat(ui): give Operate a front door and fold six rail groups into four Opening Operate ran firstVisiblePath() and landed on Backends, because Backends happens to be written first in operateConsole.groups. The section that should answer "is anything wrong" opened on a package manager, and nothing was reported until you visited it. Adds /app/operate. Its one irreplaceable block is "Needs attention", which is empty when nothing is wrong and says so in a line rather than rendering a reassuring green panel. It collects stale backends, failed operations and unhealthy nodes. Everything else on the page is a summary you could already assemble by visiting four others. The rail regroups from six headings to four: Inference and Activity were both "the runtime right now", Access and System were both administration. No destination is removed and no gate changes, so isConsoleItemVisible and consolePaths are untouched. Overview leads the first group, which is what makes firstVisiblePath() return it without knowing it exists. Rail entries now carry a signal beside the label. This does not replace the sidebar badge and is not built as if it does: the badge stays on the always-visible sidebar entry for the reason recorded in Sidebar.jsx, that the rail exists only on Operate routes and can be collapsed. The signals are orientation while inside Operate, so they are aria-hidden and nothing urgent depends on them alone. OperateSummaryContext polls once for the whole console, following OperationsContext, which exists because per-consumer setInterval against one endpoint was the defect it fixed. It is mounted by ConsoleLayout for the Operate console only, so "poll only while in Operate" needs no route check. Built on usePolling, so it pauses on a hidden tab. Operations are read from OperationsContext rather than polled a second time, and each source degrades to no-signal on its own so one dead endpoint cannot blank the rest. It reads the cached GET /api/backends/upgrades and never the POST that forces a real registry check. Traces and Usage get no signal yet: /api/traces returns the list, so a count would mean fetching every trace to render one number. A counts endpoint is the honest fix and is scoped separately. Full e2e suite green (369 passed, 4 skipped), including a render-smoke entry for the new route. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * feat(ui): open Studio on what this machine can actually make Studio was a tab strip over six generators that opened on Images, which was never a decision, only the first entry in BASE_TABS. Nothing said which modalities this installation could run, so the way to learn that video had no model was to pick the tab and find an empty select. Adds an overview tab and makes it the fallback. Explicit tabs still win, so existing deep links keep working; anything unrecognised or gated now lands on the overview rather than Images. Each tab carries a dot: filled when an installed model advertises that modality, hollow when nothing serves it. That is the feature in one detail, turning the strip from navigation into a report of what the machine can do before anything is clicked. The dot is aria-hidden because the overview states the same facts in words and the dots change as models load. Two kinds of unavailable, which had to stop looking alike: - switched off, via a permission: no tab and no lane, unchanged - available with no model: a lane, and a route to installing one Studio now owns one MODALITIES table so the tab strip and the overview cannot disagree about what exists, and calls useModels() once, unfiltered, grouping in the browser. useModels(capability) fetches the whole list and filters locally, so a hook per modality would have been six identical requests to /api/models/capabilities on every mount. There is a test for that. Recent outputs read every localStorage store through a new readAllMediaHistory(), which avoids mounting five hooks that carry save timers the overview has no use for. 3D is read separately through use3DHistory rather than folded in: its entries are GLB blobs in IndexedDB, so they cannot come from the same synchronous read. Typical cost is the median of this machine's own history, not a guess, and renders as a dash when there is nothing to go on. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * fix(ui): stop the stat cards and the console rail breaking on small screens Two unrelated causes behind one report that /app/manage looks wrong when the window is narrow. The stat cards were being laid out by the wrong rule at every width. Two different components both claimed `.stat-grid`: the dashboard card strip that holds .stat-card children, and the detail-pane StatGrid the split views introduced further down App.css. Being later, the second won every shared property, so the cards got its 120px columns and its 1px hairline gap in place of their own 180px columns and spacing-md. Four cards were packed onto a row that fits two, labels wrapped to three lines and clipped, and the icon crowded the value. Renamed the strip to `.stat-cards`, after the children it actually holds, which also removes the mismatch of a `.stat-grid` container full of `.stat-card`s. The split-view component keeps `.stat-grid` and its BEM parts. The expanded console rail had no bounded height. Thirteen destinations stacked in one column is taller than a phone, so opening the menu pushed the page's own heading past the fold: the menu replaced the page rather than annotating it. Capped at 55vh with internal scrolling below 768px, so the content behind stays reachable. Both are asserted on behaviour rather than markup: no stat-card label may be clipped, the card gap must not be the detail pane's hairline, and expanding the rail must leave the page heading on screen. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * feat(ui): retemper the palette to localai.io and add the lane primitive The token half of the style transfer, plus the shared list idiom the two overviews had each grown their own copy of. theme.css moves from Nord to the website's palette, variable names preserved so every consumer moves with it: ground #13171f -> #0d1117, accent frost cyan #88c0d0 -> action blue #4f8cff, success sage -> mint #56d6a4, warning -> the amber #f1b95d the site spends only on the thing asking for a decision. Eyebrows go mint. Dividers become an opaque #29384a hairline rather than alpha over a varying surface, which is what makes stacked surfaces read crisply on the site. Light is derived, not inverted. The site ships one theme and never had to answer this, but the app does: blue darkens to #2f62d8, mint to #0d8b60 and amber to #8a5d0b, all clearing 4.5:1 on a cool paper ground, where the dark-mode values sit near 2:1. Same three roles, different values. Three files restate the palette because CSS variables cannot reach them: cmTheme.js (the whole CodeMirror theme), VoiceVisualizer and WaveformPlayer (canvas). Left alone they would have quietly kept the app half-Nord. The `.lane` primitive replaces the near-identical row CSS that OperateOverview and StudioOverview had each written: a full-bleed row on a hairline that insets on hover, with no card and no shadow. Callers supply only the column template. Both pages now use it, along with `.lane-head` for section rhythm and a `.page-pad` container for top-level pages outside a console shell — without which Studio sat flush against the sidebar with its eyebrow clipped. Studio's tab strip wraps rather than running off the edge at narrow widths. Full e2e suite: 386 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * feat(ui): put Home's resident models on lanes and give the footer one line Home's status line was three chips saying a thing was true. It now reports figures: how many models are resident, how many nodes are healthy, what share of memory is in use, set in tabular monospace so the digits line up. A chip answers whether; a figure answers how much, which is what someone opening the page at a glance is after. Resident models move from status chips to lanes, with the id set in a new `.lane__name--id` because an id is something you might type or paste and the UI face makes it read as a label. /api/system-information carries only the id, so there is deliberately no backend or memory column: inventing one would mean a server change this does not make. The footer was three centred rows and cost the bottom sixth of every page for chrome. It is one line now, version left and links right, wrapping to centred when the viewport is too narrow to hold both. Every link it had, it keeps. Full e2e suite: 392 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * fix(ui): correct three contrast failures and stop a guaranteed-404 poll A contrast audit of the new palette found three values below WCAG AA, one of which the previous commit message claimed was fine: - White on the #4f8cff button is 3.22:1, which is large-text only. The website does exactly this, but a button label in an app is not large text, so the label goes to dark ink at 5.88:1. Light mode keeps white, which is 5.44:1 on its darker blue. - Light-mode success was 4.08:1 on paper, not the 4.5 claimed. Darkened to #0a734f, 5.56:1. - Nord red was already 4.28:1 on raised surfaces, a pre-existing miss carried over unexamined. Lifted to #c96f78, 5.02:1. Lanes gain the two states they were missing: a 44px target on coarse pointers, matching what EntityRail already does so the two list idioms feel the same under a thumb, and a reduced-motion variant that keeps the background feedback while dropping the hover inset, which is a position change. The Operate summary no longer asks for /api/nodes on a single-node install. The cluster API answers 503 when distributed mode is off, so it was a guaranteed miss every fifteen seconds; it is now gated on useDistributedMode, the same condition the rail already uses for the Nodes entry. Full e2e suite: 392 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * fix(ui): restore the gap between overview blocks, and stop claiming zero nodes Two defects a design review surfaced. `.lane-head:first-child { margin-top: 0 }` was meant to stop the first block on a page carrying a top margin. But every <section> makes its lane-head a first child, so the reset applied to all of them and the gap between blocks vanished: "Sections" sat flush against the attention row above it. The header supplies its own bottom margin, so a uniform top margin is correct everywhere. The Cluster summary read "0 nodes" on a single-node install, which looks like a fault when the cluster API is simply switched off. It now says "Single node". Full e2e suite: 392 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * feat(ui): open dark by default, and stop clipping the collapsed sidebar footer Dark is the identity rather than a preference: localai.io ships one theme and it is this one, so an install should look like LocalAI before anyone has chosen anything. The OS setting no longer selects light on first load. The toggle still does, and a stored choice wins forever after, which the tests assert both ways. The collapsed sidebar footer stacked its controls but kept the expanded row's inline padding, so their edges were clipped against the 51px rail. Full e2e suite: 394 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * feat(api): count traces server-side and give the Operate overview real totals The overview's headline block had no source. /api/traces returns the trace list, so "37 errors in 24h" meant fetching every buffered exchange to count it in the browser — waste that grows with the buffer, to produce three integers. Adds GET /api/traces/summary: totals, failures, p95 and a bucketed series for sparklines, over a window that defaults to 24 hours and is capped at a week. Deliberate calls, each with a spec: - A 4xx is the caller getting it wrong, not the installation being unhealthy, so only 5xx and transport errors count as failures. - p95 is a nearest-rank percentile rather than the slowest request, which is what a max would report and what makes latency panels lie. - Buckets are oldest-first so a sparkline reads left to right, and the slice is never nil: nil serialises as null and breaks .map() on the other side, which is a silent runtime error rather than an empty chart. - Exchanges outside the window are not counted at all. The route is registered before /api/traces/:id so "summary" is not captured as a trace ID. On the client, Traces and Usage gain the rail signals they were shipped without, the Observability section summary now states counts instead of listing its destinations, and an installation that has served nothing says so rather than showing three zeroes dressed as telemetry. Sparkline is a bare stroke with an emphasised endpoint and no axes: the figure above it already states the value, so its only job is the shape. Go: 185 middleware specs pass. Full e2e suite: 396 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * fix(ui): stop the memory chart calling a trade-off an error The VRAM-by-context chart rendered any build over the limit in error red, and escalated the verdict to the error tone as soon as two context sizes crossed it. But an over-limit build still installs — #11288 keeps a test on exactly that — so red overstates what is happening. A model that fits at 32k and not 64k is a trade-off, not a fault. Over-limit bars and the limit line now use the warning tone, which is the constraint colour used everywhere else in this branch: know what you are doing, not you may not. The error tone is reserved for "fits nowhere", where the model genuinely cannot run on this host. Full e2e suite: 397 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * feat(ui): give the new surfaces orchestrated motion Uses the reveal system already in the codebase rather than adding a library: pageReveal, .reveal-stagger and staggerStyle() were built for exactly this, and anime.js would be ~17KB duplicating four lines of CSS for list reveals. The overview's headline figures, attention rows and section lanes stagger in, as do Studio's modality lanes and recent outputs, so a page assembles in the order it is read instead of appearing all at once. Two additions beyond stagger. Rail signals transition on opacity when a poll lands, so a number changing reads as an update rather than a jump cut, and it stays on the compositor so it cannot reflow the rail. The attention block animates its left edge in — the one thing on the page that should announce itself, and on the border rather than the text so nothing moves under a reader. Both are dropped entirely under prefers-reduced-motion, alongside the lane hover inset already handled. Full e2e suite: 397 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * feat(ui): put the generators on a hairline field stack and record the request The workbench treatment from the mocks, applied where it costs least: both changes land on shared surfaces, so all six generators get them at once rather than drifting apart page by page. The control column stops being a shadowed card of boxed groups and becomes a hairline field stack — the panel is the page's left half, not an object floating on it — with uppercase micro-labels matching the eyebrow treatment used elsewhere. Because .media-controls is shared, Images, Video, 3D, Speech, Sound and Audio FX all move together. RequestPanel shows the request the form actually built, with a copy-as-curl. LocalAI is API-first and Studio is the best place in the app to teach its own endpoints: the form stops being a black box, and a result worth keeping can be reproduced from a shell without reverse-engineering which fields the page sent. It records what was sent rather than what the form currently holds, and renders nothing until a request has been made — a panel describing a request nobody made is a tutorial, not a record. Wired into Images and Speech. Full e2e suite: 401 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * feat(ui): make Chat a transcript instead of a bubble thread Rounded, filled, asymmetric bubbles fight a system built on hairlines, and they carry the speaker in shape and side rather than in words. The assistant side had already given up its bubble; this finishes the job. Both roles now run full width down one column, separated by a rule, each with a mono role label. The user turn keeps a left edge in the action tone so the two are still told apart at a glance, without a fill or a corner radius. The avatars go: the accent and the label carry the speaker, so the glyph was decoration once neither side had a bubble. Saying who is speaking in words rather than in geometry is also what survives being read aloud, printed, or looked at by someone who cannot pick the sides apart by colour. Full e2e suite: 404 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * feat(ui): dress the API reference in LocalAI's palette The Swagger page was the last surface still shipping in someone else's colours, which is conspicuous now that everything it links from is dark. Swagger UI has no theming hook, so rather than fork it we serve our own index ahead of the library's wildcard and restate the palette over its stylesheet. The library's own bundle and assets are still what load, so a swagger-ui upgrade cannot silently break the page — this is a skin, not a fork. Two things needed real care. Swagger tints the entire operation row per method via .opblock.opblock-post and friends, so the palette had to match that specificity rather than reach for !important; the method now lives on one edge instead of washing across the row, because a page where every row is a status colour has no status colour left. And the filled method chip put white on pale green, which was the least readable thing on the page — it is an outlined mono chip now, carrying the method in its border and text. Palette values are copied from theme.css rather than referenced: this page is served by Go and never sees the app's CSS. The comment says so, and says to keep them in step. Go: routes and middleware suites pass. Full e2e suite: 405 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] * fix(ui): make tall split-view pages reachable, repair the Agents header, scale titles Three things found by actually using the app rather than measuring it. **Host was unusable.** The shell above a split view is overflow:hidden so the document cannot grow, which left anything taller than the viewport simply unreachable — and Host stacks a resources card, four stat cards and a tab bar above its split, so the bottom of the pane fell off at every window height with nothing to scroll. Every sweep I ran for this was horizontal, which is why it kept coming back clean. The page now scrolls inside the pinned shell. The pane keeps its own scroller: letting it grow instead pushes the document taller and stretches the rail to match, which is the regression e2e/discover-height.spec.js exists to catch, and which the first version of this fix duly caused. **The Agents header controls were unstyled** — "Create Agent" was rendering with the browser's default chrome. The markup had been mangled at some point: six unrelated classes merged into one string on the link, and the label and button left with none at all and empty icons. Repaired, with the inline flex replaced by a shared .header-actions class. **Page titles take the editorial scale from the site**: larger, tracked at -0.04em, on a line height near 1, so a two-word title reads as a statement rather than a label. The typeface is unchanged — DESIGN.md keeps the existing type system — so the whole difference is scale, tracking and leading, which is where the site gets its voice from. This was the biggest reason the running app still did not look like the mocks. Full e2e suite: 404 passed, 4 skipped. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Edit] [Bash] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
74b7ea2829 |
feat(ui): replace the gallery and inventory tables with a rail and a detail pane (#11288)
* feat(ui): rename the Install Models nav entry to Discover "Install Models" named the action rather than the destination, and it was the only multi-word entry in a rail of one-word ones (Home, Chat, Studio, Talk, Build, Operate). A bare "Models" was the obvious fix but it collides with the installed-models view under Host, which is a different page for a different job. "Discover" keeps the rhythm and says what the page is for. The icon moves from a download arrow to a compass for the same reason: the page is browsed before it is installed from. Translated in all seven locales rather than left to fall back, so a locale switch does not leave the entry in English. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * feat(ui): replace the gallery table with a rail and a detail pane The eight-column table was not the real problem; the click-to-expand row underneath it was. Variants, files and a VRAM estimate never fitted inside a <tr>, so they were pushed into a drawer that could hold one model at a time, could not be linked to, and had no room to say anything useful. The gallery is now a rail to scan and a pane that answers. The pane has two states and no third: with nothing selected it is the discovery page, and with a model selected it is that model's detail. Selection lives in the URL, so a model is linkable and Back steps out of the detail instead of off the page. The rail groups by capability while browsing and flattens to results the moment a term is typed. That is a rule rather than a toggle: once someone has said what they are looking for, the buckets are between them and the answer, and making the user choose would be handing them our problem. The detail pane plots VRAM against context length with the host's own limit drawn across it. This is new information, not a restyle. A single number invites "so will it run?", and the honest answer is usually "yes, up to a 32k context", which is a shape rather than a number. The estimates were already fetched for every context size, so it costs no new request. Backends that take no context length say so instead of being given a meaningless chart, and a host with no GPU gets no chart at all rather than bars with nothing to compare against. The split-button variant menu goes with the actions column. The pane lists every build with its backend, quantization, size, fit and a details disclosure, each installable, which is what the dropdown was a cramped substitute for. Its tests move onto that list; the three contracts it alone carried (fetch-once caching, the loading state, an unfit build staying installable) are backfilled against the pane. RecommendedModels moves inside the pane, where it has the width to argue for a model instead of listing one, and keeps its own dismissal and collapse. Rail entries carry no description. Two lines is the budget and the second is better spent on whether the thing will run; the stripped-Markdown contract moves to the pane's lede, tooltip included. e2e: 123 passing across models-gallery, navigation, recommended-panel, model-artifact-operation, operations-strip and page-render-smoke. Inline styles in Models.jsx drop from 82 to 41. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * refactor(ui): extract the split view into shared components Discover shipped its rail, pane and detail header as private functions inside Models.jsx. Backends and Host have the same defect and want the same shape, so leaving them there guarantees three rails that drift. SplitView, EntityRail, DetailHeader and StatGrid now live under components/split/. EntityRail is deliberately data-driven: a surface maps its own entity onto { id, name, icon, meta, stripe, groupId } and keeps its vocabulary to itself, which is what stops the rail learning about models, backends and loaded state all at once. The CSS moves with it. What was .discover__rail is .entity-rail, .discover__ pane is .split-view__pane and so on, because a class named after one page is a lie on the next two. Only what is genuinely Discover's stays behind the old prefix: the shelves, the hero and the VRAM-by-context chart. Two additions the shared rail needs and Discover did not: a state stripe, for surfaces read by condition before they are read by name, and an empty label. Discover passes neither. No behaviour change. e2e 100 passing across models-gallery, navigation and models-recommended-panel. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * feat(ui): put the backend gallery on the split view Same defect as the model gallery, so the same shape: a seven-column table over a click-to-expand row that was the only place the repository, licence, tags and links could go. The rail groups backends by the use case they serve, sharing Discover's taxonomy on purpose: a backend is the runtime a use case needs, so "vision" ought to mean the same thing one level down. It flattens on a query for the same reason it does on Discover. The zero state is the one real departure. A backend's fitness is not free memory, it is the accelerator and platform it was built for, so the pane leads with what this host is, then what is not installed yet, then whether anything installed has gone stale. The table listed 37 runtimes and left "which of these can even run here" entirely to the reader. Distribution moves into the pane, which is the one thing a row could never carry: which nodes hold a copy and which do not, with the install-on-more control next to it rather than squeezed against a chip. The distributed and target-node action logic is unchanged, including the guard that keeps a hardware-specific build off the fan-out path. The split-button popover loses its per-row anchoring because there are no rows; one pane, one anchor. Selection lives in ?backend=, preserving the ?target= scope rather than clobbering it. e2e: 139 passing across models-gallery, navigation, backends-management, models-recommended-panel, nodes-per-node-backend-actions, page-render-smoke, operations-strip and model-artifact-operation. The backends spec gains six split-view tests; its three description-cell tests move onto the pane lede. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * feat(ui): put the Host inventory on the split view The last of the three surfaces, and the one that is not a catalog. Both tabs had the same click-to-expand row, so the shell transfers; what does not transfer is the zero state, because there is nothing to discover in your own inventory. With nothing selected the pane reports what is happening: how many models are loaded, what failed, what has an update, and which models are holding VRAM right now. Every number was already on the page. None of them had been assembled into one statement, so "what is going on" was a question the tabs could not answer however long you looked at them. The rail buckets by state rather than capability - Running, Idle, Disabled for models; Update available, Installed for backends - which is the opposite of the galleries and deliberately so: nobody opens Host wondering which of their models does vision. Entries carry a state stripe for the same reason. Load and Stop are promoted out of the kebab, because that is what an operator came for; the rest stays behind the menu rather than diluting it. Adopted, pinned and alias badges follow the model into the pane: they are facts about the thing, not about its state, and the rail line is spent on state. Deliberately NOT done: folding the two tabs into one rail, as the mock had it. It costs five URL parameters, the manage-tab localStorage key and the stat-card shortcuts, all of which are live deep-links today. The tabs stay as the group selector; merging them is a follow-up with its own migration. e2e: full suite 355 passing. New host-split-view spec; alias-template, manage-logs-link, manage-action-menu-position and model-editor-back-nav move off `.table` and the row kebab onto the rail and the pane. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * polish(ui): accessibility and consistency pass over the three split views Findings from a pass over what the previous four commits actually shipped, rather than what they were supposed to. The rail was not a listbox. ARIA lets a listbox contain options and groups, and nothing else, but each group's collapse control is a button that has to sit inside the scroller with the entries it folds. It is now a labelled group of buttons, which is the honest description; selection is announced with aria-current and the arrow keys are unaffected. Every entry was its own tab stop, so tabbing past a forty-entry rail to reach the pane took forty keystrokes. Roving tabindex makes the rail one stop, and arrowing now moves focus with the selection instead of leaving it behind on an entry Tab can no longer reach. The rail rounds its corners with overflow:hidden, which was clipping the focus ring off the first and last entries entirely. Inset outlines fix it. A 30px row is fine under a mouse and too small under a thumb, so coarse pointers get a 44px target without costing density on a desktop. One slot said three different things: "9 models loaded" on Discover, "12 loaded" on Backends, "3 of 9" on Host. All three lists are a page of a larger set, so all three now say so the same way. Also removed: an emptyLabel prop on EntityRail that nothing passed, its dead CSS rule, and MODELS_COLSPAN and ResourceRowDesc, which died with the tables. e2e: full suite 355 passing. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(ui): correct three defects only a real gallery exposed Running the branch against a live instance with 1,595 models and 1,017 backends, rather than against mocked fixtures, surfaced three things the e2e suite could not. Grouping did nothing. The rails matched on the use-case keys the filter chips send (`chat`, `tts`, `transcript`), but those are a server-side vocabulary the handler maps onto entries. What entries actually carry is free-form and inconsistent: models come back tagged `llm`, `gguf`, `vision`, `coding`, and backends `LLM`, `text-to-text`, `audio-transcription`. Nothing matched, so every model landed in "Everything else" and the feature was decorative. Grouping now lives in utils/entityGroups.js, shared by both galleries, matching case-insensitively against the vocabulary the API really uses, with the entry's backend as a fallback signal - a backend named `whisper` is a speech backend whatever its tags say. Order is specific before general and that is load-bearing: a vision model is tagged `llm` too, so testing text first would swallow it. The zero state claimed GPU memory on a machine with no GPU. The resources endpoint reports system RAM in the same field when gpu_count is 0, so the hero read "84.4 GB of GPU memory" next to the recommendations panel correctly saying "No GPU detected". The number was never wrong, only its label; it now says system memory unless a GPU is actually present. The page title still said "Install Models" under a nav entry saying Discover. Also: the keyboard test named the model it expected to arrive at, which made it a hostage of the grouping table and broke the moment the buckets were fixed. It now asserts that the selection moves and returns. e2e: full suite 355 passing. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(ui): the filters and the rail were fighting over the same job Four things you find odd on Discover, and they turn out to be one mistake seen from four sides. The rail grouped the current page. The listing is paginated at nine rows, so those bucket headers described nine entries out of 1,595, and turning a page reshuffled the sections under the reader. The structure was never stable because it was computed over the wrong set. The chips were redundant for the same reason, seen from the other side. They send tag= and filter all 1,595 server-side. The rail grouped nine of them client-side by the same axis. Two controls for one job, and the weaker one was the one this branch added, so it goes. Grouping stays only on Host, where the list is complete, local, and bucketed by state rather than capability. The search bar felt odd because it sat in a full-width band while the thing it narrowed was a 290px rail below and to the left. The whole band now lives in the rail column: search, backend, use cases, refinements, then the list it narrows. One column to say what you want, one to show what you got. Nineteen chips do not fit at that width, so they fold into a disclosure that states the selection. A disclosure and not a popover, deliberately: picking use cases is multi-select and interleaves with the backend select and the toggles below, and a popover dismisses itself the moment you touch either. The header held two counts and two buttons at arm's length from all of it. The counts were the third statement of the same number on one screen, after the rail's "9 of 1,247" and the pane's own headline, so they go. The buttons move into the pane's zero state, which is the surface that answers "what do I do here". Also: the two first-run empty states wore .loading-center, which is display:flex in the default row direction because it exists to centre one spinner. With four children that put the icon, the heading, the sentence and the buttons on a single line with no gap. They are now a proper full-height empty state. e2e: full suite 353 passing. Grouping tests are replaced by ones asserting the rail stays flat; chip tests open the disclosure first; two filter-layout tests that asserted the old three-band arrangement now assert the column. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * polish(ui): make Discover a full-height view, group the chips, name the refinements Four things, all of them the same complaint: the page read as a document with controls scattered on it rather than as one view. The header is fused. A title block with its own padding, a subtitle and two counts made the split view look like an attachment to a document that happened to sit below it. It is now a slim bar carrying the title, the count and the two page-level actions, and the split fills the rest of the window. Rail and pane scroll independently, so the filters and the pane's headline stay put while a long list moves under them. The chips group. Nineteen in a flat row is a lot to scan even behind a disclosure, and they already belong to the four families the rest of the UI speaks, so they are bucketed by those. "All" sits on its own above them without a heading, because it is a reset rather than a use case. The refinements stop looking dumped. When the band became a column they were three controls left where they landed; they now read as a named section with one control per row. The zero state suggests again. It had decayed into a "Browsing / 9 of 1,247 / select a model" line that restated the count for the third time on one screen. It now offers the four use cases as tiles that set the filter, which is the shelf idea from the mock without inventing curation or paying for a second fetch. Two bugs found by looking at it rather than at the tests: the disclosure was clamped to 190px, which cut it off partway through its third section so two of the five never appeared at all; and the creation actions rendered twice, once in the new bar and once in the pane hero a few pixels away. e2e: full suite 353 passing. The chip-row test now holds its contract across the per-family rows rather than a single one, and additionally asserts every family is present and non-empty. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(ui): pin the split view's height so a long detail scrolls the pane Selecting a model with a long description grew the whole page and dragged the rail down with it, which is the opposite of what "full height" was supposed to buy. The flex chain was right and the ceiling was missing. .app-layout and .main-content are min-height:100dvh, which is a floor: flex distributes free space but nothing caps growth, so a pane taller than the viewport expanded the column, the document scrolled, and the rail stretched to match. height:100% on the pane then resolved against an auto-height parent and did nothing. The chat route already solves this by pinning .main-content to 100dvh. The same treatment now applies to any route containing a .page--app, selected with :has() so the shell does not have to learn which pages happen to be split views. Below the stacking breakpoint the pin is lifted, because two stacked halves in two short scrollers is worse than a page that scrolls. Measured on a live instance: document height stays at the viewport across selection (950px either side) and the pane overflows internally instead. Adds discover-height.spec.js, which asserts the page height and the rail height are unchanged by selection and that the pane is the thing that scrolls. The existing specs could not have caught this: they mock short descriptions, and the bug only appears when the pane has more content than the viewport holds. e2e: full suite 355 passing. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * feat(ui): give Backends and Host the full-height view, and fix the Update button Backends now matches Discover: the header fuses into a slim bar carrying the title, the count and the page-level actions, the filters move into the rail column where they narrow the rail and nothing else, and the split fills the window. Its seven chips fit at rail width, so unlike Discover's nineteen they need no disclosure. Host gets the bar and the height; its resource monitor, summary cards and tabs stay above the split, because those are read once while the rail and the pane are worked in. Two things the height change surfaced. The console layout is a flex row with align-items:flex-start, so its body sizes to content. Right for the pages it was built for, wrong for a split view, which needs a ceiling to scroll inside: without it the Backends rail ran past the viewport and over the footer. Pinned with :has() so only split-view routes are affected. The filters vanished when nothing matched. Both galleries swapped the whole shell for an empty state, which took the search box and the chips with it, so the page said "try adjusting your search or filters" while offering neither. The shell now stays and the empty state moves into the pane. Also fixes the Update control on Host, which had no className at all and rendered as bare text, next to a status span that had picked up btn classes and two copies of `fas` and so rendered as a button you cannot press. They have swapped appearances back. e2e: full suite 355 passing. The render-smoke selector learns .view-bar__title, since the pages it checks no longer all use PageHeader. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(ui): keep the view mounted while searching, and bring rail grouping back Searching replaced the whole view with a loader. The search box lives in the rail column, so every debounced refetch unmounted the field being typed into and dropped its focus with it. The list, the filters and the pane went too. The shell now stays and the rail says it is busy: a sweep bar under its header and the stale list dimmed, so the eye knows the answer is being replaced without losing its place. A cold start still gets the skeleton, because there is nothing to keep. The condition for that is "nothing has loaded yet", not "the list is empty". Those differ exactly when someone is editing a query that matched nothing, and getting it wrong there would unmount the view on the keystroke after a no-results search - the worst possible moment. Grouping comes back on both galleries. It was removed because nine rows could not fill five buckets, so a page turn rebuilt the rail's whole structure. That was a symptom of the page size rather than of grouping: the rail now asks for 30 rows instead of 9 (Backends 60 instead of 21), which is enough for the sections to read as structure and turns five times fewer pages. The order of the sections is fixed, so what changes between pages is membership, not arrangement. Grouped while browsing, flat while searching, as before: once a term is typed the buckets stand between the reader and the answer. Also gives GalleryLoader a class and a testid instead of six inline style declarations on a bare div, which is why nothing could select it. e2e: full suite 359 passing, including a new spec asserting the search box keeps its focus and its value across a refetch, and that a cold start still shows the skeleton. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * perf(gallery): stop invalidating the VRAM estimate caches on every request Searching or turning a page felt slow. It was not the search and not the listing: /api/models answers in 3-9ms. It was the VRAM estimate, which the gallery asks for once per row, and which took ~2.3s every single time however often the same model was asked about. pkg/vram already caches what makes that expensive - the remote content-length probes, the GGUF metadata reads and the HF repo sizes. Those caches key on a gallery generation counter, and AvailableGalleryModelsCached triggered a background refresh on every call, with each refresh bumping the counter. One page view is one listing request plus thirty estimate requests, each of which re-read the gallery and started another refresh, so the generation moved constantly and every cache entry was stale before it could ever be read. The caches were dead in production. Three changes, each doing one thing: A refresh interval. The cached list is still served immediately; this only decides how often re-fetching from upstream is worth starting. Five minutes, as a package variable so tests can drive it without waiting. A generation bump only when the gallery actually changed. An unchanged gallery re-fetched on schedule must not throw away work that is still valid, which is the difference between an estimate costing nothing and costing a network round trip. A separate "loaded" flag. The cache engaged on `cached != nil`, so a gallery that legitimately holds nothing read as never-loaded and took the blocking path on every call, bumping the generation each time. Found by the test for the interval, which could not pass while this was true. Measured against a live instance with 1,595 models: one estimate, repeated 2.3s -> 2ms a page of 30, in parallel 10s -> 0.04s A first, genuinely unseen model still costs its remote probe. That is inherent; what changed is that it is now paid once per model per gallery version rather than once per request. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * perf(gallery): warm VRAM estimates at startup, and stop the UI waiting on them Two halves of the same complaint: the gallery stalls on VRAM estimation. Server side, the estimates are now warmed in the background at startup. Estimating an entry nobody has asked about costs a remote probe of its weight files, and the gallery needs one per row, so the first visitor was paying for the whole page. The warm-up walks the gallery in the order the UI lists it, so the first page is ready before anyone reaches it. It is bounded and it never blocks: 300 entries at 4 at a time by default, on its own goroutine, stopping with the server's context. Warming the whole gallery would be thousands of probes on every boot, which is rude to the upstream and slow to finish; warming nothing leaves the first page paying two seconds a row. Anything past the limit still warms itself on first view. LOCALAI_VRAM_WARM_LIMIT=0 turns it off for an air-gapped host, LOCALAI_VRAM_WARM_CONCURRENCY=1 slows it for a metered link. Client side, the page no longer waits on estimates it does not need yet. It fired one request per row at once; a browser allows about six connections per host, so thirty estimates took every slot and the request behind a click - the variant list, an install - queued behind work nobody asked for. That is the freeze: the list was already usable, and the UI was busy fetching sizes. Four at a time leaves room for the interactive request to overtake, and a row whose estimate is still in flight says "sizing…" rather than leaving a blank where a number will appear. buildEstimateInput moves to core/gallery as EstimateInput, since the handler and the warmer both need it. Measured against 1,595 models, from a cold boot: page 1, 30 estimates in parallel 10s -> 0.04s full warm-up (299 of 300 entries) 3m, in the background Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * chore: untrack data/.local_user_id and ignore the runtime data dir `local-ai run` writes its instance state under ./data when started from the repo root, which is exactly what a contributor testing a build does. The identity file ended up committed on this branch by a `git add -A` while verifying the gallery changes against a live instance. Anchored, so it matches the runtime directory at the repo root and not a `data` directory nested inside some package. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
|
d0119bf62c |
feat(chat): local-ai chat is now a terminal agent (#11291)
* chore(deps): bump cogito to v0.11 ahead of the nib harness nib is the agent harness that becomes 'local-ai chat'. It requires cogito v0.11, so pull that bump forward on its own: minimal version selection would apply it to LocalAI anyway, and both repos use cogito and cogito/clients. Landing it separately keeps the harness change reviewable. nib itself is not pinned yet. Nothing in LocalAI imports it, and 'go mod tidy' runs as a goreleaser before-hook in CI, so an unimported require line does not survive. It lands with its first importer. No LocalAI call site needed a change. Both cogito.WithMaxAttempts callers guard the argument above zero, so v0.11's new clamp is unreachable, and LocalAI's Multimedia values implement only URL(), so v0.11's new TypedMultimedia routing treats them as images exactly as v0.10 did. Binary size (cmd/local-ai): 200,301,381 -> 200,336,045 bytes (+34,664). A throwaway probe that links nib measured 201,042,243 bytes (+740,862 over the pre-change baseline). Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(chat): resolve and seed the agent state directory Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): write the agent config atomically and tighten its modes Replacing config.yaml in place truncated it first, so an interrupted write would have destroyed the api_key nib keeps in the same file. Stage through a sibling temp file and rename over the target instead, and match nib's 0700 directory mode. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(chat): probe the endpoint and classify failures Probe lists what a LocalAI endpoint advertises and separates the two failures that need different advice: nothing listening, and rejected credentials. go-openai reports a rejected key as one of two concrete types depending on the error body, and both occur against a real LocalAI. The normal error handler sends an OpenAI error envelope, which arrives as *openai.APIError; the opaque-errors handler replies with a bare status and no body, which arrives as *openai.RequestError. Classifying on only one of them misses half the cases, so the status is read from either. A cancelled probe is not reported as an unreachable server, because it learned nothing about the endpoint, and neither is a reply that could not be parsed, because something did answer. Both would otherwise send the user off to start a server that may already be running. The model list is returned verbatim and in server order. LocalAI lists whatever it finds in the models directory, including stray archives and dotfiles, and deciding which advertised ids are real belongs to whoever presents them. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(chat): resolve the model from flag, config, or the server Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(chat): pin that model resolution sorts a copy of the caller's slice The sort spec asserted only on what the chooser was offered, so replacing the defensive copy with an in-place sort of req.Available still passed all 37 specs. Assert the input slice's order after the call, so the guarantee cannot be dropped silently by a later refactor. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(chat): offer to start a server when none is reachable Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): bound the server wait and pin readiness and stop semantics Set cmd.WaitDelay so a backend subprocess holding the child's stderr pipe cannot block cmd.Wait forever, which would leave exited unclosed, burn the whole shutdown grace on a clean exit, and leak the waiter goroutine. Two test gaps closed alongside it: the readiness spec now counts polls, so treating 503 as ready is observable, and Stop's single-interrupt contract is pinned by giving StartedServer interrupt/kill hooks that a spec can count. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(chat): drive Stop through one process interface, hide exec plumbing Two independent interrupt/kill func fields plus a nil check admitted wirings no test could distinguish: the pair swapped, so a SIGKILL would strand the backends SIGINT exists to let local-ai run clean up, or kill left nil, so a wedged server never escalates. One two-method interface that *os.Process already satisfies leaves nothing to swap and nothing to nil. Also translate exec.ErrWaitDelay, whose text names an os/exec struct field, into what the user can act on. os/exec only substitutes that sentinel when the process exited without an error of its own, so no exit status is swallowed. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(chat): replace the REPL with the built-in agent local-ai chat is now the nib agent harness compiled into the binary: tool use behind an approval gate, sub-agents, MCP, plugins, and skills, all auto-configured against the local server. The REPL goes with it. Its model listing and its 401 classifier were duplicates of the ones Probe now owns, and the classifier was the version that misreads a bare 401 with no OpenAI error envelope, so keeping either would leave the package with two divergent answers to the same question. github.com/mudler/nib lands in go.mod in this commit rather than earlier: go mod tidy runs as a goreleaser before-hook on every PR, so a require line with no importer is stripped before it reaches CI. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(chat): split the pre-agent phase out of Run and pin it Everything before the handoff is testable and nothing after it is: once app.Run owns the terminal there is no seam left. prepare draws that line, takes interactivity as a parameter so the prompts can be driven over a pipe, and hands Run the state dir, the model, and any server it started. The questions move onto one prompter that owns its buffered reader. A fresh bufio.Reader per question reads ahead and discards what it buffered, so the model choice typed behind an answer to "start a server?" was lost and the next question saw EOF. choose answers with a list index and refuses an empty offer, so a value that was never on the list cannot reach ResolveModel, which persists it and starts every later run against it. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): bound each server check with a deadline Nothing bounded the model listing, so pointing chat at an address that accepts the connection and then never replies left the user with no output and no offer to start a server. The budget is context.WithTimeout rather than a cancel plus a timer. Probe deliberately refuses to call an endpoint unreachable on a context.Canceled, since a caller who gave up learned nothing about the server, and only honours a deadline. A cancel-based budget therefore expires as the one error that suppresses ErrUnreachable, exactly for the hung servers the offer exists to rescue. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): tell the user when their model choice cannot be saved The choice is meant to be asked for once. When saving it fails the user is silently asked again on the next run, and the only trace was an xlog.Warn: the agent runs at log level error, and a --log-level=error run swallows it entirely. ModelRequest gains Notify for exactly this class of problem, one that is worth telling the user about but not worth failing over, and the chat wiring points it at the same writer the question was asked on. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): stop a session's server when the process is signalled A server started for the session is stopped by a deferred call, and a signal skips deferred calls: a SIGTERM between the spawn and the exit left 'local-ai run' reparented to init with nothing left that knew to shut it down. Ctrl+C was already safe, but only incidentally, because the child shares this process' foreground process group. A signal handler rather than Pdeathsig on the child. Pdeathsig is Linux-only and, in Go, is delivered when the OS thread that forked exits rather than when the process does, so it can fire on a healthy parent. Setpgid would break the Ctrl+C that works today by taking the child out of the foreground group. SIGHUP joins SIGINT and SIGTERM: a terminal program whose terminal is gone has nobody left to talk to. The same context is what cancels the agent, which nib leaves to its embedder. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): only skip the server checks for work that stays local Two argument shapes were classified wrongly. Every 'mcp ...' invocation counted as management, so 'local-ai chat mcp --stdio', which serves the agent over MCP and needs a model like any other session, was handed an empty one. And --init, whose shell snippet a user pastes into an rc file long before any server exists, went the other way: it demanded a running server to print a static string. The mcp split is asked of nib's own IsMCPManageSubcommand rather than restated here, so a verb added upstream cannot drift out of this list. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): exit with the agent's status instead of reporting it twice nib writes what went wrong to stderr and returns nothing but an exit code, so returning that error unchanged had main log "Error running the application error=exit status 1" underneath the message the user had just read. The refusal to render the full-screen interface into a pipe is the one they meet in practice: it names --cli, and burying that hides the fix. ExitCodeError says "already reported, exit with this status". main honours it and prints nothing more, so a piped or redirected chat still fails a script the way it should. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * style(chat): route interactive chatter through one writer helper The prompts and notices all write to a terminal, where a failed write is not worth failing the session over and the read that follows the question reports the real problem. say says that once instead of five discarded error returns. The command's one-line help comes along: chat is no longer "an interactive chat session". Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(chat): record why the agent gets this process' streams Injecting them is what makes nib refuse to draw its full-screen interface into a pipe and name --cli, instead of rendering onto a terminal the caller may not own. The tradeoff is worth stating where the wiring is. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): stop the session's server on cancellation, not on the way out The deferred Stop is reached only if the agent returns, and cancelling the context does not make it: nib hands the TUI to bubbletea without the context, so what actually unwinds a running session today is bubbletea's own SIGINT and SIGTERM handler. SIGHUP has no such backstop, and registering for it removed the default disposition that used to end the process outright, so kill -HUP left a live TUI with a cancelled context and the started server still running. runSession watches the context alongside the agent and stops the server the moment it is cancelled, so the guarantee no longer depends on what the agent does with cancellation. Stop is idempotent, so the deferred call stays correct and free. The doc comment on shutdownContext described the mechanism it was supposed to work by rather than the one that does. Corrected, bubbletea's handler included. ResolveModel now checks the chooser's answer against what it offered. The shipped chooser answers by list index and cannot be wrong, but ModelChooser is exported, the answer is persisted, and every later run starts against it, so the invariant belongs at the consumer. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(chat): bump nib to v0.5.1 v0.5.1 carries four fixes that matter to 'local-ai chat': - --init now names the embedder's command, so the emitted widget invokes 'local-ai chat' rather than a bare 'nib' the user does not have. - A piped CLI session that succeeds exits 0 instead of failing with EOF. - EOF at a tool-approval prompt denies the call rather than approving it, and the session exits 3 (app.ExitCodeApprovalNoInput) so a script can tell "answered" from "refused to act" without reading stdout. Read-only tools are unaffected and still run. ExitStatus already unwraps app.ExitError, so the code propagates with no change here. - RunTUI passes the context to bubbletea and gives up bubbletea's own signal handler, which makes shutdownContext the single owner of the signal and stops a SIGHUP leaving a wedged TUI behind. Verified against a live server on 127.0.0.1:8080: the three --init shells, a piped prompt exiting 0, a denied 'touch' that left no file and exited 3, a read-only 'ls' that still ran and exited 0, and a SIGHUP that unwound a TUI running under a pty. Two comment blocks in run.go described the old TUI behavior and are now wrong, so they are corrected in the same change. No behavior change: both shutdownContext and runSession are untouched, and stopping the server on cancellation is still worth keeping independent of how promptly nib unwinds. One known gap, not addressed here. The widget --init now emits runs 'output=$(local-ai chat --height 50%)', and runAgent injects Stdout unconditionally, so under $(...) nib refuses the TUI for a non-terminal stream. This is the cost the runAgent comment already anticipated, now that the snippets no longer hardcode standalone nib. Ctrl+Space should not be documented until that is decided. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): let nib own stdout, so the Ctrl+Space widget works The widget 'local-ai chat --init' emits runs 'output=$(local-ai chat --height 50%)', which puts a pipe on stdout by construction. runAgent injected os.Stdout unconditionally, and nib refuses every mode but --cli when a stream it was handed is not a terminal, so Ctrl+Space printed "Re-run with --cli to use the injected streams" and inserted nothing. Verified against a pty before and after. nib reads a nil stream as "not injected" and falls back to the process stream, which is how an embedder asks for nib's own behavior. That is what stdout needs: the interface renders on /dev/tty but writes the chosen command to stdout even when stdout is a pipe, and that write is the whole of the shell-capture idiom. Stdin is deliberately left injected. A piped or redirected stdin really is ignored by the interface, so the refusal is the honest answer there, and it is the one users meet: 'echo q | local-ai chat' still says to re-run with --cli, once, exit 1. Nilling stdin the way stdout is nilled would delete that silently. Stderr is not gated by nib at all and is unchanged. One case does change and cannot be kept: 'local-ai chat > out.txt' from a terminal no longer refuses, because it is indistinguishable from the widget. It renders on /dev/tty and writes the capture line to the file, which is what standalone nib does. The app.Options literal moves into agentOptions so the decision is reachable from a spec rather than being a detail of a function that takes the terminal. Both sides of the asymmetry are pinned: reinstating 'Stdout: opts.Out' fails "hands nib nothing for the process stdout", and nilling stdin fails "hands the process stdin over". Also rewrites the last comments describing the pre-v0.5.1 behavior. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(chat): say what the stream refusal actually keys on Two comments still called it the refusal to render the interface "into a pipe". That was true when both stdin and stdout were injected, but a pipe on stdout no longer refuses, so the wording now points at precisely the case that was un-refused to make Ctrl+Space work. Only a stdin that cannot be read triggers it, and both comments now say so and name the command a user meets it with, 'echo q | local-ai chat'. The agentOptions doc also said a "file a caller chose" stays injected and refused, which reads as though 'local-ai chat > out.txt' still refuses. It does not: a shell redirect arrives as os.Stdout and is nil-ed like the widget's pipe, because the two differ only in being a regular file rather than a FIFO and nib's gate does not look at that. What stays injected is a writer an in-process caller chose for itself. Says that now, in the doc and in the spec comment that had the same ambiguity. Comments only. No behavior change. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: local-ai chat is now the built-in terminal agent `local-ai chat` was a plain chat prompt and is now an agent that runs shell commands behind an approval gate, so the pages that described a REPL were wrong rather than merely thin. Adds a Terminal agent feature page at /features/terminal-agent covering the approval gate, piped runs and their exit codes, Ctrl+Space, model resolution, state directory, and the pass-through management commands (including the `--yes` caveat that leaves a plugin installed but disabled in a script). The three-way "looking for something else" notice becomes four-way and moves into an agentic-routing shortcode. Four hand-kept copies of the same paragraph is what produced the drift the new page would otherwise have added to; the shortcode takes `current=` so each page still marks itself, and errors the build on a name that is not one of the four. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * website: the agent is in the binary, not a second install The nib section sold a separate tool you also install, with a GitHub link as the only way in, which is now the wrong order: the agent ships compiled into local-ai, and the standalone binary is the second reason to care rather than the first. Leads with `local-ai chat`, keeps nib as the SSH-anywhere story, and adds a docs CTA pointing at the new Terminal agent page. id="nib" is left alone because localai.io/#nib is linked from outside. The two credits on the demo clip named nib as the thing that drove the machine; they now credit the agent in LocalAI, which is the same agent. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * website: fix the exit keys, the plugin warning, and the redirect gap Three claims on the chat-agent pages that the code does not back. try-it-out told readers to press Ctrl+D. nib has no Ctrl+D handler: the full-screen interface quits on Esc or Ctrl+C, and Ctrl+D is only an exit in --cli, where it arrives as ordinary tty EOF. That sentence had replaced the removed /exit and /quit text, so the page was left with no working way to leave a session. Document both modes, since they differ. The plugin warning said nothing tells you the install stopped short. It does: the command prints that the plugin was left disabled. What it does not do is say so in its exit code, which is 0 either way. That is the part a script cannot work around, and it is the reason to pass --yes. Overstating it in the paragraph that gives the advice only makes the advice easier to dismiss. Redirecting stdout no longer refuses; the interface goes to /dev/tty and only the yanked command reaches the file. It is what lets the Ctrl+Space widget capture a command at all, since a redirect and out=$(...) are the same thing to the stream gate. It was documented nowhere. A non-terminal stdin is still refused, and the new text says which of the two it is. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): make the CLI flags outrank the agent config file local-ai chat routed --endpoint, --model, --api-key, --trace-dir and --yolo through nib's app.Options.Defaults. Defaults are seeds: they sit beneath the config file, so the file silently undoes them. That made the flags accepted and inert, and not in an edge case, since EnsureStateDir writes base_url on the first run and the interactive picker writes model, so from the second run on the file carried a value for both. Observed against a live server: with base_url: http://127.0.0.1:9999/v1 in the config and --endpoint http://127.0.0.1:8080 on the command line, the probe hit 8080 and every agent turn posted to 9999. With model: gemma-4-e2b-it-qat-q4_0 in the config, --model lfm2.5-8b-a1b was ignored on the wire. nib v0.6.0 adds app.Options.Overrides, applied above the config file and above the bare environment block. Move the whole block there: all five values are decisions this invocation already made on the user's behalf, and a flag the config file can undo is not a flag. Nothing is left in Defaults, because LocalAI's one genuine seed, the initial base_url, is written into the config file by EnsureStateDir rather than handed to nib. Two limits come with the channel and are documented on agentOptions rather than worked around. An override can only raise a field, since nib cannot tell "set to the zero value" from "not set", so --yolo can turn approval off but nothing on the command line turns it back on over an approval_mode: auto in the file. And nib's own NIB_TRACE_DIR and NIB_YOLO are resolved after the config load and still outrank these, deliberately, upstream. The existing spec pinned that the right values reach app.Options, which they always did, which is exactly why it could not see nib discarding them. The new specs resolve the config the way app.Run resolves it, against a real config file that disagrees with every flag, and one asserts Defaults stays empty. docs/content/features/terminal-agent.md already documented --model as winning over the saved model; that was false before this change and is true now, so no docs edit was needed. Assisted-by: Claude Code:claude-opus-5 [Bash] [Edit] [Write] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): document intentional config file read Assisted-by: Codex:gpt-5 [gosec] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
Dimitris Karakasilisandlocalai-org-maint-bot
|
c089caf320 |
feat(sycl): make the intel llama.cpp backend self-contained on any host (#10991)
* feat(sycl): make the intel llama.cpp backend self-contained on any host The SYCL backend shipped an incomplete oneAPI runtime AND relied on a host-provided GPU driver, so it only ran inside the build container. On a bare host it died with "libze_loader.so.1 / libdnnl.so.3: cannot open shared object file", and even with the host's Intel driver installed it SIGSEGV'd during SYCL init when the host driver was built against a newer glibc than the backend's bundled loader (rolling-release distros). package_intel_libs now bundles the complete, coherent oneAPI runtime (the missing MKL ILP64 / sycl_blas / tbb_thread + oneDNN + the dlopen'd UR adapters, plus a sweep of the backend binaries' own direct deps) and the Intel GPU userspace driver (libze_intel_gpu + libigdrcl + IGC + gmm) with its OpenCL ICD manifest, mirroring how package_vulkan_libs bundles Mesa. run.sh points the Level Zero and OpenCL loaders at the bundled driver, and install-base-deps.sh installs it in the SYCL build image. Bundling the driver is safe across kernels because it talks to the host i915/xe via the stable DRM UAPI (unlike NVIDIA's kernel-locked userspace). Validated on Arch (glibc 2.43, i915): the backend loads and runs on an Iris Xe with no host Intel packages installed. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> * fix(sycl): install a driver that exists, and let the user choose their own The driver install added earlier in this branch asked apt for intel-level-zero-gpu, which is not a package in Ubuntu 24.04. apt fails outright on an unknown name, so neither driver was installed, nothing was there to copy, and the images carried no driver at all. It now comes from Intel's own repository, which has 25.18 for this Ubuntu release, against 23.43 from late 2023 in the Ubuntu archive. The archive driver does not know any card released since, so a machine with a recent Intel GPU would end up carrying a driver that cannot drive it. Anything that goes wrong during that install fails the build on purpose: an unreachable repository is a passing problem that a retry fixes, while quietly carrying a different driver, or none, is a difference nobody would notice until a user reports an idle GPU. run.sh used to overwrite whatever driver the user had chosen. Level Zero uses only the driver it is given, so on a machine with a card too new for the carried driver, the GPU would go unused with no way back. Both that setting and the OpenCL one are now left alone when already set, and the docs say how to point a backend at the machine's own driver. The OpenCL setting also used to be applied whenever the backend held a driver list, even when the driver it named had not been copied, which leaves OpenCL with nothing instead of falling back to the machine's own driver. It now requires the copied driver to be present, and the packaging leaves out the list entry of any driver it did not copy. The oneAPI images list a processor-only OpenCL library, which was being carried with nothing behind it. Two more corrections in the packaging. The scan for libraries a program is linked against only looked at files named llama-cpp-*, so turboquant and bonsai, which are also built for Intel GPUs, were left with the incomplete set of libraries this branch set out to fix; it now looks at every program in the directory. And a build that should carry a driver but ends up without one now says so, which is what a stale prebuilt base image looks like: such a backend still runs on a machine that has its own driver, so nothing fails and the only other symptom is a user reporting an idle GPU. Backends now also ask the driver to report how much graphics memory is free, without which llama.cpp reads zero on an integrated GPU, since such a chip shares the system memory instead of having its own. turboquant and bonsai get the same run.sh handling as llama.cpp. The driver is only carried by the builds that start through run.sh, because run.sh is what points Level Zero and OpenCL at it. The Python backends for Intel GPUs start differently and would never load it, so they keep using the machine's own driver rather than carrying several hundred megabytes they cannot use. Checked in a container on Ubuntu 24.04: the install brings driver 25.18 with the files where the packaging expects them, an unreachable repository fails the build, and the copied set resolves on its own once the machine's Intel packages are moved away. Assisted-by: Claude:claude-opus-5 Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> * fix(ci): rebuild every Linux backend when the GPU packaging script changes scripts/build/package-gpu-libs.sh decides which GPU libraries end up inside an image. The filter that builds the backend matrix listed it as an input of the Python images only, so changing it rebuilt no Go and no C++ backend, even though those run it from their own package.sh. A packaging fix aimed at the Intel llama.cpp backend could merge and reach no image, which is the same failure this rule was written to prevent. Assisted-by: Claude:claude-opus-5 Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> * fix(sycl): carry only the driver Level Zero uses, not the OpenCL one llama.cpp reaches an Intel GPU through Level Zero, which hands the driver programs that are already compiled and so needs only the back end of the graphics compiler. The OpenCL driver can be handed source code instead, so it needs the compiler's front end as well, and that arrives with its own copy of clang. Carrying it cost about 139 MB in every backend built for Intel GPUs, and took the carried set from 123 MB to 261 MB. Nothing here takes that path. No LocalAI code selects an OpenCL device, each backend image holds one backend, and the documentation never described OpenCL as a way to run models: the only mentions are a stale clblas row in the BUILD_TYPE table, for a llama.cpp backend that no longer exists and that no build matrix entry uses, and the sycl-ls troubleshooting hint. Before this branch the packaging carried the OpenCL loader and adapter but no driver, so the path could not work in a released image either. There is nobody to keep working. The driver list that OpenCL reads is no longer carried, and run.sh no longer sets OCL_ICD_VENDORS, so OpenCL inside a container keeps using whatever the image provides rather than being pointed at a directory with no driver in it. Checked in a container against the real 25.18 driver: the carried set is 123 MB with nothing unresolved, and Level Zero still reports the GPU with the machine's own Intel packages moved out of the way. Neither the Level Zero driver nor the compiler back end names the front end or clang among the libraries it opens by name, so the leaner set is complete for this path. Assisted-by: Claude:claude-opus-5 Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> --------- Signed-off-by: Dimitris Karakasilis <dimitris@karakasilis.me> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com> |
||
|
Adiraandlocalai-org-maint-bot
|
ef724a3c9d |
feat(api): add /v1/detokenize endpoint (#9620)
* feat(api): add /v1/detokenize endpoint Closes #1649. Mirror of the existing /v1/tokenize path, requested by @benniekiss in the issue thread for "complete API workflow" use cases that need to turn token IDs back into text without local processing. - Add Detokenize gRPC RPC with DetokenizeRequest{tokens} / DetokenizeResponse{content} messages. - Implement in the llama.cpp backend using common_token_to_piece, the same primitive TokenizeString already uses internally. - Other backends inherit the default Unimplemented from base.Base, in line with how Detect, Rerank, etc. are gated per-backend. - Wire up the Go gRPC interface, server, client, and in-process embed wrapper alongside their TokenizeString counterparts. - Add the schema types, ModelDetokenize wrapper, HTTP handler, route registration, RouteFeatureRegistry entry (gated by FeatureTokenize so no new feature flag is needed), and the discovery map entry under ai_functions. - Regenerated swagger reflects the new endpoint and types. - Update authentication.md to list /v1/detokenize alongside /v1/tokenize. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> * test(e2e): add mock backend tests for /v1/detokenize Add Detokenize to the mock gRPC backend and wire up two e2e tests in the MockBackend suite: one that posts known token IDs and asserts a non-empty content response, and a round-trip that tokenizes first then detokenizes the returned IDs. Addresses reviewer feedback on #9620. Assisted-by: Claude:claude-sonnet-4-6 Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> * fix(kokoros): implement detokenize in the Rust backend service The Detokenize RPC added in this PR grows the tonic-generated Backend trait. Unlike the other languages there is nothing to inherit a default from — Rust trait impls must list every method — so backend/rust/kokoros failed to compile: error[E0046]: not all trait items implemented, missing: `detokenize` --> src/service.rs:72:1 72 | impl Backend for KokorosService { Go backends pick up the Unimplemented default from base.Base, and the generated C++/Python servicer bases default to UNIMPLEMENTED, which is why the Rust backend was the only one that broke. kokoros is the sole Rust crate in the tree, so this is the full extent of the fallout. Return Status::unimplemented("Not supported"), matching how this same file already gates tokenize_string and ~20 other unsupported RPCs. Fixes the tests-kokoros and backend-jobs-singlearch-4 (-cpu-kokoros) failures on the previous head. Assisted-by: Claude:claude-opus-5 cargo Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> --------- Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com> |