mirror of
https://github.com/mudler/LocalAI.git
synced 2026-10-28 09:08:54 -04:00
e45dc5e9f72ea510bc8aab3248d2bef3be8f2263
83
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 mudler-agentandEttore Di Giacinto
|
e45dc5e9f7 |
feat(addons): installable add-ons foundation and the Tables add-on (#12702)
* feat(addons): add the add-on runtime, navigation and first-party add-ons Add the add-on runtime to LocalAI: manifests, the host operation broker, admin gating by account role, the add-on UI bundle builder and the SDK for add-on pages. Move the navigation to Chat, Models, Add-ons and Settings, with pinned add-ons in the sidebar and an Add-ons drawer. Ship the first-party add-ons (voice, face, fine-tuning, quantization) as packages for linux/amd64, linux/arm64 and darwin/arm64. Intel macOS (darwin/amd64) is enabled in the platform files. Squashed from the review branch. The tree is unchanged; the pre-squash history is kept on a local backup branch. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gosec): tighten file modes and annotate false positives Write job state, configs and imported models with mode 0600, clean the paths that are read from disk, and mark the route tables, the header name and the retry jitter as reviewed false positives. Assisted-by: Claude Code:claude-sonnet-5-5 gosec Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): keep API keys off recovery mutations and same-site cookies POST /api/addons/dispatch forwarded the recovery and update mutations to the owner for any administrator key, which skipped the browser-session rule of their own routes. Refuse those actions there. Also apply the same-origin rule to the legacy token cookie, not only the session cookie, because the global CSRF check lets same-site requests through. Do not fail the start of LocalAI when an add-on library cannot be refreshed; log it and keep the installed add-ons up. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): refuse a reinstall that would migrate retained data unbacked An upgrade takes a verified backup before the new package migrates the data. A reinstall over retained data has no running generation to stop, so the new package would migrate it without one. Refuse a reinstall whose data format differs from the retained one. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): release the manager lock while a release is fetched Submit held the state lock across the catalog lookup and the package download, so the apps list and status reads stalled for the whole install. Serialize Submit and Reconcile with their own lock and drop the state lock around the fetch only. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): kill an add-on only after three missed health probes One slow 100 ms probe killed a busy add-on and dropped the work it was serving. Require three misses in a row. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): move settings under the operate hub The Operate hub already has Settings as a tab, so the sidebar no longer shows a separate Settings entry. The /app/settings route stays, and the hub tab still reaches it. The navigation test now expects Chat, Models and Add-ons as the primary items. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): accept same-host browser requests that carry no fetch metadata Browsers do not send Sec-Fetch-* headers over plain HTTP on an address that is not localhost, so every add-on request from a LAN address was refused as cross-site. Keep refusing a Sec-Fetch-Site value other than same-origin. When the header is absent, accept the request only if its Origin, or failing that its Referer, names the same host and port as the request. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): show the server's reason when an add-on request is refused A 403 or 400 answer showed a generic message and hid the reason in the response body, so a refusal by the browser guard could not be told from a missing permission. Show error.message from the body as the server sent it, and keep the generic text when the body has none. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): run the add-on owner in the normal binary The owner was only enabled when an embedding host set the add-on config, so a normal run answered every add-on request as unavailable. Add --addons-enabled (default true) and --addons-ownership-mode (default named-user, or local-owner; anything else stops startup). The node is its own owner: state lives under the data path, and the carrier uses loopback keys made at each start and kept in memory. --addons-enabled=false turns the owner off completely. Named-user mode needs the account database. With authentication off the owner stays off and a warning says how to use local-owner. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): keep the loopback owner address stable across restarts The single-node loopback transport took a new ephemeral port at each start. The owner stores its address in its identity file and refuses to start when the address differs, so every start after the first failed with "invalid addon owner configuration". Choose the port once, keep it in the add-on state root (loopback-port, mode 0600) and reuse it. If the port is taken at a later start, stop with an error that names the port and --addons-enabled=false, because another port would break the identity check. The keys and certificate stay in memory. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): let add-on pages keep the host modules they were written against A page written for the web UI imports React, the router, i18next and a few shared components. The add-on builder refuses a bundle that holds a host file, so moving such a page into an add-on meant rewriting it. Add a compat mode to the shared builder. It answers those imports from a directory of stand-ins that read the SDK, bundles the dynamic import of the page code into the one output file, and collects the style sheets the page imports. A relative import that is not in the add-on's own source resolves to the file at the same path under the compat directory. Widen the SDK for these pages, without a version change: more host components (spinner, modal, lightbox, model picker) and hooks (models, resources, voice profiles, operations, MCP client), a style on icons, the icon names the pages draw, sdk.navigation.openApp to open a page of the app (only addresses under /app, and the way back as the only state) and sdk.portal. The shell passes the query string to the page. The stand-ins cover the router (the tab in the path, the model in the query), the translator over the host locale files, the core API calls through sdk.core.fetch, and byte copies of the pure helpers, with tests that keep the copies equal to the originals. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): ship Studio and Talk as add-ons instead of built-in pages Studio and Talk were compiled into the web bundle and always listed in the Add-ons drawer. They are add-ons now, like Voice, Face, fine-tuning and quantization: a person installs and removes them, the drawer lists them only while they are installed, and they are packaged the same way. Move the pages, their components, hooks and helpers into addons/studio and addons/talk. Each has a manifest, a UI bundle, a release platform matrix and a catalog template, and is built by the compat mode of the shared builder. The core API stays where it is: the pages call it with the signed-in person's session. The history keeps its browser storage keys. The unit tests of the moved helpers move with them. In the core, drop the two pages, the Create group of the sidebar (Chat joins Home and Models in the old layout), the built-in catalog entries (the catalog is empty) and the talk locale namespace. The old addresses (/app/studio, /app/talk and the generator links) redirect to the add-on pages, with the model as ?model=. The Home /studio command is offered only while the Studio add-on is installed and opens it. The release workflow and its tests now list seven add-ons. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): run the Studio and Talk specs against the add-on bundles The pages are add-ons now, so a spec that draws one installs the real built bundle first: the apps list names it and its module and style sheet are served from the build output, the way the add-on owner serves them. studio-addon.js holds that helper, and the Studio and Talk fixtures call it. Update the specs for the new addresses (/app/addons/studio/studio, the model as ?model=), the session permission the add-on route needs, the legacy redirects, the Home /studio command that appears only while the add-on is installed, and the sidebar without a Create group. Add specs for the add-ons themselves: every tab draws with no script error, the shell titles the page once, the old addresses say so when the add-on is not installed, a button of the page reaches a core page with the way back, and the drawer lists both only after they are installed. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): centre the icon of a built-in tile in its icon box The drawer drew the icon of a built-in tile (Studio's palette was one) in a bare 40px box. Only the placeholder box was laid out as a centred grid with the 20px size, so the real icon sat in the top-left corner at the size of the page text. Give the built-in tile the same classes as the placeholder box, and check in a browser that the icon is centred. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(addons): describe Studio and Talk as add-ons Replace the built-in list (now empty) with the Studio and Talk add-on sections, the new targets of the old links, the Home command, and the seven packaged add-ons. Describe the SDK additions the pages use: components, hooks, icons, openApp and portal. Say on the Studio page that it is installed from the drawer. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): click the shell title in the Studio slash-key spec The add-on shell titles the page, and the Studio overview no longer draws a second title, but the generator tabs keep theirs, so a bare h1 locator matched two elements. Click the first, which is the shell's. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): read the loopback port file through os.Root The persisted port file was read with a path built from the add-on root, which gosec reports as a file inclusion through a variable. Open it through os.Root so the name cannot leave the root, even by a symbolic link, and require a regular file with no group or other access. A looser file now fails closed instead of being trusted. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): expect Settings under the Operate hub, not the sidebar Settings moved out of the sidebar into the Operate hub. The navigation spec still expected a sidebar link. Assert that the sidebar has no Settings entry, and that the Operate hub opens Settings as a tab. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): drop Settings from the 404 places Settings is a tab of the Operate hub and not a destination of its own, so the 404 page does not list it. Assert that it is absent. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): drive the media history and TTS specs through the Studio add-on The image, video, sound and speech generators are pages of the Studio add-on now, and the old addresses redirect to it. Install the add-on with the Studio fixtures, open its pages and use its compose card, so the specs run against the page that exists. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): count the first-run steps, not every list item The recommended models are a list inside the second step, so counting all list items read 3 only while the models were still loading. Count the step elements, which does not depend on the load order. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): fail the node refresh by state, not by call count The spec served the list to the first request and failed the rest. The first request is not always the one from the page, so the page could get the failure first and show no list. Serve the list until the page shows it, then fail the refreshes. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): default to local-owner when auth is off A server without authentication has no accounts, so named-user left the add-on owner off. Resolve the unset mode in one place: local-owner when authentication is off, named-user when it is on. A flag or environment value still wins, and an unknown value is still refused. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): register an official add-on library by default Add gallery/addons/index.yaml, a YAML index of the four first-party add-ons (fine-tuning, quantization, voice, face) in the library format. The entries come from addonpkg catalog over locally built linux/amd64 packages, so the digests pin those images. On the first start, when the owner has no sources.json, register the raw GitHub URL of that index on master as the "official" library and write the file. The file is then the record: a library the admin removes stays removed, and an existing file is never changed. --addons-official-library=false skips the default. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(addons): document the ownership default and the official library Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): list Studio and Talk in the first-party index Add the Studio and Talk entries to gallery/addons/index.yaml next to the four existing ones. They come from addonpkg catalog over locally built linux/amd64 packages, so the digests pin those images. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): expect Studio and Talk in the official index The official index now lists six add-ons. Update the test that decodes it so it accepts Studio and Talk. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): add a real icon to each first-party add-on Each icon is a 256 by 256 PNG: a rounded square in one colour of the UI kit palette with a white outline glyph from the kit sprite. Fine-tuning uses sliders, quantization layers, voice waveform, face user, studio palette, talk phone and tables database. They were rendered with headless Chrome from the sprite paths. Assisted-by: Claude Code:claude-sonnet-5-5 chrome Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): ship a committed icon.png instead of the generated square The shared UI builder and the Tables builder copy ui/icon.png from the add-on source tree to the package unchanged when the file exists. They write the generated placeholder only when it does not. Add a fixture with an icon and a test for both cases. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): list the tables add-on in the first-party index Add the Tables entry to gallery/addons/index.yaml next to the six existing ones. It comes from addonpkg catalog over a locally built linux/amd64 package, so the digest pins that image. The official index test now expects seven add-ons. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): make the Voice add-on one page with tabs The Voice add-on had four pages: recognition, library, create and speech, and the drawer listed two of them. It now has one page, "voice", and one drawer entry. Recognition, Library, Text to speech and From a recording are tabs of that page, drawn with the same tab strip the recognition page already used. The tab is the first path segment (voice/library), as in the Studio add-on, and the form that adds a voice is a view of the Library tab (voice/library/new). The old page names stay declared in the manifest. Each one redirects to its tab and keeps the query string, so saved links still open. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): send the old voice routes to the Voice tabs /app/voice, /app/voice-library and /app/voice-library/new now open the matching tab of the Voice add-on page while the add-on is installed and running, and the core pages otherwise. The moved-page helper compares the first segment of the target with the declared pages, so a target can name a tab. The docs and the navigation spec list the new addresses. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): cover the Voice tabs and the old page redirects The add-on unit tests render the tabs instead of the four pages. The Playwright spec checks the tab strip, the address of each tab, the missing Library tab for a member, and that the old page addresses of the add-on redirect to their tab with the query string kept. The migration table test follows the new targets. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(studio): keep the model menu inside a narrow screen The Home model picker sizes its menu from the viewport, and assumes the picker sits at the page gutter. In the Studio composer card the field is further in, so on a 390px screen the menu of the Sound expansion picker ended 7px past the right edge. Size the menu from the field it opens under instead. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): build the add-on bundles once for the whole run Each worker built the Studio and Talk bundles into the same dist directory, and the build empties that directory first. A worker that started, or restarted after a failure, removed the files another worker was serving, so a Studio spec saw "the style sheet of the add-on could not be loaded". Build once in a global setup before the workers start. A run without the setup still builds per worker. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): register the worker when the add-node spec says so The mocked roster added the worker on the third read of /api/nodes. The shell and the page both read it, and which one asks first depends on load, so the page could take the new worker as part of the roster it opened with and never show it as found. The worker now joins after the page shows it is listening. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): hold the gallery answer until the skeleton is seen The spec delayed the answer by a fixed 800 ms and then looked for the skeleton. The assertion polls at growing intervals, so on a slow machine it could step over that window. The answer now waits until the skeleton has been seen, and the spec then checks that the skeleton goes away. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): give the PDF tag the time the other PDF specs have pdf.js and its worker load on first use. The other specs in this file wait up to ten seconds for the result of that load. The first one waited five, and missed the tag on a busy machine. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): build the Face and Fine-tuning bundles in the global setup The navigation spec serves the built Fine-tuning bundle, but nothing built it, so two specs failed on a fresh checkout when the dist folder was missing. The spec also built the Face bundle in a hook that every worker runs, and a build rewrites the folder the other workers serve. The global setup now builds Studio, Talk, Face and Fine-tuning once, before the workers start. The spec builds the two bundles itself only when it runs without the global setup. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): fix the Home answers in the stale pin spec The spec asserts that the page shows no alert. The Home page raises its own alert when the model list fails or a staging operation has failed, and both answers come from the shared test server. A slow reply or a failed install left there counted as an alert about the pin. The spec now fixes both answers, so the alert count tells only about the pin. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): read the server address in the add-node spec from the page The spec expected the register address to carry port 8089, so it failed on any other port. It now expects the origin the page was opened from. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): give the navigation routes spec its own catalog No page is built in today, so the catalog is empty and the three specs that expect the studio and talk entries failed on the list and on the pin seed. They passed only while Studio and Talk were built in. The spec now sets a catalog of two ungated entries for each example and restores the old one afterwards. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(cli): report a failed write of the MLX hostfile The tunnel callback ignored the error of os.WriteFile and set the hostfile variable even when the file was not written. It now logs the error and leaves the variable alone. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): discard the close error of the held listener errcheck flags the deferred Close of the listener that keeps the port busy. The test has no use for that error, so the defer drops it explicitly. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): move the add-on tool tests to Ginkgo The fine-tuning, quantization and add-on host library tests used t.Errorf, t.Fatal and t.Run, which the forbidigo rule rejects. They now use Describe, It and Gomega, with a suite bootstrap in each package. The cases and the assertions are the same. The tables in the host client and the tool schema tests keep one entry per case. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): move the host operation tests to Ginkgo The hostops tests used the testing package calls that forbidigo rejects. They now run as Ginkgo specs under one suite bootstrap. Every check and every per-operation case is kept. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): move the pin store tests to Ginkgo The pin service, file store, sweep and shared store contract tests used the testing package calls that forbidigo rejects. The store contract is now a function that registers its specs for each store, so the file store, the memory-only store and the database store run the same cases. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): move the remaining add-on tests to Ginkgo The built-in catalog, tool arguments, loopback carrier, host operation contract, packaging, application, config, CLI and endpoint tests used the testing package calls that forbidigo rejects. They now run as Ginkgo specs. The endpoint tests also drop the unchecked Close errors that errcheck reports when the auth tag is on. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): wait for the targeted Reinstall button before the install click The "generic deletion blocks targeted install at confirmation" spec dispatches a popstate to set the target node, then clicks Reinstall with a script click. The click can run before the targeted page has rendered or while the button is still disabled, so no install request is sent and the poll on the install count times out. It failed 1 run in 10 with two workers and on every retry in CI. The spec now waits for the button to be enabled first. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(addons): refresh the voice package digest in the first-party index The Voice add-on became one page with tabs and got a new icon, so the digest in gallery/addons/index.yaml no longer matched the build. Render the entry again with addonpkg catalog over a locally built linux/amd64 package; the main page is now voice. The other entries are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): wait for the target node's Reinstall button after the popstate The "generic deletion blocks targeted install at confirmation" spec dispatches a popstate to set the target node, then waits for a button named Reinstall and clicks it. Until React renders the targeted page, the page still draws the generic Reinstall button. It is enabled and has the same name, so the wait passed on it, the click started a global install, and no node install request was sent. The poll on the install count timed out with the confirm dialog still open. With the CPU throttled to 8x, the spec failed on 20 of 20 runs, and the button it clicked had the title "Reinstall". The spec now waits for the button titled "Reinstall on <node>". With the throttle it passes 30 of 30 runs. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): give the Studio hand-off spec a budget for the cold bundle load The "a result made through a hand-off records which result it came from" spec opens the Studio add-on straight from an address, twice. Each is a cold load: the page shows "Loading" until the add-on module has run, and the next assertion had the default five seconds. The load takes about two seconds on an idle run and seven to ten under CPU load, so the spec failed on a loaded machine with the hand-off note not yet drawn. With the CPU throttled to 6x and eight busy loops running, it failed on 21 of 30 runs. The first assertion after each load now waits up to 20 seconds, and the spec then passed 30 of 30 under the same load. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): carry the identity page styles in the Face add-on The Face page is drawn with the shared identity views, but its style sheet held only the clip and error rules. The page, card, model list, known-faces table, enrol sheet and distance scale classes had no rules. They looked right only when the host had already loaded the style sheet of its own Faces page, which a visit through the add-on drawer does not do. Opened that way, the cards had no gap, the model list showed bullets, and the install buttons ran into the names. The Face sheet now holds the same identity rules as the core sheet, so the page looks the same whichever way it is opened. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): scope the narrow-screen identity rules of the Voice add-on The rules for narrow screens (single-column pairs, smaller tool padding, compact rank rows) were copied into the Voice style sheet without their media query, so they applied at every width. They now sit in a max-width 720px query, and the reduced-motion rule in its own query, as in the core sheet. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): drop the repeated title from the Voice and Face pages The add-on shell already titles the page, so the recognition pages showed the name twice: "Voice" above "Voices", "Face" above "Faces". Keep the supporting line and the model picker, as the Studio page does, and remove the unused title strings. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): replace the Pinned sidebar section with an Apps entry The sidebar no longer lists pinned add-ons. It shows one Apps entry that opens the add-ons drawer, with a Show all link below it that opens the same drawer. Pins are used from the drawer, where a pinned tile keeps its pin and move buttons. The server pins API is unchanged. Remove the PinnedNav component and its locale title, rename the items.addons label to Apps, and update the e2e specs and the add-ons documentation to match. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): assert the single shell title on the Voice and Face pages The add-on shell now shows the one title, "Voice" or "Face", and the page no longer repeats it. The tests still looked for the h1 "Voices" and "Faces". Assert that the add-on page has exactly one h1 and that it is the shell title. The core pages keep their own "Voices" and "Faces" titles. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): drop the repeated title from the Voice Library and Text to speech pages The add-on shell already titles the page "Voice", so these two tabs showed a second heading below it ("Speech voices", "Speak text"). Keep the supporting line, as the Studio and recognition pages do, and remove the unused title strings. The e2e spec now checks that each tab has exactly one h1, the shell title. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): list pinned apps under Apps in the sidebar A pinned app now shows as an indented child of the Apps entry, with its icon and title, in pin order. Show all stays the last child. The pins come from GET /api/addons/pins through the navigation context, and the apps from the add-on registry, which the drawer feeds on every list read, so the links follow the drawer. A pin without an installed app is not drawn, nothing extra shows when no app is pinned, and the children are hidden when the sidebar is collapsed to the rail. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(addons): say that pinned apps are listed under Apps in the sidebar The sidebar lists the pinned apps between Apps and Show all, in pin order. The pins section and the navigation section said that pins do not appear in the sidebar. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): remove the Show all link under Apps in the sidebar The link repeated the Apps entry, which opens the same drawer, and looked redundant when no app was pinned. Apps is now the only add-on entry; pinned apps still list under it in pin order. The link, its locale key and the style comment are gone, and the e2e spec and the add-ons docs now say that Apps is the only entry. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): show a first-run card on Apps when nothing is installed An administrator who opens Apps with no add-on installed now sees a short card with three steps: choose a library, browse and install, start the add-on. Its button opens the Browse tab. The card is hidden once an add-on is installed and for anyone who cannot install. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): hide the pinned apps in the icon rail When the window is narrower than 1024 px the sidebar shrinks to the icon rail by CSS alone, without the collapsed class. The pinned app links were still drawn there, with the indent meant for the full sidebar, so their icons sat to the right of the rail and were cut off. Hide the pinned app links in both rail states. The Apps entry keeps its centred icon. The e2e spec checks that the rail shows no pinned app and that every icon is centred at 1280 px collapsed and at 1000 px. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): stop the backend redirect test racing the installed list The test reads the redirected URL right after the path changes. The Backends page then loads the installed list and drops a selected backend that is not installed, so the backend parameter was sometimes already gone when the test read it. The mock server has no llama-cpp, and about 15 of 40 repeated runs failed on the backend assertion. The redirect itself was correct, including show_all. Serve the selected backend in the installed list, so the page keeps the selection and the assertions stay as strict as before. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): accept a same-host Referer or Origin for human confirmation Browsers send no Sec-Fetch-* headers over plain HTTP on an address that is not localhost, so creating a resource from a LAN address failed with "Creating a resource requires your own browser session". When Sec-Fetch-Site is present it must still be same-origin with an empty destination, as before. When it is absent, Origin or, failing that, Referer must name the request host and port. A destination or mode that is present must still be the expected one. This matches the rule in BrowserProtection. The credential, token cookie, API key and tool-context checks are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): keep hub entries off the Apps route in the sidebar The Build hub still owned /app/addons through its legacy Add-ons tab, so the sidebar lit both Apps and Build on the Apps page. A tab whose gate is closed now owns no route when the sidebar asks, so only the Apps entry is active there. The hubs keep their own routes, and the legacy navigation still gives /app/addons to the Build hub. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): label the pin control on app tiles and in the detail view The pin was a small icon-only button and hard to find. Each app tile now shows a button reading Pin, or Pinned with a filled pin, and the detail view has the same control in its header. Both toggle the same stored pin. The Apps grid lists pinned apps first, in pin order, then the others. The sidebar list and the server pins API are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): check the Voice From a recording tab has one title The Library and Text to speech tabs already assert that the add-on shell title is the only h1. Do the same for the From a recording tab, so a heading added there later fails the navigation spec. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): stop the add-on shell drawing a title over add-on pages The app decides what its page shows. The shell drew the add-on name as a header above every add-on page, so an add-on could not choose its own title. The shell now draws no header on a live, loading or frozen add-on page; each add-on titles its own pages. The unavailable and missing-page states keep the name, because no add-on page is drawn there. The runtime and shell specs now check that the shell adds no page title. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): let each add-on draw its own page title The shell no longer titles add-on pages, so each add-on draws one h1 per page. Voice titles its tabs "Voices" (Recognition and From a recording), "Speech voices" (Library) and "Speak text" (Text to speech); "Add a speech voice" already had its own. Face draws "Faces", Studio "Studio" and Talk "Talk". Tables promotes the database list and database name from h2 to h1. Fine-tuning and Quantization already drew a title. The add-on unit tests and the navigation and Studio specs now assert exactly one h1 per page, with the page's own title. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): add a kind to library entries and read unknown kinds tolerantly A library entry now has an optional kind: addon (the default when it is absent), backend or model. Every index written before kinds existed stays valid and its entries are add-ons. A reader keeps the entries of the kinds it knows and skips the others without reading their fields, so a newer kind never makes an older host refuse the whole index. The refresh result and the library state report how many entries were skipped and of which kinds, and the refresh logs a warning. A kind that is not a plain kind name still makes the index malformed. Search lists add-ons unless the caller asks for a kind, and Resolve returns add-ons only, so an add-on install cannot be pointed at a backend image. A library policy may name the signer with an anchored identityRegex instead of an exact identity, so builds from master and from tags both verify. An add-on install refuses a policy that carries one. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backends): install a backend from a library entry A library can now list backends. An entry of kind backend names the image of the backend gallery by digest, and POST /backends/library/install installs it on the local node. The gallery stays the source of truth and its install path is the one used: the library resolves the entry, takes the signature policy from the library, pins the image by digest and calls the same install as the gallery. Installing a name that came from a gallery, another library or by hand is refused and changes nothing. The signature policy of a backend library names the signer with an anchored identityRegex, so images signed by the merge workflow on master and on tags both verify. A pattern that is not anchored is refused. An install is bounded by a size limit per kind: add-ons keep 512 MiB and a backend image defaults to 8 GiB, set with --addons-backend-max-bytes. The limit reads the manifest after the signature check and before any layer is fetched. The metadata of an installed backend records the library it came from. scripts/backends/cmd/backendcatalog renders entries from backend/index.yaml and pins them against the registry. The pilot is the CPU build of local-store, in gallery/backends/index.yaml. The library is registered on the first start only with --addons-backend-library, which is off by default. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): show the kind of library entries and install backends from them The Browse tab lists the entries of every kind this host reads and marks each one as an add-on or a backend. A backend installs through the backend route and not through the add-on lifecycle: the confirmation says that the backend gallery is not changed and that a backend runs native code, and a refusal of the host, such as a name installed from another origin, is shown with its own reason. The details give the kind, the alias and the signer pattern. The library editor can name the signer with a pattern. Without this a save would drop the pattern of a backend library. The Libraries tab reports the entries a refresh skipped because their kind is not read. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(addons): describe backends in the library Document the entry kinds, how a reader skips a kind it does not know, the backend entry format, the signer identity pattern, the install route, the size limit of each kind, the pilot library and the command that renders it. The backend page links to it. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(mcp): install a backend from a library entry The backend library route installs a backend from an add-on library entry, and the admin REST surface and the MCP catalog must stay in step. Add the install_library_backend tool with an in-process and an HTTP client, the route map entry, the safety-rule listing and a skill recipe. Listing library entries stays on the add-on routes, which the assistant does not expose, so no list tool is added. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): carry the add-on library keys in every locale The add-on library and backend library strings existed only in English, so the other locales had no value for them. Add the missing keys of the addons namespace to de, es, id, it, ko, pt-BR and zh-CN. The values are the English text until a translation is contributed. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Revert "feat(mcp): install a backend from a library entry" This reverts commit |
||
|
Stefan Walcz
|
6fc2bc4967 |
fix(chat-history): create the store after the auth database is opened (#12788)
The conversation routes are only registered when the chat history store exists, and the store needs the auth database. It was created in start(), which New() runs before it opens the auth database, so the store was always nil and /api/conversations answered 404 with auth enabled; the React UI fell back to localStorage and nothing was synced between devices. The route tests register the routes with a store directly and never went through New(). Create the store right after the auth database is opened, and cover it through New(): with auth the store exists, without auth or with the WebUI disabled it stays nil. Docs-Not-Needed: bug fix; the documented server-side chat history now works as described Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Stefan Walcz <stefan.walcz@walcz.de> |
||
|
mudler-agentandEttore Di Giacinto
|
dce91ba7aa |
fix(backends): prefer installed development builds (#12782)
* fix(ci): capture and surface ginkgo errors in run-coverage.sh The coverage script ran ginkgo per test root with `|| fail=1`, which swallowed the exit code. Combined with `make --output-sync=target`, which buffers all output and flushes at the end, the actual ginkgo error was invisible in interleaved CI logs. Every test suite passed but the script still exited 1, and the error message was buried in 22 000 lines of buffered output. Each ginkgo invocation now captures combined stdout+stderr to a per-root log file. On failure the last 80 lines are printed to stderr so the error is visible in the CI log. On success the log is replayed to stdout so test output remains visible. The build flags (--tags, --coverpkg) are quoted correctly so values containing spaces (e.g. --tags="debug auth") are preserved through eval. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ci): create output dir before writing ginkgo logs The previous commit added per-root log files but forgot to create the output directory before the first ginkgo invocation. On CI the coverage/ dir did not exist yet, so the shell redirect to cover-pkg.out.log failed with "No such file or directory" before ginkgo even ran. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(core/http): add canvas preview route to auth coverage allowlist Commit |
||
|
mudler-agentandEttore Di Giacinto
|
39b3b31153 |
fix(backends): flag a llama-cpp backend that predates native decisions (#12733)
A decision request (/v1/systemone, /v1/decisions) sends Score with question_type "systemone". A backend built before native decisions ignores that field and runs plain scoring, so the caller saw "Score was not enabled" (500) or "candidates must be non-empty" (400). The gallery "latest" image can lag behind the server, and the upgrade check compares only tag, version and digest, so /backends/upgrades did not list the backend. Recognise those two errors for a decision request, return 501 with a message that says the backend is older than the server and must be upgraded, and mark the backend stale. The upgrade check now reports a marked backend with a `reason`, and refreshes at once. Reinstalling the backend drops the mark; if the new image is still old, the next request marks it again. Model loading is not affected. Refreshing the "latest" backend image is a separate publishing step and is not part of this change. Refs #12693 Assisted-by: Claude Code:claude-sonnet-5-5 [go vet] [go test] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
c7ce0c602c |
fix: stop listing the models directory on request paths and cache GGUF header reads (#12683)
* fix(http): stop listing the models directory on embeddings and bearer requests Every embeddings request read the whole models directory before the body was parsed, even when the body named its model. On a network-backed models directory this cost seconds per request. Embeddings now binds the body first and looks up a default model only when the body has none, like chat. The pre-bind default lookup and the gpt-4o constant default are gone from that route. The first-available default lookup skips the directory when the filter cannot accept an unconfigured file. Every usecase filter rejects files without a config, so the scan never added results for them. Match-all, nil and name filters keep the scan. The bearer token check is now an exact test: a configured name, or one stat of that name in the models directory. The value is a literal, not a regular expression, and a name that is not a local relative path never matches a file. Assisted-by: Claude Code:claude-haiku-5-5 * perf(config): cache GGUF header reads for model defaults Model defaults re-parsed the GGUF header on every SetDefaults call, which runs per request and at startup reload. Put a bounded LRU in front of the parser, keyed by absolute path and validated by size and modification time. The cached value is a snapshot of the fields the defaults use, not the parser output. A changed size or mtime is a miss. Missing files and parse errors are not cached, so the existing fallback is unchanged. LOCALAI_DISABLE_GUESSING, the empty-path guard, the alias early return, the panic recovery and the context-size default are all kept in front of the cache. Assisted-by: Claude:claude-haiku-5-5 * test(http): cover listing-free default lookup on every usecase route Assisted-by: Claude Code:claude-haiku-5-5 * feat(config): persist the GGUF header cache per instance The header cache kept parsed GGUF results only in memory, so every restart parsed each model file again before its defaults were known. Write the cache to a per-instance file in the data path and load it at startup. A restarted frontend with a fixed instance ID starts warm. The file is versioned and records the estimator inputs (gguf-parser version, candidate contexts, auto context size, offload count). A mismatch on any of them makes the file ignored. Each entry keeps path, size and mtime, so a changed model file is a miss on load. Writes run on one background saver, debounced, through a temp file and rename. The entry bound applies to the file as well. Without a data path the cache stays in memory only. Documented as a per-instance exception in the model configuration docs. Assisted-by: Claude:claude-haiku-5-5 * docs(embeddings): document model selection and the no-model error Assisted-by: Claude Code:claude-haiku-5-5 * refactor(config): split the header cache into a generic cache The GGUF header cache mixed two things: the mechanics (bounded LRU keyed by path and identity, per-instance file, debounced atomic save, tolerant load) and the GGUF fields. Move the mechanics into headerCache[S], driven by a headerCacheSpec[S] that names the file prefix, schema, fingerprint, parser and the snapshot encode/decode pair. The GGUF cache becomes one instance of it. File names, the identity check, LRU size, debounce, atomic write and the corruption handling are unchanged. The on-disk JSON keeps the "estimator" key so existing files still load. Assisted-by: Claude Code:claude-haiku-5-5 * feat(config): share one GGUF header cache file between frontends Every frontend used to write its own header cache file, named after its instance ID, so a header read by one frontend was never reused by another. The cache now uses one file, header-cache.json, in the data path, for all frontends and for standalone mode. The instance ID no longer changes the file name. Frontends share the file in three ways. A save first merges the file's entries in, so one frontend's write does not drop another's entries. A miss reloads the file when its modification time has changed, and a hit never reads it. Writes go to a temp file that is renamed into place, so a reader never sees a partial file. Two frontends saving at the same moment can still drop each other's last entries. That costs one re-parse, never a wrong value, because every entry is checked against its model file before use. The shared file is valid only when every frontend mounts the same data volume. The distributed-mode and model-configuration docs state this mode and its volume limit. Assisted-by: Claude:claude-haiku-5-5 * feat(config): give an instance ID its own header cache file Assisted-by: Claude Code:claude-haiku-5-5 * style(config): gofmt the distributed config struct Assisted-by: Claude Code:claude-haiku-5-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
895d50385f |
fix(distributed): converge model configs across frontends (#12558)
* fix(galleryop): announce model changes before the preload After a gallery install or delete, the replica that ran it replaced its config loader, then preloaded every installed model, and only then published the models invalidation. The preload does remote lookups and checksums for each model, so on a large models directory peers learned about the change minutes after the originator listed it. When the preload failed or the operation was cancelled, the event was never sent. Publish the invalidation, and apply the delete lifecycle, as soon as the loader holds the new set. The preload still runs afterwards with its own error handling. Its failure is reported on the operation, but it no longer rolls back a deletion that peers have already applied. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): resync model configs from the models directory Frontends refresh their model configs only when a models invalidation arrives on NATS. NATS keeps no history, so a frontend that is disconnected when the message is published never applies the change. It keeps serving the old config, for example an alias that points at the previous model, until some later change happens to touch it. Each frontend now reruns the peer reconcile against the shared models directory after every NATS reconnect, and every --model-config-resync-interval (default 30s) when a config file changed. The pass names no model, so only models whose file changed get a revision transition, and an unchanged directory costs one read of the config files. The reconcile replaced the whole loader with a parse of the models directory, which dropped models loaded with --config-file and published a deletion revision for them. Configs defined outside the directory are now kept, both there and after a gallery install. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nodes): stop stale frontends from retargeting alias rules A scheduling rule keyed by an alias derives its target from the alias mapping of the frontend that reads it. Every frontend keeps its own copy of the model configs, so after an alias is repointed a frontend that has not reloaded it still resolves the old target. Two frontends then rewrote the rule's stored target_model against each other on alternate reconciler ticks, and the outdated one scaled up the model the alias used to point at. The registry already records the accepted config revision of each model. A frontend now derives a rule's target from its own alias mapping only when its config revision for the rule's name matches that record. Otherwise it keeps the stored target_model: it neither writes the column nor reconciles replicas of the old target. The check reads the database only for a rule whose stored and derived targets differ. With no accepted revision on record, the old behaviour stays. Assisted-by: Claude Code:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
99043b442c |
feat(router): route with native decision models (#12449)
* fix(schema): preserve SystemOne image inputs Assisted-by: OpenAI * test(schema): follow Ginkgo conventions for decision inputs Assisted-by: OpenAI * feat(llama-cpp): dispatch native decisions through Score Upgrade the stock dependency and reconcile Score/TTS patches. Reuse native decision parsing, tasks, formatting and response-reader cleanup; preserve ordinary scoring admission and guard older dependencies. Assisted-by: OpenAI * refactor(systemone): share request and model validation Assisted-by: OpenAI:gpt-5 * fix(systemone): preserve HTTP wire-byte validation limit Keep structural validation separate from the serialized internal request bound so HTML escaping cannot reject valid HTTP payloads. Assisted-by: OpenAI:gpt-5 * feat(systemone): bound images and account native decisions Preserve public wire limits independently from router serialization. Reject unsupported NER images, map native request/capability errors, and stamp explicit usage once. Advertise decisions for stock llama-cpp. Assisted-by: OpenAI * fix(systemone): record usage on registered native route Exercise real registration and billing with a mock native backend. Reject empty native responses, malformed image URLs, trailing JSON, and wire overflow including whitespace. Assisted-by: OpenAI * feat(router): add lazy native decision transport Bind named models through internal ModelSystemOne calls with shared validation and bounded abandoned operations. Remove request and echoed-error contents from decision traces. Assisted-by: OpenAI:gpt-5 * feat(router): classify overlapping policies with native decisions Ask independent noul questions, validate probabilities and preserve first-superset routing. Wire the central factory with config-sensitive invalidation and cancellation-safe resolution. Document native framing and bounded operation limits. Assisted-by: OpenAI:gpt-5 * feat(gallery): add pinned Julia-1 native decision model Add a separate text-only llama-cpp Q8 entry with pinned Apache-2.0 source provenance and checksum. Installed using the gallery installer and exercised choice, score and noul on CPU. Assisted-by: OpenAI * test(router): verify native decisions through central factory Add an opt-in real-model Ginkgo integration covering the native Go loader and C++ transport, token usage, independent overlapping labels, and candidate selection. Document owned-server execution and the intentionally non-quality threshold. Assisted-by: Codex:gpt-5 * fix(llama-cpp): align upstream pin and preserve decision signatures Advance to bed0a856 without losing the automated upstream bump. Detect full-request fill_task support at compile time and forward every question for Nimble framing while retaining the earlier native signature. Preserve reconciled SCORE/TTS patches; add standalone compatibility coverage. Assisted-by: Codex:gpt-5 * feat(gallery): add native decision family defaults Pin Laya, Kev-4B, lev, OpenJev and Nimble artifacts. Verify Laya/Kev/lev gallery installs and CPU contracts on both native pins; clearly mark OpenJev/Nimble runtime validation pending and their noncommercial licenses. Assisted-by: OpenAI * docs(decisions): clarify integrated Nimble prerequisite Record the exact combined backend pin while retaining pending OpenJev and Nimble installation/runtime validation status. Assisted-by: Codex:gpt-5 * fix(gallery): indent native decision model sequences Match repository yamllint indentation for Laya, Kev, lev and OpenJev list fields. Parsed gallery data is unchanged; reproduce CI gallery lint failure before the whitespace-only fix and pass the same command afterward. Assisted-by: Codex:gpt-5 * docs(decisions): record OpenJev and Nimble CPU validation Record gallery installation, checksum/metadata verification and multiquestion native smoke results on bed0a856. Retain noncommercial and text-only limitations without accuracy or deterministic-output claims. Assisted-by: OpenAI * fix(ui): expose native Decisions router classifiers Select classifier models using metadata-driven capability routing, retain tuned thresholds, and validate native decision selections before saving. Cover both native backends and create/save/reopen in the real React editor. Assisted-by: Codex:gpt-5 * fix(router): exclude aliases from native decision discovery Check the originally named config before advertising native Decisions eligibility. Retain target capability inheritance for ordinary generation aliases. Exercise the actual capabilities endpoint with native models on both backends, aliases, and disabled models. Assisted-by: Codex:gpt-5 * feat(systemone): share bounded multimodal input validation Preserve text wire limits while admitting bounded PNG/JPEG decision input. Share collection and header validation across internal and public callers and keep the native runner response budget independent. Assisted-by: OpenAI:API-assistant * fix(systemone): bound admission lifetimes and validate complete images Retain shared admission leases through actual work completion, including abandoned internal operations. Decode bounded image pixels, cap public native responses before usage stamping, and preserve oversized malformed text status precedence. Assisted-by: OpenAI:API-assistant * fix(router): classify images before media fetching Preserve ordered structured probes for native decisions. Defer OpenAI media preparation until routing selects the served model, so rejected decision URLs cannot trigger downloads before shared validation. Guard direct image collection with context-aware shared admission. Keep text classifiers and embedding caches from discarding image input. Retain fail-closed classifier configuration and runtime fallback policy. Add middleware, typed-content, admission, cancellation and cache tests. Assisted-by: OpenAI:API-assistant * fix(router): bound extraction before serialization Check probe budgets before copying text or marshaling message state. Count JSON escaping so oversized internal inputs fail before allocation. Preserve typed Anthropic blocks through selected-model conversion and fallback. Keep retry coverage in Ginkgo without global test registration. Assisted-by: OpenAI * fix(router): bound supported probe serialization Arbitrary structs can bypass the probe budget through pointer marshalers, string tags, and promoted fields. Accept concrete chat schema types and plain JSON values instead of emulating arbitrary struct serialization. Budget escaped direct prompts before marshaling so raw length cannot hide serialized expansion. Preserve runtime fallback and reject oversized input before invoking the decision runner. Add Ginkgo allocation, boundary, and marshaler invocation regressions. Six-package tests, three-package race tests, and full-T2 delta lint pass. Assisted-by: OpenAI:GPT-5 golangci-lint * feat(decisions): enable bounded OpenJev images Validate native decision images before permissive media parsing and pixel allocation. Require both decision image support and a vision projector; missing or audio-only projectors cannot silently become text decisions. Pin the OpenJev Q8 projector and document its license and disk footprint. Add native safety tests, canonical limit parity, gallery and load-option checks, and a reproducible CPU direct-RPC contrasting-image smoke. Assisted-by: OpenAI:GPT-5 * fix(decisions): reject incomplete image streams stb accepts corrupt PNG Adler checksums and truncated JPEG scans. Use bounded zlib validation and strict libjpeg decoding before parsing. Keep dimension and aggregate pixel checks ahead of decoder allocations. Wire decoder dependencies into native builds and runtime packaging. Add regressions for appended EOI and embedded marker bypasses. Assisted-by: OpenAI:GPT-5 * fix(ci): gate native decision image validation Run the decoder security tests outside the stdlib-only native suite. Fetch vendor headers at the backend pin and provision decoder dependencies. Gate Go limit parity and production CMake wiring without model downloads. Assisted-by: OpenAI:GPT-5 * test(decisions): cover multimodal public API paths Exercise shared image contracts through the registered HTTP routes and external mock backend. Add opt-in cached gallery installation and real OpenJev image decisions through SystemOne and both routing APIs. Assisted-by: Codex:gpt-5 * test(decisions): assert isolation and cache bypass Observe external RPC calls and compare complete classifier history. Winner-only and cache-miss checks could hide dropped history or cache use. Give real inference its own application and model directory so shared backend mappings and loaded processes cannot affect mixed suite order. Assisted-by: OpenAI:ChatGPT * test(decisions): isolate fixture globals Disable optional global services in the isolated HTTP fixture and register cleanup before setup assertions. Verify meter provider identity survives fixture creation and destruction. Snapshot observed usage before assertions so failures cannot retain the mutex. Require a successful usage stamp before checking error responses. Assisted-by: Codex:gpt-5 golangci-lint * fix(application): honor optional telemetry controls Skip failover gauge registration when metrics are disabled. Register against the application meter rather than looking up the global provider. Allow embedders to retain the bounded routing log without billing stats. Keep the existing default when stats are disabled. The isolated HTTP fixture uses this option without losing its native router assertions. Assisted-by: Codex:gpt-5 golangci-lint --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler-agentandEttore Di Giacinto
|
e4fa051ee3 |
refactor(distributed): put the NATS-only paths behind interfaces (#12395)
* feat(messaging): add shared subject rules Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(messaging): cover BroadcastRoots, ControlRoots and SubjectRoot Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(messaging): add Broadcaster and enforce subject rules in every carrier Broadcaster is the fan-out half of MessagingClient. The NATS client and the in-memory FakeBus now refuse a subject outside the served roots and any wildcard other than a whole single token, and FakeBus shares MatchSubject instead of its own copy. FakeBus Unsubscribe now removes its own subscription instead of the first one with the same subject. A shared conformance suite in messagingtest runs against both carriers. The distributed e2e specs that used invented test.* subjects, and the one that subscribed with a > filter, now use subjects from subjects.go. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: depend on Broadcaster where only publish and subscribe are used Narrowed to messaging.Broadcaster: nodes/staging_progress.go, nodes/install_progress_publisher.go, galleryop/operation.go, galleryop/service.go, agentpool/user_services.go, agentpool/agent_jobs.go, openresponses/store.go, openresponses/sync.go, syncstate/syncstate.go, finetune/service.go, quantization/service.go and failover/distsync/distsync.go. SubscribeJSON now takes a Broadcaster because it only calls Subscribe, which lets the narrowed consumers use it. Stayed wide: worker/supervisor.go, because its client field also serves the SubscribeReply handlers in worker/lifecycle.go. The request/reply, queue and wiring files (nodes/unloader.go, nodes/file_stager_s3.go, jobs/dispatcher.go, agents/dispatcher.go, agents/events.go, worker/file_staging.go, cli/agent_worker.go, http/app.go) are unchanged by design. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): name the no-route condition and confine the carrier error Consumers matched nats.ErrNoResponders, which names an absence, to demote a node. They now match ErrNoRoute, the control path maps the carrier's failure onto it, and timeouts and worker refusals are pinned as not being no-route. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(nodes): state which FileStager implementations return ErrNoRoute Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): build backend clients through one node-aware seam Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: describe the distributed transport seams Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: correct comments that overclaim after the seams refactor Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(agent-worker): refuse an unserved LOCALAI_AGENT_SUBJECT at startup The messaging client now refuses a subject whose root no carrier serves. An agent worker started with a custom LOCALAI_AGENT_SUBJECT such as tenant-a.agent.execute used to start and then wait on a subject the frontend never publishes to. After the subject rules landed it exited at subscribe time with an error that did not name the setting. Behaviour change: the worker now checks LOCALAI_AGENT_SUBJECT before it registers or connects, and exits with an error that names the variable and says to use a served subject under the agent root, for example agent.execute. The served roots are not widened: a custom root was never delivered by the frontend, and a wider set would reopen the drift the subject rules exist to close. The flag help and the agent worker docs state the constraint. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(nodes): pin the reactions to ErrNoRoute Three callers react to ErrNoRoute and had no spec: the reconciler's upgrade drain falls back to the legacy forced install, the reconciler marks the node unhealthy when a pending op has no route, and the backend-op fan-out marks the node unhealthy. Each spec drives the real caller with a scripted no-responders reply and reads the result from the registry or the recorded requests. A fourth spec pins the other side: a pending op that times out leaves the node healthy and only counts the attempt, so mapping timeouts onto ErrNoRoute would fail here. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(messaging): pin client subject checks and fail the carrier suite in CI Add specs that call Publish, Request, Subscribe, QueueSubscribe, SubscribeReply and QueueSubscribeReply on a client with no connection. Each call must return ErrUnservedSubject for bogus.thing and ErrUnsupportedWildcard for jobs.>. This proves that the subject check runs before the connection is used, and needs no server. The NATS conformance suite is the only check that runs the subject rules against a real carrier. Before this change it skipped without output when Docker was missing. Now it fails when CI is set, so a Linux runner without Docker cannot hide it. It still skips on local runs and on macOS CI, which has no Docker. Add SubjectNodeBackendInstallProgress to the list of constructors that must build served subjects, and ask contributors to extend the list. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: state what ErrNoRoute may change, and group the distributed guides The seams note said MarkUnhealthy was the only state change allowed on ErrNoRoute. A pending backend op still records the failed attempt, counts toward the reconciler's retry limit and is dead-lettered after the maximum attempts. The note now says that MarkUnhealthy is the only change to the node's own state, and that the per-op accounting is not a verdict about the node. The note also documents that the NATS conformance run fails under CI when Docker is missing. The distributed-seams row moves next to the distributed-state row in the topics table. The liveness ping spec header now says no route is a reason to skip the worker, not proof that the worker is gone. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): give the backend client factory the node id Mechanical: the method gains a nodeID parameter and the eight test fakes are updated. No behaviour change. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): drop the optional node-aware factory The node id is now in the main method, so the optional interface and its helper had no behaviour of their own. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): dial backend probes through the client factory Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): dial workers' file servers through a per-node dialer Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(http): proxy backend logs through the per-node worker dialer The admin backend-logs proxy (list, lines and the WebSocket stream) now reaches a worker through the same per-node dialer as the HTTP file stager, so every frontend-to-worker dial goes through one seam. The shared direct dialer keeps alive for 15s where the proxy used 30s. Harmless for requests bounded at 15s. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(http): keep the backend-logs proxy independent of the admin connection The proxy request had no context before the dialer change and is bounded only by its 15s timeout. Keep it that way. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: move the worker control payloads to workerctl Mechanical move of the request and reply structs, the install progress event and the file payloads out of messaging. The verbs no longer belong to one carrier. No alias is left behind. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(worker): serve the lifecycle verbs through a controlServer The worker registers one handler per verb and a NATS server maps each verb to its subject. Registration errors now name the verb. node.stop is served with SubscribeReply, which is identical on the wire because the handler never replies. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(worker): report install progress through the control sink Install and upgrade now emit download progress through the sink the control server hands them. The debounce and the terminal flush stay in the handler path, built over that sink by the new nodes.NewDebouncedInstallProgressSink, which replaces NewDebouncedInstallProgressPublisher. The subject and payload on the wire are unchanged. The supervisor no longer holds the bus, and installFn and upgradeFn let specs drive both verbs without a gallery. The malformed-request log lines are restored for install, upgrade, backend.delete, model.unload, model.stop and model.delete, with the reply bytes unchanged. The signal adapter is renamed noReply, which also lets worker.go import os/signal without an alias again. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(worker): serve the file-staging verbs through a controlServer An empty list-dir answer is now {} rather than {"files":null}, because the typed reply omits an empty Files slice. The frontend decodes both to a nil slice in nodes/file_stager_s3.go. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(messaging): add WorkQueue and the NATS producer This is the producer side of the competing-consumer seam. The work kinds map one to one to today's subjects and queue groups: task to jobs.new and mcp-ci to jobs.mcp-ci.new (both in group workers), agent-run to agent.execute (group agent-workers). Enqueue publishes the payload as Publish does today, with one JSON marshal. FakeBus now records queue groups and keeps reply handlers so later specs can pin and drive them. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(messaging): add the NATS WorkConsumer An in-flight limit of one runs the handler inline on the delivery goroutine, as the MCP CI consumer does today. Any other limit spawns per delivery, as the agent consumer does. Queue groups are unchanged. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: publish queued work through WorkQueue The job dispatcher, the agent pool and the agent scheduler enqueue through messaging.WorkQueue; the NATS implementation publishes to the same subjects as before. DistributedServices builds the queue next to the NATS client and hands it to the dispatcher and the agent pool, whose distributed mode switch now reads a non-nil WorkQueue. The unused AgentPoolService.SetNATSClient is removed. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: consume queued work through WorkConsumer The agent dispatcher and the MCP CI consumer register through messaging.WorkConsumer. The NATS implementation keeps the inline one-at-a-time model for MCP CI and the per-delivery model for agent runs. handleMCPCIJob reports on the events publisher the carrier hands it instead of a captured client. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: delete the consumers nothing in production reached jobs.new has a producer and no production consumer, and the agent dispatcher's Dispatch was only called from tests. Publishing jobs.new is unchanged. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(mcp): send MCP requests to agent workers through AgentControl Timeouts still honour only the deadline, not cancellation, exactly as today. The NATS no-responders error maps to ErrNoRoute and a timeout does not. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(agent-worker): serve MCP requests and backend.stop through agentRPCServer The agent worker's MCP tool and discovery reply subscriptions and its backend stop listener move behind an unexported agentRPCServer interface, served on NATS by nodes.NATSAgentRPCServer. The handlers become typed mcp.ToolHandler and mcp.DiscoveryHandler values that answer every failure with a reply carrying Error. Queue group (agent-workers), inline execution on the delivery goroutine, the background handler context, the unmarshal error reply texts and the reply-less backend stop subscription are unchanged. The backend stop handler takes the decoded backend name, so it can still close that backend's MCP sessions. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(messaging): remove helpers that only tests used BroadcastRoots, ControlRoots and SubjectRoot had no production caller. The roots spec now asserts every served root through ValidateSubject instead. MatchSubject moves back into the test support package, the only place that used it, with its table. NATSAgentRPCServer drops the subscription list it stored and never read, and NewNATSAgentRPCServer gets a doc comment. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(mcp): round trip the agent RPC server over a real NATS server One spec sends a tool request and a discovery request through NATSAgentControl to NATSAgentRPCServer and checks that the handlers see the decoded requests and the replies come back. It also puts an undecodable body on the tool subject and checks the server answers with an unmarshal error instead of leaving the requester to time out. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: describe the distributed transport seams The developer note now lists the final seams: fan-out, queues, both halves of the control verbs and of agent RPC, and the dial. It records the open items a second carrier has to handle. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test: pin the in-flight limit each queue consumer asks for The work queue specs pin what Consume does for a given limit, but nothing pinned which limit each production consumer passes. Changing the agent worker's MCP CI limit from 1 to 0 would have let MCP CI jobs run concurrently on each worker with every test green. Move the MCP CI Consume call into startMCPCIConsumer with the same wiring and pin that it asks for (WorkMCPCI, 1). Pin that NATSDispatcher.Start asks for (WorkAgentRun, maxConcurrent) for several limits. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: remove helpers the branch left without a caller SubjectJobCancelWildcard lost its last subscriber when the frontend stopped listening on jobs.*.cancel; the NATS permissions and conformance suite spell the subject out, so nothing reads the constant. decodeBackendStopRequest returned a stopAll flag that production dropped and only a test read. decodeBackendStop is now the single decoder with the same semantics: an empty body is stop-all, an empty Backend is stop-all, malformed JSON is an error. stopBackends still derives stop-all from Backend, so no reply changes. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(messaging): keep an explicitly empty agent queue a plain subscription Before the work queue seam the agent worker passed LOCALAI_AGENT_QUEUE straight to QueueSubscribe, so an explicitly empty value made a plain subscription and every agent worker ran every agent run. WithAgentRunRoute replaced an empty queue with agent-workers, which silently changed that. Keep the queue as given once the option is applied. An empty subject still falls back to agent.execute, since it never had a meaning of its own. The flag default stays agent-workers, so only an explicitly empty value reaches this. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: correct comments and record the PR B notes Fix the recordingFactory comment (it also records the parallel flag), document that a negative maxInFlight is unbounded and that Unsubscribe from a handler deadlocks, and say a permanently undecodable payload returns nil. Record controlHandler's undecodable return as a kept exception, and add the second carrier notes to the developer note: the reconciler has no ClientFactory option, the logs proxy honours HTTP_PROXY, verbs one carrier serves need an opt-out, terminal replies come from the result event, and agent runs publish through the NATS-bound EventBridge, which is not an additive change. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
d30c33a074 |
feat(failover): run one prober per cluster and pin warm targets on workers
Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-bot
|
e62854c340 |
fix(failover): pass credential lookup from CLI
Pass the API key environment lookup through ApplicationConfig to satisfy core configuration lint. Keep credential resolution dynamic and exclude the callback from serialization. Handle the five close results reported by errcheck. Assisted-by: Codex:gpt-6 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
5d2b90b848 |
fix(failover): never load a warm target inside a probe
A warm target's liveness probe called ModelLoader.Load, which blocked until the model finished loading (while the warm preload loaded it too). Tick waited for every probe, so all probing froze, and the probe then ran HealthCheck on an expired context and tripped the target at every startup. The prober now takes a function that returns the running backend without loading it. A target that is not loaded passes liveness; its recovery is neither confirmed nor failed and it returns to healthy after min_dwell, like a cold target. Tick no longer waits for probes: each probe applies its own result and a target whose probe is running is skipped. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
191d2d6301 |
feat(failover): add MCP tools to list chains and pin targets
Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
3e8ca016ee |
fix(failover): stop judging cold local targets by their model file
A missing file is not a reliable signal: models download on first use, some backends need no file, and dotted names like Phi-3.5-mini look like paths. Marking such a fallback down removed the retry a chain exists for. Cold targets are now judged only by real requests. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
5ea35393fa |
feat(failover): run the chain manager and keep warm targets loaded
Warm local targets are pinned in the watchdog and preloaded. Switches and target health are exported as metrics, skipped attempts as traces. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
505a6d040b |
fix(distributed): publish the revision a request actually carries
Two code paths computed a model's revision. Inference resolves the config through the loader, which applies SetDefaults a second time. Everything that publishes a revision hashed the stored config instead, with SetDefaults applied once. SetDefaults is not idempotent for every model: it re-runs the GGUF guess and the hardware defaults, both of which read state the stored config does not carry. Where the two disagree, a publisher wrote a revision no request would ever carry, and the model became unroutable the moment it was published. On this cluster the startup resync republished one such value and every request for that model was then rejected against it. The publishers now resolve the revision through the loader, exactly as a request does, so there is one definition rather than two that agree only when SetDefaults happens to be idempotent. This covers the startup resync, a saved config edit, and enabling or disabling a model. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
Ettore Di Giacinto
|
5c9d8190d9 |
fix(distributed): resync revisions after the configs are loaded
The resync added in
|
||
|
Ettore Di Giacinto
|
3953448f60 |
fix(distributed): resync stored config revisions at startup
The controller pins a model's replicas to a stored revision and rejects any request carrying a different one. Nothing ever re-derived that value from the configuration on disk: it moved only on an edit, a gallery install, or a peer's change broadcast. An inference request may only establish a revision, never replace one. So any other way for the two to diverge left the model permanently unroutable. A configuration edited while a frontend was down lands there, and so does a change in what the revision is computed over: an upgrade that alters the hashed form leaves every stored revision describing a configuration that no longer exists. The only recovery was deleting the row by hand, which is not something a cluster should need. Each frontend now reconciles the stored revisions against the loaded configurations at startup and republishes the ones that disagree. Only those: republishing quarantines every replica loaded under the old revision, so doing it for a model that did not drift would unload a healthy replica for nothing. A model with no stored revision has never been served and is left for its first request to establish. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [golangci-lint] |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
8f56e4e042 |
fix(vram): persist remote probe metadata (#11487)
* fix(vram): persist remote probe metadata The startup warmer repeated remote size and GGUF metadata probes after every restart because both caches lived only in memory. Store successful HTTP probes for 24 hours so frequent restarts reuse the prior results. Bound the cache, reject invalid records, and purge it when gallery data changes. Local model files continue to bypass persistence. Assisted-by: Codex:gpt-5 * fix(vram): check temporary file cleanup The lint gate rejects the unchecked cleanup call in the persistent cache writer. Assisted-by: Codex:gpt-5.6 [golangci-lint] * fix(vram): make persistent cache optional Remote metadata probes can transfer enough data that operators need control over disk reuse and startup warming. Gallery autoload now gates both behaviors, and the runtime setting applies changes immediately. Assisted-by: Codex:gpt-5 * fix(ui): expose gallery startup pre-warm The existing gallery autoload setting also gates the startup metadata warmer. Name both effects in Settings so operators can find the requested boot control. Assisted-by: Codex:gpt-5 --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
82c191afad |
fix(distributed): keep model replicas config-consistent (#11664)
* docs: design configurable copy buffering Document the context-aware copy buffer option and its validation plan. Assisted-by: Codex:gpt-5 * docs: design durable distributed staging operations Assisted-by: Codex:gpt-5 * docs: design distributed model config revisions Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] * feat(config): add stable model revisions Hash typed model configuration and effective protobuf options deterministically for distributed revision comparisons. Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] * feat(worker): acknowledge exact model stops Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] * feat(nodes): track model config revisions Assisted-by: Codex:GPT-5 [apply_patch] * fix(distributed): retry quarantined model cleanup Stop quarantined replicas by exact process identity, retain failed cleanup as durable capped retries, and compare-and-delete only the claimed registry row. Process one sufficiently leased row at a time so multiple frontends cannot duplicate slow cleanup work. Assisted-by: Codex:gpt-5 * fix(distributed): bind loads to config revisions Assisted-by: Codex: GPT-5 [OpenAI Codex] * fix(modeladmin): apply config revisions consistently Route model edits, patches, state changes, deletion, and peer refreshes through the same revision lifecycle. Quarantine stale replicas before exact cleanup and report durable pending cleanup without failing successful config writes. Assisted-by: Codex: GPT-5 [OpenAI Codex] * feat(distributed): expose model config revision state Document replica revision observability and durable cleanup behavior. Keep pending cleanup explicit in model mutation responses and verify endpoint contracts expose revision state without serialized load options. Assisted-by: Codex:GPT-5 [OpenAI Codex] * test(distributed): cover model revision convergence Exercise cross-frontend quarantine, stale replay rejection, exact cleanup retry, worker re-registration, and current-generation replica convergence against the distributed PostgreSQL harness. Assisted-by: Codex:gpt-5 * fix(distributed): pass config revision CI checks Keep configured gallery sources out of authoritative runtime snapshots only after validating their real schema, and harden rollback snapshots against symlink races and non-regular files. Assisted-by: Codex: GPT-5 [OpenAI Codex] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
74b7ea2829 |
feat(ui): replace the gallery and inventory tables with a rail and a detail pane (#11288)
* feat(ui): rename the Install Models nav entry to Discover "Install Models" named the action rather than the destination, and it was the only multi-word entry in a rail of one-word ones (Home, Chat, Studio, Talk, Build, Operate). A bare "Models" was the obvious fix but it collides with the installed-models view under Host, which is a different page for a different job. "Discover" keeps the rhythm and says what the page is for. The icon moves from a download arrow to a compass for the same reason: the page is browsed before it is installed from. Translated in all seven locales rather than left to fall back, so a locale switch does not leave the entry in English. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * feat(ui): replace the gallery table with a rail and a detail pane The eight-column table was not the real problem; the click-to-expand row underneath it was. Variants, files and a VRAM estimate never fitted inside a <tr>, so they were pushed into a drawer that could hold one model at a time, could not be linked to, and had no room to say anything useful. The gallery is now a rail to scan and a pane that answers. The pane has two states and no third: with nothing selected it is the discovery page, and with a model selected it is that model's detail. Selection lives in the URL, so a model is linkable and Back steps out of the detail instead of off the page. The rail groups by capability while browsing and flattens to results the moment a term is typed. That is a rule rather than a toggle: once someone has said what they are looking for, the buckets are between them and the answer, and making the user choose would be handing them our problem. The detail pane plots VRAM against context length with the host's own limit drawn across it. This is new information, not a restyle. A single number invites "so will it run?", and the honest answer is usually "yes, up to a 32k context", which is a shape rather than a number. The estimates were already fetched for every context size, so it costs no new request. Backends that take no context length say so instead of being given a meaningless chart, and a host with no GPU gets no chart at all rather than bars with nothing to compare against. The split-button variant menu goes with the actions column. The pane lists every build with its backend, quantization, size, fit and a details disclosure, each installable, which is what the dropdown was a cramped substitute for. Its tests move onto that list; the three contracts it alone carried (fetch-once caching, the loading state, an unfit build staying installable) are backfilled against the pane. RecommendedModels moves inside the pane, where it has the width to argue for a model instead of listing one, and keeps its own dismissal and collapse. Rail entries carry no description. Two lines is the budget and the second is better spent on whether the thing will run; the stripped-Markdown contract moves to the pane's lede, tooltip included. e2e: 123 passing across models-gallery, navigation, recommended-panel, model-artifact-operation, operations-strip and page-render-smoke. Inline styles in Models.jsx drop from 82 to 41. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * refactor(ui): extract the split view into shared components Discover shipped its rail, pane and detail header as private functions inside Models.jsx. Backends and Host have the same defect and want the same shape, so leaving them there guarantees three rails that drift. SplitView, EntityRail, DetailHeader and StatGrid now live under components/split/. EntityRail is deliberately data-driven: a surface maps its own entity onto { id, name, icon, meta, stripe, groupId } and keeps its vocabulary to itself, which is what stops the rail learning about models, backends and loaded state all at once. The CSS moves with it. What was .discover__rail is .entity-rail, .discover__ pane is .split-view__pane and so on, because a class named after one page is a lie on the next two. Only what is genuinely Discover's stays behind the old prefix: the shelves, the hero and the VRAM-by-context chart. Two additions the shared rail needs and Discover did not: a state stripe, for surfaces read by condition before they are read by name, and an empty label. Discover passes neither. No behaviour change. e2e 100 passing across models-gallery, navigation and models-recommended-panel. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * feat(ui): put the backend gallery on the split view Same defect as the model gallery, so the same shape: a seven-column table over a click-to-expand row that was the only place the repository, licence, tags and links could go. The rail groups backends by the use case they serve, sharing Discover's taxonomy on purpose: a backend is the runtime a use case needs, so "vision" ought to mean the same thing one level down. It flattens on a query for the same reason it does on Discover. The zero state is the one real departure. A backend's fitness is not free memory, it is the accelerator and platform it was built for, so the pane leads with what this host is, then what is not installed yet, then whether anything installed has gone stale. The table listed 37 runtimes and left "which of these can even run here" entirely to the reader. Distribution moves into the pane, which is the one thing a row could never carry: which nodes hold a copy and which do not, with the install-on-more control next to it rather than squeezed against a chip. The distributed and target-node action logic is unchanged, including the guard that keeps a hardware-specific build off the fan-out path. The split-button popover loses its per-row anchoring because there are no rows; one pane, one anchor. Selection lives in ?backend=, preserving the ?target= scope rather than clobbering it. e2e: 139 passing across models-gallery, navigation, backends-management, models-recommended-panel, nodes-per-node-backend-actions, page-render-smoke, operations-strip and model-artifact-operation. The backends spec gains six split-view tests; its three description-cell tests move onto the pane lede. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * feat(ui): put the Host inventory on the split view The last of the three surfaces, and the one that is not a catalog. Both tabs had the same click-to-expand row, so the shell transfers; what does not transfer is the zero state, because there is nothing to discover in your own inventory. With nothing selected the pane reports what is happening: how many models are loaded, what failed, what has an update, and which models are holding VRAM right now. Every number was already on the page. None of them had been assembled into one statement, so "what is going on" was a question the tabs could not answer however long you looked at them. The rail buckets by state rather than capability - Running, Idle, Disabled for models; Update available, Installed for backends - which is the opposite of the galleries and deliberately so: nobody opens Host wondering which of their models does vision. Entries carry a state stripe for the same reason. Load and Stop are promoted out of the kebab, because that is what an operator came for; the rest stays behind the menu rather than diluting it. Adopted, pinned and alias badges follow the model into the pane: they are facts about the thing, not about its state, and the rail line is spent on state. Deliberately NOT done: folding the two tabs into one rail, as the mock had it. It costs five URL parameters, the manage-tab localStorage key and the stat-card shortcuts, all of which are live deep-links today. The tabs stay as the group selector; merging them is a follow-up with its own migration. e2e: full suite 355 passing. New host-split-view spec; alias-template, manage-logs-link, manage-action-menu-position and model-editor-back-nav move off `.table` and the row kebab onto the rail and the pane. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * polish(ui): accessibility and consistency pass over the three split views Findings from a pass over what the previous four commits actually shipped, rather than what they were supposed to. The rail was not a listbox. ARIA lets a listbox contain options and groups, and nothing else, but each group's collapse control is a button that has to sit inside the scroller with the entries it folds. It is now a labelled group of buttons, which is the honest description; selection is announced with aria-current and the arrow keys are unaffected. Every entry was its own tab stop, so tabbing past a forty-entry rail to reach the pane took forty keystrokes. Roving tabindex makes the rail one stop, and arrowing now moves focus with the selection instead of leaving it behind on an entry Tab can no longer reach. The rail rounds its corners with overflow:hidden, which was clipping the focus ring off the first and last entries entirely. Inset outlines fix it. A 30px row is fine under a mouse and too small under a thumb, so coarse pointers get a 44px target without costing density on a desktop. One slot said three different things: "9 models loaded" on Discover, "12 loaded" on Backends, "3 of 9" on Host. All three lists are a page of a larger set, so all three now say so the same way. Also removed: an emptyLabel prop on EntityRail that nothing passed, its dead CSS rule, and MODELS_COLSPAN and ResourceRowDesc, which died with the tables. e2e: full suite 355 passing. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(ui): correct three defects only a real gallery exposed Running the branch against a live instance with 1,595 models and 1,017 backends, rather than against mocked fixtures, surfaced three things the e2e suite could not. Grouping did nothing. The rails matched on the use-case keys the filter chips send (`chat`, `tts`, `transcript`), but those are a server-side vocabulary the handler maps onto entries. What entries actually carry is free-form and inconsistent: models come back tagged `llm`, `gguf`, `vision`, `coding`, and backends `LLM`, `text-to-text`, `audio-transcription`. Nothing matched, so every model landed in "Everything else" and the feature was decorative. Grouping now lives in utils/entityGroups.js, shared by both galleries, matching case-insensitively against the vocabulary the API really uses, with the entry's backend as a fallback signal - a backend named `whisper` is a speech backend whatever its tags say. Order is specific before general and that is load-bearing: a vision model is tagged `llm` too, so testing text first would swallow it. The zero state claimed GPU memory on a machine with no GPU. The resources endpoint reports system RAM in the same field when gpu_count is 0, so the hero read "84.4 GB of GPU memory" next to the recommendations panel correctly saying "No GPU detected". The number was never wrong, only its label; it now says system memory unless a GPU is actually present. The page title still said "Install Models" under a nav entry saying Discover. Also: the keyboard test named the model it expected to arrive at, which made it a hostage of the grouping table and broke the moment the buckets were fixed. It now asserts that the selection moves and returns. e2e: full suite 355 passing. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(ui): the filters and the rail were fighting over the same job Four things you find odd on Discover, and they turn out to be one mistake seen from four sides. The rail grouped the current page. The listing is paginated at nine rows, so those bucket headers described nine entries out of 1,595, and turning a page reshuffled the sections under the reader. The structure was never stable because it was computed over the wrong set. The chips were redundant for the same reason, seen from the other side. They send tag= and filter all 1,595 server-side. The rail grouped nine of them client-side by the same axis. Two controls for one job, and the weaker one was the one this branch added, so it goes. Grouping stays only on Host, where the list is complete, local, and bucketed by state rather than capability. The search bar felt odd because it sat in a full-width band while the thing it narrowed was a 290px rail below and to the left. The whole band now lives in the rail column: search, backend, use cases, refinements, then the list it narrows. One column to say what you want, one to show what you got. Nineteen chips do not fit at that width, so they fold into a disclosure that states the selection. A disclosure and not a popover, deliberately: picking use cases is multi-select and interleaves with the backend select and the toggles below, and a popover dismisses itself the moment you touch either. The header held two counts and two buttons at arm's length from all of it. The counts were the third statement of the same number on one screen, after the rail's "9 of 1,247" and the pane's own headline, so they go. The buttons move into the pane's zero state, which is the surface that answers "what do I do here". Also: the two first-run empty states wore .loading-center, which is display:flex in the default row direction because it exists to centre one spinner. With four children that put the icon, the heading, the sentence and the buttons on a single line with no gap. They are now a proper full-height empty state. e2e: full suite 353 passing. Grouping tests are replaced by ones asserting the rail stays flat; chip tests open the disclosure first; two filter-layout tests that asserted the old three-band arrangement now assert the column. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * polish(ui): make Discover a full-height view, group the chips, name the refinements Four things, all of them the same complaint: the page read as a document with controls scattered on it rather than as one view. The header is fused. A title block with its own padding, a subtitle and two counts made the split view look like an attachment to a document that happened to sit below it. It is now a slim bar carrying the title, the count and the two page-level actions, and the split fills the rest of the window. Rail and pane scroll independently, so the filters and the pane's headline stay put while a long list moves under them. The chips group. Nineteen in a flat row is a lot to scan even behind a disclosure, and they already belong to the four families the rest of the UI speaks, so they are bucketed by those. "All" sits on its own above them without a heading, because it is a reset rather than a use case. The refinements stop looking dumped. When the band became a column they were three controls left where they landed; they now read as a named section with one control per row. The zero state suggests again. It had decayed into a "Browsing / 9 of 1,247 / select a model" line that restated the count for the third time on one screen. It now offers the four use cases as tiles that set the filter, which is the shelf idea from the mock without inventing curation or paying for a second fetch. Two bugs found by looking at it rather than at the tests: the disclosure was clamped to 190px, which cut it off partway through its third section so two of the five never appeared at all; and the creation actions rendered twice, once in the new bar and once in the pane hero a few pixels away. e2e: full suite 353 passing. The chip-row test now holds its contract across the per-family rows rather than a single one, and additionally asserts every family is present and non-empty. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(ui): pin the split view's height so a long detail scrolls the pane Selecting a model with a long description grew the whole page and dragged the rail down with it, which is the opposite of what "full height" was supposed to buy. The flex chain was right and the ceiling was missing. .app-layout and .main-content are min-height:100dvh, which is a floor: flex distributes free space but nothing caps growth, so a pane taller than the viewport expanded the column, the document scrolled, and the rail stretched to match. height:100% on the pane then resolved against an auto-height parent and did nothing. The chat route already solves this by pinning .main-content to 100dvh. The same treatment now applies to any route containing a .page--app, selected with :has() so the shell does not have to learn which pages happen to be split views. Below the stacking breakpoint the pin is lifted, because two stacked halves in two short scrollers is worse than a page that scrolls. Measured on a live instance: document height stays at the viewport across selection (950px either side) and the pane overflows internally instead. Adds discover-height.spec.js, which asserts the page height and the rail height are unchanged by selection and that the pane is the thing that scrolls. The existing specs could not have caught this: they mock short descriptions, and the bug only appears when the pane has more content than the viewport holds. e2e: full suite 355 passing. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * feat(ui): give Backends and Host the full-height view, and fix the Update button Backends now matches Discover: the header fuses into a slim bar carrying the title, the count and the page-level actions, the filters move into the rail column where they narrow the rail and nothing else, and the split fills the window. Its seven chips fit at rail width, so unlike Discover's nineteen they need no disclosure. Host gets the bar and the height; its resource monitor, summary cards and tabs stay above the split, because those are read once while the rail and the pane are worked in. Two things the height change surfaced. The console layout is a flex row with align-items:flex-start, so its body sizes to content. Right for the pages it was built for, wrong for a split view, which needs a ceiling to scroll inside: without it the Backends rail ran past the viewport and over the footer. Pinned with :has() so only split-view routes are affected. The filters vanished when nothing matched. Both galleries swapped the whole shell for an empty state, which took the search box and the chips with it, so the page said "try adjusting your search or filters" while offering neither. The shell now stays and the empty state moves into the pane. Also fixes the Update control on Host, which had no className at all and rendered as bare text, next to a status span that had picked up btn classes and two copies of `fas` and so rendered as a button you cannot press. They have swapped appearances back. e2e: full suite 355 passing. The render-smoke selector learns .view-bar__title, since the pages it checks no longer all use PageHeader. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(ui): keep the view mounted while searching, and bring rail grouping back Searching replaced the whole view with a loader. The search box lives in the rail column, so every debounced refetch unmounted the field being typed into and dropped its focus with it. The list, the filters and the pane went too. The shell now stays and the rail says it is busy: a sweep bar under its header and the stale list dimmed, so the eye knows the answer is being replaced without losing its place. A cold start still gets the skeleton, because there is nothing to keep. The condition for that is "nothing has loaded yet", not "the list is empty". Those differ exactly when someone is editing a query that matched nothing, and getting it wrong there would unmount the view on the keystroke after a no-results search - the worst possible moment. Grouping comes back on both galleries. It was removed because nine rows could not fill five buckets, so a page turn rebuilt the rail's whole structure. That was a symptom of the page size rather than of grouping: the rail now asks for 30 rows instead of 9 (Backends 60 instead of 21), which is enough for the sections to read as structure and turns five times fewer pages. The order of the sections is fixed, so what changes between pages is membership, not arrangement. Grouped while browsing, flat while searching, as before: once a term is typed the buckets stand between the reader and the answer. Also gives GalleryLoader a class and a testid instead of six inline style declarations on a bare div, which is why nothing could select it. e2e: full suite 359 passing, including a new spec asserting the search box keeps its focus and its value across a refetch, and that a cold start still shows the skeleton. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * perf(gallery): stop invalidating the VRAM estimate caches on every request Searching or turning a page felt slow. It was not the search and not the listing: /api/models answers in 3-9ms. It was the VRAM estimate, which the gallery asks for once per row, and which took ~2.3s every single time however often the same model was asked about. pkg/vram already caches what makes that expensive - the remote content-length probes, the GGUF metadata reads and the HF repo sizes. Those caches key on a gallery generation counter, and AvailableGalleryModelsCached triggered a background refresh on every call, with each refresh bumping the counter. One page view is one listing request plus thirty estimate requests, each of which re-read the gallery and started another refresh, so the generation moved constantly and every cache entry was stale before it could ever be read. The caches were dead in production. Three changes, each doing one thing: A refresh interval. The cached list is still served immediately; this only decides how often re-fetching from upstream is worth starting. Five minutes, as a package variable so tests can drive it without waiting. A generation bump only when the gallery actually changed. An unchanged gallery re-fetched on schedule must not throw away work that is still valid, which is the difference between an estimate costing nothing and costing a network round trip. A separate "loaded" flag. The cache engaged on `cached != nil`, so a gallery that legitimately holds nothing read as never-loaded and took the blocking path on every call, bumping the generation each time. Found by the test for the interval, which could not pass while this was true. Measured against a live instance with 1,595 models: one estimate, repeated 2.3s -> 2ms a page of 30, in parallel 10s -> 0.04s A first, genuinely unseen model still costs its remote probe. That is inherent; what changed is that it is now paid once per model per gallery version rather than once per request. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * perf(gallery): warm VRAM estimates at startup, and stop the UI waiting on them Two halves of the same complaint: the gallery stalls on VRAM estimation. Server side, the estimates are now warmed in the background at startup. Estimating an entry nobody has asked about costs a remote probe of its weight files, and the gallery needs one per row, so the first visitor was paying for the whole page. The warm-up walks the gallery in the order the UI lists it, so the first page is ready before anyone reaches it. It is bounded and it never blocks: 300 entries at 4 at a time by default, on its own goroutine, stopping with the server's context. Warming the whole gallery would be thousands of probes on every boot, which is rude to the upstream and slow to finish; warming nothing leaves the first page paying two seconds a row. Anything past the limit still warms itself on first view. LOCALAI_VRAM_WARM_LIMIT=0 turns it off for an air-gapped host, LOCALAI_VRAM_WARM_CONCURRENCY=1 slows it for a metered link. Client side, the page no longer waits on estimates it does not need yet. It fired one request per row at once; a browser allows about six connections per host, so thirty estimates took every slot and the request behind a click - the variant list, an install - queued behind work nobody asked for. That is the freeze: the list was already usable, and the UI was busy fetching sizes. Four at a time leaves room for the interactive request to overtake, and a row whose estimate is still in flight says "sizing…" rather than leaving a blank where a number will appear. buildEstimateInput moves to core/gallery as EstimateInput, since the handler and the warmer both need it. Measured against 1,595 models, from a cold boot: page 1, 30 estimates in parallel 10s -> 0.04s full warm-up (299 of 300 entries) 3m, in the background Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * chore: untrack data/.local_user_id and ignore the runtime data dir `local-ai run` writes its instance state under ./data when started from the repo root, which is exactly what a contributor testing a build does. The identity file ended up committed on this branch by a `git add -A` while verifying the gallery changes against a live instance. Anchored, so it matches the runtime directory at the repo root and not a `data` directory nested inside some package. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
localai-org-maint-botandlocalai-org-maint-bot
|
1189c6825b |
feat(tracing): persist bounded trace histories (#11203)
* feat(tracing): persist bounded trace histories Retain API and backend traces below the data path, restore them at initialization, and serialize clears with asynchronous consumers. Assisted-by: Codex:gpt-5 * fix(tracing): satisfy persistence security checks Document why persisted filenames cannot escape the trace directory and explicitly ignore the best-effort temporary-file cleanup result. Assisted-by: Codex:gpt-5 [golangci-lint] --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
878a0d00a1 |
fix(distributed): reaper reaps live backends, ghost model stubs, in_flight leak, sidecar staging runaway (#11142)
* fix(distributed): stop the probe reaper from orphaning busy backends
The reconciler's liveness probe is a 1s gRPC HealthCheck, and a single
failed probe deleted the model's node_models row. A backend that is
merely busy cannot answer it: single-threaded Python backends (video and
avatar generation) block for minutes inside one request, so the reaper
was deleting registry rows for backends that were alive and mid-request.
The model then vanished from the nodes page while it was still
generating, and because the row was gone the in-flight decrement had
nothing to decrement ("DecrementInFlight: no matching row or already
zero"). Every subsequent request re-routed and re-staged the full model
from scratch.
Two guards:
- Replicas with in-flight requests are excluded in SQL. A row that is
actively serving is proof of life, and the running request is
exactly what stops the backend from answering the probe.
- Idle replicas must miss three CONSECUTIVE probes before removal, so
a transient blip cannot orphan a live replica. A successful probe
resets the streak.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(distributed): drop the local model stub when its last replica goes
In distributed mode every routed model leaves an in-process stub in the
frontend's ModelLoader, and DistributedModelStore.Range reports local
stubs UNION the registry rows. Every registry removal path deletes only
the DB row, so the stub outlived the replica and the model was reported
as loaded forever.
That is the "loaded on the home page, absent from every node" ghost:
/system reads the union and still sees the stub, while /api/nodes/models
reads the registry and correctly sees nothing. It never self-healed,
and both frontend replicas showed it independently.
The replica-removed chokepoint could not fix this as it stood, because
it held a SINGLE hook that the prefix cache already owned, and it was
registered only when the prefix cache was enabled. Registering a second
listener would have silently displaced the first.
- Turn replicaRemovedHook into a list (AddReplicaRemovedHook), so
independent subsystems can each register without displacing others.
- Add NewLocalStubInvalidator, which drops the local stub once no
healthy replica of the model remains anywhere in the cluster, and
wire it unconditionally in startup.
The stub is kept while another node still serves the model: the
frontend is right to consider it loaded, and each request re-routes
through SmartRouter to pick a live replica anyway.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(distributed): stop staging checksum sidecars back to workers
The file transfer server writes a "<file>.sha256" sidecar next to every
file it accepts. The sender walked the model directory with no filter,
so it staged those sidecars too, and the receiver duly wrote a sidecar
for each sidecar. Every staging pass multiplied the tree:
config.json -> config.json.sha256 -> config.json.sha256.sha256 -> ...
One LongCat snapshot had grown to 498 files, 466 of them chained, up to
29 levels deep, and the staged file count climbed on every pass. This
inflates each transfer and grows disk without bound on both ends.
Skip hash sidecars in stageDirectory, and mirror the skip in
countStageableFiles so the progress bar still reaches 100%. The check is
"a sidecar sitting next to a real file" rather than a blanket suffix
ban, so a model that genuinely ships a .sha256 payload with no
corresponding base file is still transferred.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(distributed): classify the liveness probe instead of gating on in_flight
The previous commit excluded replicas with in-flight requests from the probe
reaper. That was the wrong guard, and could invert the bug it fixed.
in_flight has no decrement guarantee: track() balances its increment with a
defer, but a frontend killed mid-request never runs it, and the load-time
reservation is released only when the first inference completes. Nothing
resets a leaked counter. Gating the reaper on it therefore meant a leaked
counter would shield a genuinely dead replica from ever being reaped.
Nor was patience alone a fix: three misses at the default interval is ~90s of
silence, while the generation that triggered this blocks for 15+ minutes.
The real conflation was in the probe itself. A gRPC HealthCheck against the
backend's serving port measures "is it idle enough to answer", not "does the
process exist", and probeLoadedModels discarded the error that tells them
apart. Because the gRPC client is lazy, the status code is decisive:
- DeadlineExceeded: transport fine, nothing serviced the RPC. Busy.
- Unavailable: nothing is listening. Gone.
ModelProber now returns a ProbeOutcome, and only ProbeUnreachable counts
toward the reap threshold. ProbeBusy clears the streak: it is evidence of
life. A blackholed network reads as busy too, deliberately, since whole-node
failure is the health monitor's job.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* feat(distributed): reconcile replicas against worker-reported processes
Probing a backend's own serving port cannot distinguish "busy" from "gone"
without inferring it from an error code. The worker can answer directly: it
spawned the process, holds the handle, and its reply is not blocked by
whatever that backend is doing.
Adds a models.running request-reply subject. The worker answers out of its
in-memory process table, reporting each live process as (modelID,
replicaIndex, address) — the supervisor's process keys are `modelID#replica`,
which is isomorphic to a NodeModel row, so the reconciler can diff the two
directly.
reconcileNodeProcesses runs before the port probe and reaps rows for models
the worker is not running. Models the worker vouches for get updated_at
bumped, which takes them out of the port prober's stale set entirely: that is
what keeps a backend deep in a long generation away from the probe in the
first place, rather than relying on classifying its silence after the fact.
A worker that does not answer is skipped, not assumed empty. A messaging
failure says nothing about the processes, and assuming the worst would delete
a node's rows on a transient NATS blip; the port probe stays as the fallback
for those nodes. Rows younger than probeStaleAfter are ignored so a freshly
created row is never judged against a process table that has not caught up.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
* fix(distributed): stop in_flight leaking and pin replicas against eviction
A leaked in_flight counter is not cosmetic. FindLRUModel,
FindGlobalLRUModelWithZeroInFlight and the router's eviction query all require
in_flight = 0, so a replica whose counter never came back is pinned and its
VRAM is unreclaimable for the lifetime of the process.
Two halves.
The source: routing reserves in_flight = 1 at load time so a freshly loaded
replica is not evicted out from under the request that caused the load. That
reservation was released ONLY by the first inference completing, so a route
torn down before any inference ran (client disconnect, handler error, failure
between load and the backend call) stranded it. newRouteResult now wires the
reservation to a sync.Once fired by whichever comes first, the first inference
or route teardown, and replaces three copies of the old wiring.
The backstop: a sweeper for counters leaked by paths that cannot run a defer
at all, such as a frontend killed mid-request.
Identifying a leak by elapsed time alone is unsafe. IncrementInFlight stamps
last_used at request START and nothing moves it while the request runs, so a
long generation is indistinguishable from a leak by age, and resetting there
would expose a serving model to eviction. The probe supplies the missing bit:
a backend that answers a health check promptly is not inside a request,
because that is precisely what a busy one cannot do. Requiring the row to also
be idle for 30 minutes covers backends that serve in parallel and can answer
while working, since those keep last_used fresh through each new increment.
Two existing tests asserted the old behaviour ("No decrement on Release").
That assertion was the leak, so both now pin the release instead.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
f01038f479 |
fix(modelartifacts): stage each writer's artifact in its own partial tree (#10995)
Every writer used to stage into the same `.artifacts/.partial/<cacheKey>`. That was safe only because the artifact lock held: two writers that both believed they had it opened the same blob with O_APPEND and interleaved their bytes into one file, while the resume probe read the other writer's in-flight size. SHA verification caught the damage only after both had burned the entire download. #10986 restored the lock's precondition on CIFS but left the dependency in place. Suffix the staging tree with a writer identity drawn once per process run, so concurrent writers cannot corrupt each other whatever the lock does. The lock stops being a correctness dependency and becomes a pure efficiency optimisation: a lock failure now costs a duplicated download, not a corrupted one. Commit stays an atomic rename. The loser of a commit race reconciles onto the winner's tree instead of surfacing a bare ENOTEMPTY for work that actually succeeded, since the artifact is content-addressed and both trees hold the same verified bytes. Writer-unique staging means a crashed writer's tree is no longer overwritten by its successor, so two things are added to keep it from becoming a disk leak and a resume regression: - A sweep reclaims trees whose contents have been untouched for 24h, matching the window the startup reaper already uses for stray *.partial files. It reads the newest mtime anywhere inside the tree, because writing a blob never touches an ancestor, and refuses any name this package did not write. A live download writes continuously, and the downloader's stall watchdog aborts a silent one long before it could look abandoned. - Adoption lets a restarted process claim a dead predecessor's tree for the same artifact and resume from its bytes, which a tens-of-gigabytes repo depends on. The claim is an atomic rename, so racing adopters cannot both win. It runs only under the artifact lock - which is released exactly when the owning process dies - and only on a tree idle for 5 minutes as a second line of defence for when the lock does not exclude. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
mudler's LocalAI [bot]andEttore Di Giacinto
|
65bdbc4ee3 |
fix(http): make /readyz reflect startup readiness, plus gitignore and coverage-ratchet fixes (#10989)
* fix(http): make /readyz reflect startup readiness instead of always 200 /readyz was registered as a static handler returning 200 unconditionally, so it carried no information: it was green whenever it could be reached at all. Readiness could not distinguish "serving" from "still starting", and any future change that started the HTTP listener earlier would silently turn the probe into a lie. Track startup completion on the Application (atomic flag, flipped at the very end of New() on the success path only) and have the readiness handler consult it per request, returning 503 with a small JSON body while startup is in progress. A nil readiness source fails open so embedders keep the historical behaviour. /healthz is deliberately left readiness-independent. Liveness and readiness answer different questions, and failing liveness during a long preload makes an orchestrator restart the pod mid-download so the preload never finishes. This matters because since #10949 the startup preload materializes HuggingFace artifacts for managed backends: tens of GB for a large model (31 GB observed on a live cluster). Both probes stay in quietPaths and stay exempt from auth. Note the listener is still started only after New() returns, so today the not-ready state is not observable over HTTP. Moving the listener earlier is a separate, deliberate decision and is not made here. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(gitignore): anchor the mock-backend pattern so its source dir is traversable The bare `mock-backend` pattern matched the *directory* tests/e2e/mock-backend/, not just the binary built into it. Git will not descend into an ignored directory even for tracked files, so `git add tests/e2e/mock-backend/main.go` required -f. This was hit while working on #10970. Anchor it to the artifact's full path. The built binary stays ignored (it is also covered by tests/e2e/mock-backend/.gitignore) while the source directory becomes traversable again. Verified with `git check-ignore -v`: a new source file under tests/e2e/mock-backend/ is no longer ignored, and the binary produced by `make build-mock-backend` still is. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * chore(coverage): raise the coverage ratchet from 48.5% to 54.2% The committed baseline had drifted well below reality: it still read 48.5% while a full instrumented run measures 54.2%. A stale-low baseline makes the gate meaningless — coverage could regress by more than 5 percentage points and still pass. Raising a ratchet is a deliberate act, not something to fold into an unrelated fix, so it gets its own commit. The headroom was earned by tests landed in #10946, #10947, #10948, #10949, #10956, #10967, #10968, #10970 and #10975. Measured with `make test-coverage` on this branch (the same instrumented run `make test-coverage-baseline` uses: ginkgo over ./pkg and ./core plus the in-process tests/e2e suite, --covermode=atomic, --coverpkg over core/... and pkg/..., generated protobuf excluded). The run completed with exit 0 and zero spec failures; the total was then written with the exact command the test-coverage-baseline target uses: go tool cover -func=coverage/coverage.out \ | awk '/^total:/{gsub(/%/,"",$NF); print $NF}' > coverage-baseline.txt Verified afterwards with scripts/coverage-check.sh, which reports OK. Note the measured figure includes the readiness specs added earlier on this branch, so it is a demonstrated floor rather than an estimate. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
c1a891662c |
refactor(settings): single declarative registry for runtime settings (fixes the #10845 bug class) (#10864)
* feat(settings): add declarative runtime-settings field registry One fieldSpec row per RuntimeSettings field, with a reflection completeness spec so a field added without a registry row is a red test instead of a silently-dropped setting (the #10845 bug class). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 * refactor(settings): drive ToRuntimeSettings/ApplyRuntimeSettings from the field registry Behavior-preserving: ~350 hand-written per-field lines become two loops over runtimeSettingsFields, gated by a To->Apply->To round-trip spec. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 * feat(settings): baseline-driven startup merge for persisted runtime settings ApplyRuntimeSettingsAtStartup compares the live config against DefaultRuntimeBaseline (option-less-run defaults incl. kong-injected flag defaults) instead of per-field == 0 guards. Fixes persisted lru_eviction_max_retries, tracing_max_items, agent_job_retention_days, memory_reclaimer_threshold, galleries and autoload flags being silently ignored at boot. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 * fix(settings): registry-driven startup merge, applied before consumers loadRuntimeSettingsFromFile becomes a thin wrapper over ApplyRuntimeSettingsAtStartup and runs at the top of New(), before model configs capture app-level defaults. WithThreads stops eagerly resolving 0 so a persisted thread count survives restart while LOCALAI_THREADS still wins (#10845); the physical-core fallback moves after the merge. Also: run.go now injects the memory-reclaimer threshold unconditionally so the option-less boot matches DefaultRuntimeBaseline and a UI-saved threshold survives restart. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 * refactor(settings): file watcher delegates to the registry merge; shared API-key merge Manual edits to runtime_settings.json now behave like a boot-time load (env still wins) instead of the inverted diverged-from-startup guard that ignored most manual edits. MergeAPIKeys dedups env keys in one place for the endpoint and the watcher. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 * docs(settings): document unified runtime-settings precedence Document the single env/CLI > runtime_settings.json > defaults rule, applied identically at boot, on POST /api/settings, and on manual file edits, plus the two known limitations (default-valued env vars are indistinguishable from unset; API-changed fields hot-apply on the next restart only). Also add a completion debug log when the watcher applies runtime_settings.json. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 * test(settings): reset the global VRAM cap leaked by the round-trip spec The round-trip spec applies vram_budget=12GiB, whose post-loop hook installs a process-global default cap; without a reset every spec ordered after it runs under that phantom budget. Also drop a stale enumeration in the ApplyRuntimeSettings doc comment. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-fable-5 --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
bbe018c1a0 |
feat(bonsai): PrismML llama.cpp fork backend + Bonsai/Ternary-Bonsai gallery models (#10834)
feat(bonsai): add PrismML llama.cpp fork backend + Bonsai gallery models Adds a new `bonsai` backend that runs the PrismML fork of llama.cpp (github.com/PrismML-Eng/llama.cpp, `prism` branch), which ships the Q1_0 (1-bit) and Q2_0 (ternary / 1.58-bit) weight-quantization kernels used by the Bonsai and Ternary-Bonsai models. Stock llama.cpp cannot decode these quants. Modeled on the turboquant backend: reuses backend/cpp/llama-cpp/grpc-server.cpp against the fork's libllama via a thin wrapper Makefile, so the sub-2-bit models are served with the same OpenAI-compatible API. No grpc-server allow-list patch is needed (bonsai adds weight quants, transparent to the server, not KV-cache types), and the reused server compiles cleanly against the fork with no skew patches (validated locally via a CPU docker build; patches/ is present but empty for any future re-pin skew). Backend wiring: backend/cpp/bonsai/, .docker/bonsai-compile.sh, backend/Dockerfile.bonsai, top-level Makefile targets, backend-matrix.yml build rows (CPU, CUDA 12/13, L4T, SYCL f32/f16, Vulkan, ROCm/hipblas), backend/index.yaml meta-backend + per-platform images, and a nightly bump_deps entry tracking the `prism` branch. Gallery: 8 entries across 4 families - bonsai-8b-1bit, ternary-bonsai-8b (+g64, +pq2), bonsai-27b-1bit (vision), ternary-bonsai-27b (+pq2, +g64, vision). The 27B models wire the mmproj vision tower; the DSpark speculative drafter GGUFs are not wired (custom semi-autoregressive drafter, not a standard llama.cpp draft model). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Tai AnandAnai-Guo
|
688f904a10 |
fix(runtime-settings): apply persisted threads/context_size/f16 at startup (#10853)
ApplyRuntimeSettings persists the performance settings (threads, context_size, f16) on the live /api/settings path, but the startup loader loadRuntimeSettingsFromFile never read them back, so a value saved via the Middleware UI was silently ignored on the next restart: the model booted with the CLI/physical-core default and GET /api/settings echoed that default instead of the saved value (#10845). Threads needs special handling: unlike context_size/f16, WithThreads eagerly resolves an unset (0) value to xsysinfo.CPUPhysicalCores() at option-apply time, so options.Threads is never 0 in the loader and the usual "== default" heuristic cannot tell an env/CLI value from the physical-core fallback. Detect LOCALAI_THREADS/THREADS explicitly so the env still wins over the persisted file value. Signed-off-by: Anai-Guo <Anai-Guo@users.noreply.github.com> Co-authored-by: Anai-Guo <Anai-Guo@users.noreply.github.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
bcc41219f7 |
feat: materialize Hugging Face model artifacts (#10825)
* feat(config): add model artifact source contract Assisted-by: Codex:GPT-5 [Codex] * feat(downloader): add authenticated raw-byte progress Assisted-by: Codex:GPT-5 [Codex] * feat(huggingface): resolve immutable snapshot manifests Assisted-by: Codex:GPT-5 [Codex] * feat(models): add artifact storage primitives Assisted-by: Codex:GPT-5 [Codex] * feat(models): materialize pinned Hugging Face snapshots Assisted-by: Codex:GPT-5 [Codex] * feat(models): bind managed snapshots at runtime Assisted-by: Codex:GPT-5 [Codex] * feat(gallery): materialize model artifacts during install Assisted-by: Codex:GPT-5 [Codex] * feat(gallery): declare managed Hugging Face artifacts Assisted-by: Codex:GPT-5 [Codex] * feat(models): preload managed model artifacts Assisted-by: Codex:GPT-5 [Codex] * fix(gallery): retain shared artifact caches on delete Assisted-by: Codex:GPT-5 [Codex] * feat(models): report artifact acquisition progress Assisted-by: Codex:GPT-5 [Codex] * refactor(backends): load managed models from ModelFile Assisted-by: Codex:GPT-5 [Codex] * refactor(backends): load staged speech model snapshots Assisted-by: Codex:GPT-5 [Codex] * refactor(backends): use staged snapshots in engine backends Assisted-by: Codex:GPT-5 [Codex] * test(distributed): cover staged artifact snapshots Assisted-by: Codex:GPT-5 [Codex] * docs: explain managed model artifacts Assisted-by: Codex:GPT-5 [Codex] * docs: add product design context Assisted-by: Codex:GPT-5 [Codex] * feat(ui): show model artifact download progress Assisted-by: Codex:GPT-5 [Codex] * Eagerly materialize Hugging Face artifacts Materialize HF-backed model references as managed GGUF artifacts during load, with lazy download retained only as fallback. Assisted-by: Codex:GPT-5 [shell] * Refactor HF downloads through a shared executor Assisted-by: Codex:GPT-5 [shell] * drop Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
40c29db8c4 |
fix(logs): capture backend logs by default in single mode (#10742)
Backend log capture into the per-model BackendLogStore (which feeds the UI "Backend Logs" page and /api/backend-logs) was opt-in and off by default in single mode, while worker/distributed mode force-enables it via SetBackendLoggingEnabled(true). There was no CLI flag either, so the only way to populate the store was the Settings UI toggle - and the page was silently empty out of the box. Distributed "just worked"; single mode looked broken. Default EnableBackendLogging to true in NewApplicationConfig so single mode matches worker mode. The store is a small in-memory ring buffer, so the cost is negligible. Now that the default is on, loadRuntimeSettingsFromFile's usual "only flip false->true" merge would ignore a persisted false and revert the UI toggle-off on every restart. There is no env var/CLI flag for this setting, so an explicit persisted value is now authoritative in both directions, letting the toggle-off survive a restart. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
eb32cd9073 |
feat(realtime): eager blocking pipeline warm-up + /backend/load API (#10662)
Realtime sessions previously lazy-loaded each pipeline sub-model (VAD,
transcription, LLM, TTS) on first use, so every cold session paid a
per-request model-load stall and load errors only surfaced mid-stream.
Warm the whole pipeline eagerly and blockingly at session start
(including the voice-gate speaker-recognition model, which an enforced
gate blocks each utterance on; compaction's summary_model stays lazy
since it only runs off the response path):
- Add backend.PreloadModel / PreloadModelByName as the single load path
for every modality (no transcription special-case; backend-omitted
configs are deprecated).
- The realtime session blocks on Model.Warmup and returns a
model_load_error to the client if any stage fails to load;
updateSession warms in the background. Opt out per pipeline with
pipeline.disable_warmup, exposed as a UI toggle via the
config-metadata registry.
Add a LocalAI-native POST /backend/load (and /v1/backend/load) that
pre-loads a model -- expanding realtime pipelines into their sub-models
-- as the inverse of /backend/shutdown. There is one preload engine
(backend.PreloadStages): the realtime Warmup methods, /backend/load and
the --load-to-memory startup flag all use it, so --load-to-memory now
also expands pipeline models and records load-failure traces. Pipeline
sub-model alias resolution is likewise shared
(ModelConfigLoader.LoadResolvedModelConfig). Surface the endpoint
everywhere an admin manages models:
- MCP admin tool load_model (httpapi + inproc clients, safety/catalog
prompts, catalog/dispatch tests).
- "Load into memory" action in the React models UI.
- Swagger regenerated; docs moved to the general backend-monitor page
since it is not realtime-specific.
Fix a Traces UI crash ("json: unsupported value: -Inf"): audio-snippet
RMS/peak now floor at a finite dBFS, and backend-trace data is sanitized
to drop non-finite floats before marshaling. The sanitizer is
copy-on-write -- it runs on every RecordBackendTrace, so containers are
only re-allocated on the paths that actually changed.
Migrate core/http/openresponses_test.go onto the prebuilt mock-backend
the rest of the http suite already uses -- it was the last spec still
pointing at a real HuggingFace model, so it 404'd wherever no vision
backend was built -- and fix its item_reference specs to send the
spec's "id" field instead of "item_id", which the handler never
accepted.
Assisted-by: Claude:claude-opus-4-8 Claude Code
Signed-off-by: Richard Palethorpe <io@richiejp.com>
|
||
|
LocalAI [bot]andEttore Di Giacinto
|
6eea3ef2ac |
fix(backends): make backend install ops idempotent unless forced (#10643)
* fix(backends): make backend install ops idempotent unless forced POST /backends/apply hardcoded force=true through LocalBackendManager.InstallBackend, so applying an already-installed backend re-downloaded and re-extracted the whole artifact every time. API clients that ensure a backend exists at startup paid a full OCI image pull on every boot. Backend install ops now default to non-forced — an installed, runnable backend short-circuits (the orphaned-meta reinstall path in InstallBackendFromGallery is preserved) — and reinstall stays available: - ManagementOp gains a Force field; the local manager passes it through instead of hardcoding true. - /backends/apply accepts an optional "force" boolean in the body. - The React UI install route keeps forcing, since its button doubles as the explicit "Reinstall backend" action. Distributed installs already behaved this way (workers skip when the binary exists unless force is set); this aligns single-node behavior. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(backends): don't force-reinstall LOCALAI_EXTERNAL_BACKENDS on boot The startup loop for LOCALAI_EXTERNAL_BACKENDS runs InstallExternalBackend for each listed backend on every boot, and its gallery-name path hardcoded force=true — so every start re-downloaded and re-extracted each listed backend's OCI image even when it was installed and runnable. Supervising apps that list several backends paid several full OCI pulls per launch. Give InstallExternalBackend an explicit force parameter (it only affects the gallery-name fallback; URI installs always write) and pass: - false from the boot loop and `local-ai backends install` (idempotent ensure — `backends upgrade` is the refresh path), - op.Force from the local manager's external-URI op, - the request's force on the worker install path and true on its upgrade path (behavior unchanged). Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
d7d7721eae |
feat(distributed): SyncedMap component + migrate finetune/quant/agent-tasks to cross-replica state (#10542)
* feat(distributed): add SyncedMap cross-replica in-memory state component Introduce core/services/syncstate.SyncedMap[K,V]: a thread-safe in-memory map that keeps itself consistent across frontend replicas via NATS, with an optional pluggable durable Store and hydrate-from-source convergence. Several features keep process-local state surfaced to the API (finetune/quant jobs, agent tasks, model configs) and each hand-wired the same in-memory + NATS broadcast + read-through-store legs - or forgot to, reintroducing cross-replica staleness. SyncedMap makes that consistency a configuration choice: - local writes mutate the map, write through the Store, then broadcast a delta; - the apply path is memory-only and never re-publishes or re-writes the Store (structural echo-loop guard, mirroring galleryop.mergeStatus); - on Start and on NATS reconnect the map re-hydrates from the source (Store, else Loader); an optional periodic Reconcile repairs silent drift; - standalone mode (nil NATS client) is a strict in-memory no-op. Reconnect re-hydrate is wired via a new *messaging.Client.OnReconnect callback, consumed through an optional type-assertion so MessagingClient stays minimal. Adds messaging.SubjectSyncStateDelta and a reusable testutil.FakeBus (synchronous in-process MessagingClient with wildcard matching) for adopter tests. Component only; service migrations follow in subsequent commits. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * refactor(finetune): back jobs with SyncedMap for cross-replica consistency FineTuneService kept jobs in a process-local map and, although it wrote them to Postgres, ListJobs/GetJob never read the store back and the wired natsClient was never used - so in distributed mode a job created on one replica was invisible to the others. Replace the map and the dead client with a syncstate.SyncedMap keyed by job ID, value *schema.FineTuneJob (the exact REST shape, so responses are unchanged). - Add a Store adapter (core/services/finetune/syncstore.go) over FineTuneStore, plus FineTuneStore.ListAll (global hydrate; per-user List kept) and an idempotent Upsert (create-or-update; Create alone fails on dup key). - Writes go through SyncedMap.Set/Delete (write-through + broadcast); reads use List/Get. The on-disk state.json path becomes the standalone Loader, keeping single-node restart recovery (stale->stopped / exporting->failed fixups). - Fold SetNATSClient/SetFineTuneStore into NewFineTuneService; app.go passes the distributed NATS client + store when distributed, nil otherwise. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * refactor(agentpool): back agent tasks with SyncedMap for cross-replica consistency AgentJobService.ListTasks read the process-local tasks map only, while ListJobs already read through the DB persister + dispatcher NATS - so in distributed mode a task created on one replica was invisible to the others. Back tasks with a syncstate.SyncedMap keyed by task ID (value schema.Task, the exact REST shape); jobs are left untouched. - Store adapter (task_syncstore.go) over the existing JobPersister (LoadTasks/SaveTask/DeleteTask); reads svc.persister/userID live so a persister swap needs no rebuild. No new persister methods required. - Task reads -> SyncedMap.List/Get; create/update -> Set (write-through + broadcast); delete -> Delete. The file persister now owns its own task set so the write-through path does not re-enter the SyncedMap lock (deadlock guard). - The distributed NATS client is not available at construction (start() precedes initDistributed), so it is injected via SetTaskSyncNATS, which rebuilds the still-empty map before Start/hydrate. Wired at the main, restart, and per-user (UserServicesManager) distributed sites. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * refactor(quantization): back jobs with SyncedMap + durable QuantStore QuantizationService kept jobs in a process-local map persisted only to a local state.json, so in distributed mode jobs were neither visible across replicas nor durable cluster-wide. Back jobs with a syncstate.SyncedMap keyed by job ID (value *schema.QuantizationJob, the exact REST shape). - New distributed.QuantStore (GORM, table quantization_jobs) mirroring FineTuneStore: Create/Get/ListAll/Upsert(idempotent)/Delete, registered for AutoMigrate via distributed.InitStores (Stores.Quant). - New adapter (quantization/syncstore.go) over QuantStore implementing syncstate.Store, with record<->schema conversion. - Reads go through List/Get, writes through Set/Delete (write-through + broadcast); state.json is kept as the standalone Loader for single-node restart recovery (stale-job fixups preserved). - app.go passes the distributed NATS client + QuantStore when distributed, nil otherwise; Start/Close lifecycle mirrors finetune. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * fix(syncstate): annotate gosec G118 false positive on lifeCtx gosec flagged the WithCancel in Start as "cancellation function not called" because the returned cancel is stored on the struct rather than called/deferred in scope. It is invoked in Close (covered by tests), and lifeCtx must outlive Start to drive the reconnect/reconcile goroutines. Suppress the verified false positive with a justified #nosec G118. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * test(distributed): e2e two-replica SyncedMap sync over real NATS + Postgres Adds the real-infrastructure counterpart to the fake-bus unit tests, in the existing distributed e2e suite (testcontainers NATS + PostgreSQL). Two SyncedMap instances stand in for two frontend replicas - each with its OWN NATS connection to a shared server and a SHARED Postgres store (the distributed-mode invariant) - and assert, over the wire: - a create on replica A is observed by replica B; - an update and a delete propagate A -> B (delete prunes, which a reload cannot); - a late-joining replica recovers a job it never received a delta for, via store hydrate on Start (the at-most-once gap a fake bus cannot exercise); - a local Set is written through to the shared Postgres store. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
64150ca7ab |
fix(distributed): broadcast admin model-config changes across replicas (#10540)
In distributed mode the admin model endpoints (/models/edit, /models/import, /models/toggle-state and the PATCH config-json endpoint) wrote the YAML to the shared models dir but reloaded only the local replica's in-memory ModelConfigLoader. With multiple frontend replicas behind one service, a save landed on whichever replica handled the request; peers kept serving their stale in-memory view, so a load-balanced request was a coin-flip between old and new config (a created alias visible on one replica and missing on the other, an edited alias target diverging, etc.). The NATS cache-invalidation channel (SubjectCacheInvalidateModels + OnModelsChanged) already existed for the gallery install/delete path; these admin endpoints simply never published on it. Wire them up via a new GalleryService.BroadcastModelsChanged helper (no-op in standalone mode). Also fix delete propagation: LoadModelConfigsFromPath is additive and never drops an entry whose file is gone, so the subscriber hook (which only reloaded from disk) could not propagate a removal. ApplyRemoteChange now honors the event op - pruning the element on "delete" and reloading otherwise - and shuts down any running instance of the affected model so the new config takes effect. This closes the same latent gap on the gallery delete path. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard PalethorpeandEttore Di Giacinto
|
e1994579f8 |
fix(pii): load default detectors at startup + add LOCALAI_PII_DEFAULT_DETECTORS (#10474)
pii_default_detectors was applied to the live config only by a live POST /api/settings (ApplyRuntimeSettings) — neither the startup loader nor the config file watcher read it back. So after a restart the persisted default detectors were dropped, and the cloud-proxy MITM listener (which resolves each intercept host's detectors once at start via ResolvePIIPolicy) came up with an empty set and forwarded intercepted traffic unredacted, even though the MITM model had pii.enabled:true and the defaults were on disk. Request-side default redaction broke the same way. - startup.go: loadRuntimeSettingsFromFile now applies pii_default_detectors, before startMITMIfConfigured, with env > file precedence. - config_file_watcher.go: apply pii_default_detectors on live file edits, matching the existing env-guard pattern used for the other fields. - settings endpoint: rebuild the MITM listener when pii_default_detectors changes (its per-host detector map is frozen at listener start), not only on a mitm_listen change — so toggling a default detector takes effect on cloud-proxy traffic immediately. - new LOCALAI_PII_DEFAULT_DETECTORS env var / CLI flag (WithPIIDefaultDetectors) so the default detector set can be pinned at boot for immutable deployments. Assisted-by: Claude:claude-opus-4-8 Claude-Code Signed-off-by: Richard Palethorpe <io@richiejp.com> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com> |
||
|
Leoy
|
b50b1fe418 |
feat(watchdog): add size-aware LRU eviction mode (#9527)
* feat(watchdog): add size-aware LRU eviction mode When the model count hits the LRU limit or the memory reclaimer fires, evict the largest model by on-disk file size first rather than the least-recently-used one. For GGUF models the file size is a reliable proxy for GPU/RAM footprint, so evicting the largest candidate maximises freed memory per eviction round while keeping small utility models (embeddings, classifiers, rerankers) resident. Changes: - `pkg/model/watchdog.go`: add `sizeAwareEviction` flag and `modelSizes map[string]int64` to `WatchDog`; sort candidates by `sizeBytes` desc (LRU time as tiebreaker) when the flag is set; add `RegisterModelSize`, `SetSizeAwareEviction`, `GetSizeAwareEviction` - `pkg/model/watchdog_options.go`: add `WithSizeAwareEviction` option - `pkg/model/initializers.go`: stat model file after load and call `RegisterModelSize` so size data is available before the first eviction - `core/config/application_config.go`, `runtime_settings.go`: add `SizeAwareEviction` field and `WithSizeAwareEviction` app option; expose via `ToRuntimeSettings` / `ApplyRuntimeSettings` for the `POST /api/settings` live-reload path - `core/cli/run.go`: add `--size-aware-eviction` flag / `LOCALAI_SIZE_AWARE_EVICTION` env var - `core/application/startup.go`, `watchdog.go`: wire the new option through to `NewWatchDog` - `pkg/model/watchdog_test.go`: 5 new specs — option enable, dynamic toggle, largest-first ordering, equal-size LRU tiebreaker, no-size fallback to LRU, and size-map cleanup on eviction Closes #9375 Signed-off-by: supermario_leo <leo.stack@outlook.com> * refactor(watchdog): use vram estimation scaffolding for model size Replace the brittle os.Stat(modelFile) approach with a proper call to pkg/vram, which handles multi-file models (DownloadFiles, MMProj) and all weight file types, not just single GGUF files. - Add estimateModelSizeBytes() in core/backend/options.go that collects all weight file URIs from the model config, resolves them to file:// URIs, and calls vram.Estimate() with the shared DefaultCachedSizeResolver (15-min TTL cache avoids redundant stat calls on repeated loads) - Thread the result through via a new WithModelSizeBytes() loader option - In initializers.go, consume the pre-computed size instead of calling os.Stat; if no size was supplied (e.g. for external/router-dispatched models) the registration is simply skipped Signed-off-by: supermario_leo <leo.stack@outlook.com> * refactor(watchdog): use EstimateModel with HF fallback for size estimation Switch estimateModelSizeBytes from calling vram.Estimate directly to the unified vram.EstimateModel entry point, which adds automatic fallbacks: file-based GGUF metadata → HF API → size string. Also extract the HuggingFace repo ID from model URIs (huggingface://, hf://, https://huggingface.co/ and org/model short-form) and pass it as ModelEstimateInput.HFRepo, so models not yet downloaded locally can still get a size estimate via the HF API. Addresses @mudler's review feedback: "better to rely on EstimateModel and pass by the HF URL of the model extracted from the URI". Signed-off-by: supermario_leo <leo.stack@outlook.com> * feat(webui): add Size-Aware Eviction toggle to settings page The size-aware eviction setting was wired through the CLI flag and the RuntimeSettings live-reload path (POST /api/settings) but had no handle on the React settings page, so it could not be toggled from the UI. Add a Size-Aware Eviction toggle to the Watchdog section, next to the existing Force Eviction When Busy / LRU eviction handles. The settings page loads and saves the whole RuntimeSettings object, so the new size_aware_eviction key is picked up with no extra plumbing. Addresses @mudler's review feedback: the application config setting should land on the same UI settings page as the other handles. Signed-off-by: supermario_leo <leo.stack@outlook.com> --------- Signed-off-by: supermario_leo <leo.stack@outlook.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
2e734bf560 |
fix(downloader): stall timeout, resume-safe cancel, and stale-partial reaping (#10406)
* fix(downloader): stall timeout, resume-safe cancel, and stale-partial reaping Large model installs would hang forever or never finish. Three defects in the HTTP download path, all hit by big GGUF pulls over a slow or flaky link: 1. No stall timeout. The shared download client sets no body deadline (correct for streaming) but also no read-idle timeout, and the transport's IdleConnTimeout does not cover an in-flight body read. A silently-dropped TCP connection (no FIN/RST) blocked the body Read forever, freezing an install at N bytes until an external reaper killed it. Add an idle-timeout reader that closes the body after a window of zero progress (DownloadStallTimeout, default 60s), turning an indefinite hang into a fast, retryable error. A read that returns data resets the clock, so a slow-but-steady transfer is unaffected. 2. Cancellation deleted the partial. On context.Canceled the code removed the .partial file, so any frontend restart (deploy, OOM) mid-download wiped all progress and the retry restarted from zero. At slow egress, files larger than the restart interval never completed. Keep the .partial on cancel so the next attempt resumes via Range. 3. Partials leaked. Cleanup only ran on the context-cancel path, never on a stall or a SIGKILL/OOM, so abandoned .partial files accumulated and could fill the models volume. Add CleanupStalePartialFiles and reap partials older than 24h on startup. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * fix(downloader): discard the .partial on a deliberate user cancel Review follow-up. The previous commit kept the .partial on every cancellation so restarts could resume, but that also left a dangling partial when a user *intentionally* cancelled an install — the file lingered until the 24h reaper. Distinguish the two: cancel the gallery operation's context with a cause (downloader.ErrUserCancelled) so the download layer can tell a deliberate abort (discard the partial) from an incidental one such as a shutdown/restart (keep it for resume). Detect cancellation via the context rather than the returned error, because an HTTP request cancelled with a cause surfaces the cause error, not context.Canceled. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * fix(downloader): resolve gosec G122 in CleanupStalePartialFiles CI's code-scanning (gosec) flagged G122 (symlink TOCTOU) for the os.Remove call inside the filepath.WalkDir callback. Collect the stale paths during the walk and delete them afterwards instead of mutating the tree from inside the callback. Behavior is unchanged; the existing specs still pass. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
3fa7b2955c |
feat(pii): NER tier engine — privacy-filter.cpp backend + NER-centric PII filter (#10360)
Squashed feat/pii-ner-tier-engine rebased onto master (was 45 commits; see backup/pii-ner-tier-engine-prerebase). Net change: - privacy-filter.cpp: standalone GGML engine for the openai-privacy-filter PII/NER token classifier, wired as a LocalAI gRPC backend (CPU/CUDA/Vulkan). TokenClassify moves off the patched llama.cpp path onto this backend. - PII filter reworked to be NER-centric (encoder/NER detection tier scanning whole conversations as one document), with a recreated bounded restricted- regex secret-matching pattern detector tier alongside it (per-model pii_detection.builtins / .patterns + core/services/routing/piipattern). - Detection labelled by source (ner vs pattern); backend trace / confidence / debug observability; analyze/redact exposed as a synchronous API. - Instance-wide default detector policy + per-usecase default-on; request filtering extended to completions, embeddings, edits & Ollama. - React UI: NER-centric PII editor, detector-models table, pattern/builtins editor, middleware default-policy UI. - Gallery: privacy-filter-multilingual token-classify model + NER install filter; token_classify known_usecase; batch sized to context for NER models. privacy-filter backend registered in the backend gallery (cpu/vulkan/cuda-13 meta + image entries with a capabilities map) matching its CI matrix jobs, and an /import-model auto-detect importer (PrivacyFilterImporter, narrow privacy-filter GGUF detection) replacing the prior pref-only registration. Reconciled against master's independent evolution: - Dropped master's PIIPatternOverrides feature (global-pattern runtime overrides + /api/pii/patterns API + runtime_settings.json persistence). The per-model NER + pattern-detector design supersedes it; it was built on the global redactor pattern set this branch replaced. - Reverted the llama.cpp Score carry-patch (0006-server-task-type-score): removed the patch and restored master's grpc-server.cpp Score RPC (direct llama_decode, slot-loop bypass) and LLAMA_VERSION pin, plus master's model_config validation forbidding score + chat/completion/embeddings on llama-cpp. token_classify is unaffected (it runs on the privacy-filter backend, not llama-cpp). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
Richard Palethorpe
|
085fc53bbc |
fix(router): production-ready request router + auto-size batch for embedding/rerank (#10104)
* fix(router): score classifier production-readiness Conversation trimming runs through the classifier model's chat template and trims by exact token count, sized to the model's n_batch which is now scaled to context so long probes can't crash the backend. Missing chat_message templates are a hard error at router build time. Router- facing factories (Embedder/Scorer/Reranker/TokenCounter) re-resolve ModelConfig per call so a model installed post-startup doesn't bind a stub Backend="" config and silently fall into the loader's auto- iterate path. New 'vector_store' backend trace recorded inside localVectorStore on every Search/Insert — including the backend-load-failure path that previously vanished into an xlog.Warn — with outcome tagging (hit/miss/empty_store/backend_load_error/find_error/insert_error/ok). Companion cleanup drops misleading similarity:0 and input_tokens_count:0 from non-hit and text-mode traces. Gallery local-store-development aliases to 'local-store' so the master image satisfies pkg/model.LocalStoreBackend lookups from the embedding cache. Misc: llama-cpp TokenizeString reads the correct 'prompt' JSON key (the original bug); ModelTokenize nil-guard; non-fatal mitm proxy startup; PII 'route_local' renamed to 'allow' with docs/UI in sync; model-editor footer no longer eats the edit area on small screens; several config-editor template/dropdown/section fixes. Tests: e2e router specs (casual/code-hint + long-conversation trim), vector_store trace specs, lazy-factory specs, gallery dev-alias resolution, Playwright trace badge + scroll regression. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(backend): auto-size batch to context for embedding and rerank models Embedding and rerank models pool over the whole input in a single physical batch (n_ubatch). With batch left at the 512 default, the backend rejects longer inputs with "input is too large to process", silently capping a large-context embedder (e.g. 8k/32k) at 512 tokens. Size n_batch to the context for these single-pass usecases, mirroring the existing FLAG_SCORE behaviour; an explicit batch: still wins. Extracts EffectiveContextSize/EffectiveBatchSize from grpcModelOpts so the effective decode window has one home for other callers to reuse. Adds an e2e-aio regression test that embeds a >512-token input. The AIO embedding model is switched to nomic-embed-text-v1.5 (2048 context) because the previous granite model was capped at 512 tokens and could not exercise the larger batch. Assisted-by: claude-code:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(gallery): raise arch-router scoring output cap via parallel:64 Scoring decodes the whole prompt+candidate in a single llama_decode and reads one logit row per candidate token. The vendored llama.cpp server caps causal output rows at n_parallel, so the default of 1 aborts with GGML_ASSERT(n_outputs_max <= cparams.n_outputs_max) on multi-token route labels. Set options: [parallel:64] on both arch-router quant entries to lift the cap; kv_unified (the grpc-server default) keeps the full context per sequence, so this does not split the KV cache. Assisted-by: claude-code:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
92dea961c2 |
fix: distributed backend reinstall/upgrade UI stuck on 'reinstalling' (#10214)
* fix(galleryop): self-evict terminal ops from OpCache.GetStatus The processingBackends map (the UI 'reinstalling' spinner source) only cleared an op when a client polled /api/backends/job/:uid. The Manage-page Reinstall and Upgrade buttons never poll, so completed installs leaked into processingBackends forever and the backend card spun 'reinstalling' even though the install had finished. Evict terminal ops on the list read instead; DeleteUUID already broadcasts the eviction so peer replicas converge. Reproduced on a live 5-node distributed cluster: 5 backends sat in processingBackends with underlying jobs reporting completed:true,progress:100. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nodes): clear pending backend ops behind offline/draining nodes ListDuePendingBackendOps filters status=healthy, so a backend op queued against a node that went offline (stale heartbeat) or draining (admin action) was never retried, aged out, or deleted - it leaked forever and kept the UI operation spinning. Add DeleteStalePendingBackendOps and run it each reconcile pass: draining nodes are cleared immediately (model rows already purged), offline nodes once their heartbeat is older than a grace window (blip protection). Reproduced on a live cluster: orphaned llama-cpp install rows targeting an offline (nvidia-thor) and a draining (mac-mini-m4) node sat at attempts=0 indefinitely. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nodes): stream per-node progress during backend upgrade The install dispatch subscribed to a per-op progress subject and streamed per-node download ticks; the upgrade dispatch did a bare 15-minute blocking NATS round-trip with no subscription, so the UI showed progress:0 the whole time (the 'reinstalling but nothing happens' report on a slow node). Thread the op ID through BackendManager.UpgradeBackend -> the distributed manager -> the adapter, and have the adapter subscribe to the per-op progress subject before the request (extracted into a shared subscribeProgress helper reused by install/upgrade/force-fallback). The worker's upgradeBackend now creates the same DebouncedInstallProgressPublisher installBackend uses. An upgrade is a force-reinstall, so it reuses SubjectNodeBackendInstallProgress rather than minting a new subject - no new NATS permission, no new rolling-update compat surface. Reconciler-driven retries pass empty opID/onProgress and stay on the silent path. Reproduced on a live cluster: upgrade of llama-cpp-development on agx-orin-slow sat at progress:0 for 4+ minutes with no per-node feedback. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(galleryop): persist cancellation + periodically reap orphaned ops Two distributed gaps surfaced when a replica was killed mid-upgrade on a live cluster, leaving the backend stuck 'processing' in the UI forever: 1. CancelOperation flipped the in-memory status to cancelled and broadcast a NATS event but never persisted the terminal status. On the next replica restart the still-active row re-hydrated straight back into processingBackends and the UI spun again. It now calls store.Cancel(id) so the cancel survives a restart. 2. CleanStale (which marks abandoned active ops failed) only ran once on startup, so an op orphaned AFTER startup - its owning replica's foreground handler goroutine gone - was never reaped until the next restart. Add GalleryService.ReapStaleOperations and run it on a 15m ticker (CleanStale now returns the reaped count for observability). Neither is covered by the OpCache self-evict fix: an orphaned op never reaches Processed, so it would never self-evict. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(review): address self-review findings on the distributed install fixes Three findings from an adversarial review of this branch: 1. CRITICAL - OpCache.GetStatus crashed under concurrent load. m.Map() returns the live internal map by reference, so deleting from it on the read path was an unsynchronized write to a map four HTTP handlers poll every ~1s -> a 'concurrent map writes' fatal. Rewritten to iterate a Keys() snapshot, build a fresh result map, and apply evictions via the locked DeleteUUID after the loop. Added a -race concurrency regression guard. 2. HIGH - GetStatus evicted failed ops too, hiding them from /api/operations and breaking the dismiss-failed-op flow (the panel keeps Error != nil ops so the admin can read the error and click Dismiss). Eviction now fires only for terminal ops with Error == nil (success/cancelled); failures are retained. 3. MEDIUM - DeleteStalePendingBackendOps missed StatusUnhealthy nodes. A node marked unhealthy on a NATS ErrNoResponders never transitions to offline (health.go skips re-marking it), so its pending ops leaked exactly like the offline case. Unhealthy is now reaped via the same stale-heartbeat grace path (a fresh-heartbeat node is recovering and keeps its op). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(review-2): don't evict the still-installing soft-path; don't spin on failed ops Second review pass found two issues: 1. MEDIUM (Go) - OpCache.GetStatus evicted the ErrWorkerStillInstalling soft-path op. That op is deliberately Processed=true with no error to show a yellow in-progress state when a worker timed out the NATS round-trip but is still installing in the background; the reconciler confirms the real outcome later. Evicting it (and broadcasting OpEnd + marking the DB completed) hid an install that may still fail. Eviction is now scoped to a clean success (progress 100 + 'completed', matching the job-poll's historical condition) or a cancellation - the soft-path (progress != 100) and failures are kept. 2. MEDIUM (React) - the Backends gallery card rendered ANY operation as an 'Installing...' spinner, so a failed op (now intentionally kept in the list for the OperationsBar error + Dismiss) spun forever. Exclude errored ops from the card spinner, mirroring Models.jsx (isInstalling already excludes op.error). The error + Dismiss still surface in the global OperationsBar. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): refresh Manage backends table when an operation settles The Manage backends table fetched installed backends only on mount/after delete and checked upgrades only on tab activation. After a reinstall/upgrade completed neither re-ran, so the installed-version cell and the 'update available' badge stayed stale until the user switched tabs - the op looked like it 'did nothing'. Watch the operations list (via useOperations) and re-fetch installed backends + available upgrades whenever the count settles, mirroring the operations.length watch Backends.jsx already uses. Consolidates the prior tab-activation upgrades check into the same effect. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
fbcd886a47 |
fix(application): stop backend processes synchronously on shutdown (#10058)
application.New wires a fire-and-forget goroutine that runs StopAllGRPC + distributed.Shutdown when the app context is cancelled. Callers (tests, CLI signal handler) cancel the context and then exit immediately, so the test binary / process can terminate before that goroutine kills the spawned backend children. go-processmanager sets no Pdeathsig, so the orphans are reparented to init and survive — leaving dozens of stray mock-backend processes after an e2e run. Add Application.Shutdown(), which runs the same cleanup synchronously on the caller's stack and is idempotent via sync.Once. The context-cancel goroutine, the CLI signal handler, and the test suites all call it, so cleanup is deterministic and the duplicated teardown logic collapses to one place. The async goroutine remains as a safety net for callers that forget; sync.Once dedupes the double call. Wire e2e_suite_test and the two mock-backend Contexts in app_test to call Shutdown in their AfterSuite/AfterEach. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
8d6548c0b9 |
fix(distributed): sync gallery OpCache + caches across frontend replicas (#9983)
When the LocalAI frontend deployment is scaled past one replica, the UI's
/api/operations poll round-robins between pods. Each pod kept the OpCache
(galleryID->jobID), OpStatus map, and the post-install in-memory caches
(ModelConfigLoader, UpgradeChecker) purely in-process. Reads never
consulted PostgreSQL or NATS even though writes already published to PG.
Symptoms:
- A user installing a model on replica A saw the operation card flicker
in and out as the load balancer alternated.
- The Models page re-fetched the whole gallery on every flicker because
useEffect([operations.length]) re-fires when the count changes.
- A chat completion that landed on replica B after the install completed
on replica A failed to find the new model — B's ModelConfigLoader was
still the old one because nothing told it to reload from disk.
- The UpgradeChecker 6-hour cache stayed stale on peer replicas after a
backend upgrade, so /api/backends/upgrades kept surfacing an upgrade
that had already shipped.
Mirror the jobs Dispatcher pattern for gallery ops:
- OpCache learns SetMessagingClient/SetGalleryStore + a Start(ctx) that
hydrates from PostgreSQL and subscribes to gallery.opcache.{start,end}.
Set/SetBackend now upsert cache_key + is_backend_op on the gallery_
operations row and broadcast OpCacheEvent so peers merge it in. The
hydrate path uses a new GalleryStore.ListActive() (status in {pending,
downloading, processing} and updated within 30 min).
- GalleryService.SubscribeBroadcasts wires a SubjectGalleryProgress-
Wildcard subscriber that calls a new lock-light mergeStatus into the
local statuses map, plus a SubjectGalleryCancelWildcard subscriber that
runs the locally-registered cancel func. Hydrate() restores active rows
from PostgreSQL on startup so a freshly-started replica is not
observably empty mid-install. CancelOperation tolerates the cancel func
living on a different replica and publishes anyway.
- modelHandler and backendHandler publish on the new
SubjectCacheInvalidateModels / SubjectCacheInvalidateBackends after
a successful install/delete/upgrade. SubscribeBroadcasts wires peers
to refresh: OnModelsChanged (re-runs LoadModelConfigsFromPath) and
OnBackendOpCompleted (re-triggers UpgradeChecker). The originating
replica reloads inline so it never enters the broadcast handler.
- OpStatus.Error (an error interface) flat-marshalled to "{}" over JSON,
so a failed install replicated to a peer arrived with a nil error and
the UI's failure banner never appeared. Add MarshalJSON/UnmarshalJSON
via an opStatusWire shim that round-trips Error as a string.
- UpdateStatus and CancelOperation now drop the mutex before publishing
to NATS or persisting to PostgreSQL. The wildcard subscriber's
mergeStatus loops back into the same service on the publishing replica
and would deadlock otherwise; this also prevents future PG round-trips
from stalling concurrent readers on every progress tick.
Tests cover the OpStatus error round-trip, OpCache propagation through a
shared in-memory bus, OpCache PostgreSQL hydration (active-only),
GalleryService progress + cancel broadcast, Nodes preservation across a
peer's bare progress tick, GalleryService hydration from PG, and the
two cache-invalidation broadcasts (models + backends). 44 specs total
in galleryop; routes/operations specs and jobs/agents suites still pass.
Assisted-by: claude-code:claude-opus-4-7
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Richard Palethorpe
|
6a80e23733 |
feat(middleware): Model routing, PII filtering, Cloud model proxies (#9802)
Add a routing middleware stack and a cloud-proxy backend. * cloud-proxy: a Go gRPC backend that forwards OpenAI- and Anthropic-shaped chat requests to upstream providers, with an optional translate mode (OpenAI request -> Anthropic /v1/messages -> OpenAI response) and full tool-calling support. * routing: admission control, content-aware model routing (embedding cache + classifier + rerank + Arch-Router score), PII detection/redaction (regex + NER) with streaming filter and OpenAI/Anthropic adapters, and a per-user/per-key billing recorder backed by GORM or in-memory storage. * middleware: UsageMiddleware records usage via the billing recorder, plus admission, route-model, usage-stamp and trace middlewares. * observability: BackendTrace ring buffer stores full request bodies (capped), MITM proxy emits structured trace events, and router classifier decisions surface at /api/router/decide. * gallery: Arch-Router-1.5B (Q4_K_M and Q8_0). * UI: cloud-proxy model-editor fields, classifier system-prompt and score-normalization config, and a Traces page rendering request bodies. Assisted-by: claude-code:claude-opus-4-7 [Read] [Edit] [Bash] Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
a0f3e26245 |
fix(distributed): make admin backend installs resilient and observable (#9958)
* feat(distributed): add configurable NATS backend install/upgrade timeouts Adds BackendInstallTimeout and BackendUpgradeTimeout to DistributedConfig with 15m defaults, following the existing MCPToolTimeout / WorkerWaitTimeout pattern. These will replace the hardcoded literals in RemoteUnloaderAdapter so admin-driven backend installs across the cluster survive long OCI image pulls that previously timed out at 3m. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * style(distributed): gofmt alignment after timeout fields Re-aligns the Validate() negative-duration map and the Default* const block so the new BackendInstall/UpgradeTimeout entries do not leave the surrounding columns mis-padded. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(cli): surface LOCALAI_NATS_BACKEND_INSTALL_TIMEOUT and _UPGRADE_TIMEOUT Parses the two new env vars on the run CLI and threads them through the existing AppOption builder so DistributedConfig picks them up. Invalid duration strings now fail loudly at startup rather than silently falling back to the default. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): inject NATS install/upgrade timeouts into RemoteUnloaderAdapter Removes the hardcoded 3m / 15m literals from RemoteUnloaderAdapter and threads in DistributedConfig.BackendInstallTimeoutOrDefault() and BackendUpgradeTimeoutOrDefault() at construction. Install now defaults to 15m (was 3m); cold OCI image pulls on Jetson Wi-Fi routinely blew past the old ceiling. Scripted messaging client captures the timeout so tests can assert the configured value actually reaches the NATS request. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): introduce galleryop.ErrWorkerStillInstalling sentinel When the NATS request-reply for backend.install (or .upgrade) times out the worker is almost always still pulling the OCI image. Wrap the timeout in a typed sentinel so the manager above can distinguish "worker hung" from "worker still working" and leave the pending_backend_ops row in place for the reconciler to confirm via backend.list. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): treat NATS install timeout as in-progress, not failure When a worker times out replying to backend.install but the install is still running on the worker, enqueueAndDrainBackendOp now reports a running_on_worker status and pushes NextRetryAt out by the install timeout so the reconciler does not immediately re-fire another install while the worker is still pulling the image. The pending_backend_ops row stays in place for the next reconciler pass to confirm via backend.list. InstallBackend wraps the result in galleryop.ErrWorkerStillInstalling so callers can branch (galleryop renders yellow in-progress instead of red error). UpgradeBackend uses the same wrap. Adds RemoteUnloaderAdapter.InstallTimeout() so the manager can push NextRetryAt by the configured timeout without reaching into a private field, and NodeRegistry.RecordPendingBackendOpInFlight as the soft cousin of RecordPendingBackendOpFailure. Also includes incidental gofmt-driven struct-field alignment in registry.go on lines unrelated to the change (touched files are re-formatted to canonical form per project policy). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): don't increment Attempts on in-flight install timeout An in-flight timeout (worker still pulling the OCI image) is not a failed attempt, it's a delayed one. Incrementing Attempts let genuinely-progressing slow installs (e.g. 30 GB CUDA images on Wi-Fi) trip the reconciler's maxPendingBackendOpAttempts cap and dead-letter the queue row while the worker was still legitimately working. RecordPendingBackendOpInFlight now only updates LastError and NextRetryAt. Also documents "running_on_worker" in the NodeOpStatus.Status enum comment so Task 6 implementers see the full surface. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(galleryop): surface ErrWorkerStillInstalling as non-error OpStatus When the distributed backend manager returns an error that wraps ErrWorkerStillInstalling, backendHandler now completes the op with a "still installing in background" message rather than marking it as a red failure. Admin UI sees a yellow in-progress state; reconciler confirms completion on its next pass. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(distributed): end-to-end install-timeout-then-reconcile Wires Task 1-6 end-to-end so any seam mismatch surfaces in CI rather than during a real cluster install. NATS times out, the queue row stays alive with running_on_worker status, the worker eventually reports the backend installed via backend.list, the manager surfaces it via ListBackends. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(distributed): document LOCALAI_NATS_BACKEND_INSTALL_TIMEOUT / _UPGRADE_TIMEOUT Add the two new operator-tunable env vars to the Frontend Configuration table in the distributed-mode docs. Explains the 15m default, when to raise it (slow links pulling multi-GB OCI images), and the new "still installing in background" admin-UI state when the round-trip times out but the worker is still working. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): clear pending install rows when backend.list confirms DistributedBackendManager.ListBackends now proactively clears pending_backend_ops install rows whose (nodeID, backend) is reported installed by backend.list. Operator UI updates immediately instead of waiting up to installTimeout (default 15m) for the next reconciler tick after NextRetryAt. Only install rows are cleared; upgrade and delete intents are not satisfied by presence in backend.list and continue to drain through their normal reconciler paths. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(messaging): add BackendInstallProgressEvent wire type and subject New NATS subject nodes.<nodeID>.backend.install.<opID>.progress lets the worker publish transient progress events (file, current/total bytes, percentage, phase) while a long-running install pulls its OCI image. BackendInstallRequest gains an optional OpID field so the worker knows which subject to publish on. Transient pub/sub (not JetStream): the install reply remains ground truth for success/failure; dropped progress events are tolerable. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * style(messaging): drop em-dash from BackendInstallProgress test comment Per project convention (no em-dashes anywhere). Comment substance is unchanged. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): worker publishes debounced install progress over NATS When BackendInstallRequest.OpID is set, the worker's backend.install handler wires a debounced publisher (250ms window) into the gallery download callback. Each tick becomes a BackendInstallProgressEvent on nodes.<nodeID>.backend.install.<opID>.progress; the publisher always emits a final event on Flush so the UI sees the terminal percentage. Old masters that do not set OpID continue to run silent installs: no behavior change for them. Lock ordering: the publisher releases its mutex before calling messaging.Publish so a slow network never stalls the install loop. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): RemoteUnloaderAdapter subscribes to install progress InstallBackend gains opID + onProgress parameters. When both are set, the adapter subscribes to nodes.<nodeID>.backend.install.<opID>.progress BEFORE publishing the install request, decodes each message into the caller's onProgress callback in a goroutine (so a slow callback never stalls the NATS reader thread), and unsubscribes after RequestJSON returns. When onProgress is nil OR opID is empty (the reconciler retry path), subscription is skipped entirely - silent installs cost nothing extra. Subscribe failure is logged at Warn and the install proceeds without progress streaming; the NATS round-trip still owns terminal status. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): forward backend install progress into galleryop OpStatus DistributedBackendManager.InstallBackend now passes the gallery op ID and a progress bridge into the adapter call. Each BackendInstallProgressEvent from the worker becomes a galleryop.ProgressCallback tick - which the existing backendHandler already turns into OpStatus.UpdateStatus, so the admin UI/SSE polling sees per-byte progress for distributed installs without any UI-side change. UpgradeBackend is intentionally left silent for now: its wire request (BackendUpgradeRequest) does not carry OpID, and rolling-update fallback is the rarer path. Will be picked up in a follow-up if the worker upgrade path also gets a progress channel. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(distributed): InstallBackend tolerates silent (pre-Phase-2) workers A worker on pre-Phase-2 code never publishes progress events. The new master subscribes optimistically; this spec pins that a silent worker still produces a green install with no progressCb ticks. The install reply is the source of truth for terminal state; the progress stream is a best-effort UX enrichment. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(distributed): document install progress streaming Note the new nodes.<nodeID>.backend.install.<opID>.progress subject and the silent-worker compatibility behavior so operators know to expect real-time progress and what happens on a mixed-version cluster. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(distributed): note progress-event ordering trade-off in InstallBackend Document near the goroutine dispatch why ordering at the consumer is best-effort, why it rarely matters in practice (worker debounce >> goroutine jitter), and what a future hardening pass would look like (Seq field + stale-by-seq drop). Stops the next reader from accidentally "fixing" the goroutine pool away. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(galleryop): add NodeProgress + OpStatus.Nodes for per-node breakdown Adds the data model the UI needs to render an expandable per-node breakdown of a fanned-out backend install. NodeProgress carries node identity (ID + name), per-node status (queued / running_on_worker / success / error / downloading), the current file + bytes + percentage from the Phase 2 progress stream, and any per-node error. OpStatus.Nodes is the slice the /api/operations handler will surface in a follow-up. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(galleryop): UpdateNodeProgress merges per-node ticks by NodeID GalleryService.UpdateNodeProgress(opID, nodeID, np) merges a NodeProgress into OpStatus.Nodes (keyed by NodeID, no duplicates) and mirrors the latest tick into the aggregate Progress / FileName / DownloadedFileSize / TotalFileSize fields so the legacy single-bar OperationsBar view keeps working unchanged alongside the new per-node breakdown. Concurrent-safe via the existing g.Mutex. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): write per-node OpStatus entries during install fan-out DistributedBackendManager now accepts a nodeProgressSink and feeds it two streams: 1. enqueueAndDrainBackendOp emits a per-node terminal entry on each status it appends to BackendOpResult (queued, success, error, running_on_worker). The opID is threaded through the function so the sink gets the right gallery op identity. 2. The install apply closure fans each BackendInstallProgressEvent into the sink as a downloading entry, alongside the legacy progressCb path so the aggregate single-bar view stays correct. Production wiring passes the GalleryService (which implements UpdateNodeProgress via Task 2) as the sink. Single-node tests pass nil. DeleteBackend and UpgradeBackend pass an empty opID so the sink path no-ops for ops that aren't gallery-tracked the same way as Install. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(operations): expose per-node breakdown on /api/operations When an operation's OpStatus has Nodes entries (populated by the Phase 4 progress sink wiring), surface them as a "nodes" array on the /api/operations response, sorted by node_name for stable rendering. Backward compatible: legacy clients ignore the field; ops without any node entries (single-node mode, model installs) omit the array entirely thanks to the empty-slice guard. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): per-node breakdown in OperationsBar When an install op fans out to more than one worker, the operations bar now shows a "N nodes" chevron that expands into a per-node list. Each row carries the node's status (color-coded pill), the current file being downloaded, byte counts, percentage, and a thin per-node progress bar. Yellow "Worker busy" pill marks running_on_worker status with a tooltip explaining the NATS round-trip timed out but the worker is still installing in the background. Backward compatible: ops without a nodes field (legacy or single-node mode) render as before. State for expand/collapse is local to the component, keyed by jobID/id - reload starts collapsed. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(distributed): document per-node breakdown in the operations bar Adds a short subsection covering the expandable "N nodes" chevron in the OperationsBar admin UI, the meaning of each status pill, and how it relates to the /api/operations nodes array. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(galleryop): UpdateStatus preserves Nodes when caller sends none Real-world bug surfaced by the Phase 4 multi-worker smoke test: the nodes[] array in /api/operations flickered between a single node at a time on a 2-worker install. Root cause: the Phase 2 progress bridge also calls the legacy progressCb -> UpdateStatus(&OpStatus{...}) on every tick. UpdateStatus then overwrote the entire status pointer, wiping the Nodes slice that UpdateNodeProgress had just merged in. Fix: in UpdateStatus, if the incoming op has an empty Nodes slice, carry forward the previous status's Nodes before storing. Callers that explicitly populate Nodes still win (their slice replaces the prior one, no merge across the two code paths). Two regression specs added pinning both directions of the contract. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(distributed): strip implementation details from user-facing docs Trim the new install/upgrade timeout rows and the install-progress sections to focus on what the operator sees and tunes. Drops: - the NATS subject names and pub/sub mechanics - "round-trip" / reconciler / backend.list jargon - /api/operations polling cadence - "pre-2026-05-22" version references Reframes the breakdown text around the admin UI (Operations Bar, chevron, status pills, "Worker busy" tooltip). Implementation context lives in the agent notes and code comments. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(config): move DistributedConfig.Validate flag names to constants The negative-duration check map was a wall of literal kebab-case strings that had to stay in sync with the kong-derived CLI flag names manually. Move them to a Flag* const block alongside the existing Default* block so a rename of either the Go field or the CLI naming convention forces a compile error rather than silent drift. Sole consumer today is Validate; the constants are exported so future operator-facing surfaces (e.g. error messages on other validation paths) can reference them by name instead of repeating the literals. Tests pin both the literal values (so a future "let's just rename this" doesn't accidentally regress the CLI flag) and the negative- duration error message for the new BackendInstall / BackendUpgrade fields. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(distributed): extract NodeStatus and Phase enums to constants Sweep for the same literal-string-as-identifier pattern called out on the Validate flag names: the per-node install status enum ("queued" | "downloading" | "running_on_worker" | "success" | "error") appeared as raw literals across managers_distributed.go (10+ sites, including 3 separate `n.Status == "running_on_worker"` checks), operation.go, and the test suite. Same shape for the Phase enum ("resolving" | "downloading" | "extracting" | "starting") in the worker-side progress publisher. Promote both to exported const blocks: - galleryop.NodeStatus{Queued,Downloading,RunningOnWorker,Success,Error} shared between galleryop.NodeProgress.Status (the wire field) and nodes.NodeOpStatus.Status (the in-process per-node summary) - messaging.Phase{Resolving,Downloading,Extracting,Starting} shared between the worker publisher and any future consumer that needs to switch on phase Tests pin both the literal values (so a future "let's just rename" doesn't silently change the JSON wire) and use the constants in setup (so the producer side stays drift-protected). Wire-format assertions on the /api/operations JSON output keep their literals deliberately, so the constant value can never silently diverge from what the UI receives. Out of scope for this PR (separate cleanup): the finetune and quantization job-status enums have the same anti-pattern with 14+ literal sites each, but predate this PR's work. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
LocalAI [bot]andEttore Di Giacinto
|
61bf34ea2f |
fix(traces): cap captured body size to keep admin Traces UI responsive (#9946)
The trace middleware buffered the full request and response bodies for every JSON exchange. With a chatty agent-pool RAG workload, /embeddings responses (large vector arrays) accumulated to tens of MB in the in-memory buffer; the admin Traces page would then download and parse 40+ MB on every load and on every 5s auto-refresh, locking the UI in a loading state. Add LOCALAI_TRACING_MAX_BODY_BYTES (default 64 KiB) that caps each captured body. The full payload still flows through to the real client; only the trace copy is bounded. Exchanges record body_truncated and original body_bytes so the dashboard can show that truncation happened. The cap is configurable via env, CLI, and runtime_settings.json. Also unblock recovery: the Traces page now keeps the Clear button enabled while loading, since "buffer too large to render" is exactly when the user needs to clear it. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Richard Palethorpe
|
5d0b549049 |
feat(gallery): verify backend OCI images with keyless cosign (#9823)
* feat(gallery): verify backend OCI images with keyless cosign Close a trust gap where a registry compromise or MITM could silently replace a backend image: the gallery YAML tells LocalAI which image to pull, but until now nothing verified the bytes came from our CI. Consumer (pkg/oci/cosignverify): - New package using sigstore-go to verify keyless-cosign signatures. - OCI 1.1 referrers API + new bundle format (no legacy :tag.sig). - Policy fields: Issuer / IssuerRegex / Identity / IdentityRegex / NotBefore. NotBefore is the revocation lever — keyless Fulcio certs are ephemeral so revocation is policy-side; advancing not_before in the gallery YAML invalidates every signature predating the cutoff. - TUF trusted root cached process-wide so N backends from one gallery do 1 fetch, not N. Plumbing: - pkg/downloader: ImageVerifier interface + WithImageVerifier option threaded through DownloadFileWithContext. Verification runs between oci.GetImage and oci.ExtractOCIImage, with digest pinning via pinnedImageRef to close the TOCTOU window. Skips the verifier's HEAD when the ref is already digest-pinned. - core/config: Gallery.Verification YAML block. - core/gallery: backendDownloadOptions builds the verifier from the policy; applied on initial URI, mirrors, and tag fallbacks. - core/gallery/upgrade: the upgrade path now routes through the same options builder. A regression Ginkgo spec pins this contract — without it, UpgradeBackend silently bypassed verification. - core/cli: --require-backend-integrity (LOCALAI_REQUIRE_BACKEND_INTEGRITY) escalates missing policy / empty SHA256 from warn to hard-fail. Producer (.github/workflows/backend_merge.yml): - id-token: write at job scope (PR-fork-safe via existing event gate). - sigstore/cosign-installer@v3 pinned to v2.4.1. - After each docker buildx imagetools create, resolve the manifest list digest and run cosign sign --recursive --new-bundle-format --registry-referrers-mode=oci-1-1 against repo@digest. --recursive signs the index and every per-arch entry, matching how the consumer resolves a tag to a platform-specific manifest before verifying. Rollout: backend/index.yaml has no `verification:` block yet, so this PR is backward-compatible — installs proceed with a warning until the gallery is populated. Strict mode is opt-in. Assisted-by: claude-code:claude-opus-4-7 [Bash] [Edit] [Read] [Write] [WebSearch] [WebFetch] Signed-off-by: Richard Palethorpe <io@richiejp.com> * refactor(gallery): plumb RequireBackendIntegrity through config instead of env The previous implementation re-exported the --require-backend-integrity CLI flag into LOCALAI_REQUIRE_BACKEND_INTEGRITY via os.Setenv, then re-read it in core/gallery via os.Getenv. This leaked process state into the gallery package and made the flag impossible to override per-call or test without touching the env. Add RequireBackendIntegrity to ApplicationConfig (with a matching WithRequireBackendIntegrity AppOption) and thread the bool through every install/upgrade path: InstallBackend, InstallBackendFromGallery, UpgradeBackend, InstallModelFromGallery, InstallExternalBackend, ApplyGalleryFromString/File, startup.InstallModels. Worker subcommands gain the same env-bound flag on WorkerFlags so distributed-worker installs honor it consistently with the worker daemon path. Add a forbidigo lint rule against os.Getenv / os.LookupEnv / os.Environ to keep the env-leak pattern from creeping back. Existing offenders (p2p, config loaders, etc.) are baseline-grandfathered by the existing new-from-merge-base: origin/master setting; targeted path exclusions cover the legitimate cases — kong CLI entry points, backend subprocesses, system capability probes, gRPC AUTH_TOKEN inheritance, test gating env vars. Assisted-by: claude-code:claude-opus-4-7 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
Richard Palethorpe
|
969005b2a1 |
feat(gallery): Speed up load times and clean gallery entries (#9211)
* feat: Rework VRAM estimation and use known_usecases in gallery Signed-off-by: Richard Palethorpe <io@richiejp.com> Assisted-by: Claude:claude-opus-4-7[1m] [Claude Code] * chore(gallery): regenerate gallery index and add known_usecases to model entries Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> |
||
|
Ettore Di Giacinto
|
bbcaebc1ef |
feat(concurrency-groups): per-model exclusive groups for backend loading (#9662)
* feat(concurrency-groups): per-model exclusive groups for backend loading Adds `concurrency_groups: [...]` to model YAML configs. Two models that share a group cannot be loaded concurrently on the same node — loading one evicts the others, reusing the existing pinned/busy/retry policy from LRU eviction. Layered design: - Watchdog (pkg/model): per-node correctness floor — on every Load(), evict any loaded model that shares a group with the requested one. Pinned skips surface NeedMore so the loader retries (and ultimately logs a clear warning), instead of silently allowing the rule to be violated. - Distributed scheduler (core/services/nodes): soft anti-affinity hint — scheduleNewModel prefers nodes that don't already host a same-group model, falling back to eviction only if every candidate has a conflict. Composes with NodeSelector at the same point in the candidate pipeline. Per-node, not cluster-wide: VRAM is a node-local resource, and two heavy models running on different nodes is fine. The ConfigLoader is wired into SmartRouter via a small ConcurrencyConflictResolver interface so the nodes package keeps a narrow surface on core/config. Refactors the inner LRU eviction body into a shared collectEvictionsLocked helper and the loader retry loop into retryEnforce(fn, maxRetries, interval), so both LRU and group enforcement share busy/pinned/retry semantics. Closes #9659. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(watchdog): sync pinned + concurrency_groups at startup The startup-time watchdog setup lives in initializeWatchdog (startup.go), not in startWatchdog (watchdog.go). The latter is only invoked from the runtime-settings RestartWatchdog path. As a result, neither SyncPinnedModelsToWatchdog nor SyncModelGroupsToWatchdog ran at boot, so `pinned: true` and `concurrency_groups: [...]` only became effective after a settings-driven watchdog restart. Fix by adding both sync calls to initializeWatchdog. Confirmed end-to-end: loading model A in group "heavy", then C with no group (coexists), then B in group "heavy" now correctly evicts A and leaves [B, C]. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(test): satisfy errcheck on new os.Remove in concurrency_groups spec CI lint runs new-from-merge-base, so the existing pre-existing `defer os.Remove(tmp.Name())` lines are baseline-grandfathered but the one introduced by the concurrency_groups YAML round-trip test is held to errcheck. Wrap the remove in a closure that discards the error. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> |
||
|
Ettore Di Giacinto
|
b1a99436c7 |
feat(branding): admin-configurable instance name, tagline, and assets (#9635)
Adds a whitelabeling feature so an operator can replace the LocalAI
instance name, tagline, square logo, horizontal logo, and favicon from
the admin Settings page. Defaults fall back to the bundled assets so
existing installs are unaffected.
The public GET /api/branding endpoint is reachable pre-auth so the
login screen can render the configured branding before sign-in.
Mutating routes (POST/DELETE /api/branding/asset/:kind) remain
admin-only. Text fields (instance_name, instance_tagline) ride the
existing /api/settings flow; binary assets get a dedicated multipart
upload route that persists files under DynamicConfigsDir/branding/.
To prevent the Settings page's stale local state from clobbering an
upload on save, UpdateSettingsEndpoint preserves whatever the on-disk
asset filename fields are regardless of the body — /api/branding/asset/*
are the sole writers for those fields.
The MCP catalog gains get_branding and set_branding tools (text fields
only; file upload stays UI-only) plus a configure_branding skill prompt.
While wiring this up, the same restart-loss class of bug surfaced for
several existing fields whose RuntimeSettings entries were never read
by the startup loader. Fix loadRuntimeSettingsFromFile() to load:
- branding (instance_name, instance_tagline, *_file basenames)
- auto_upgrade_backends, prefer_development_backends
- localai_assistant_enabled
- open_responses_store_ttl
- the 7 existing AgentPool fields (enabled, default/embedding model,
chunking sizes, enable_logs, collection_db_path)
Also exposes 3 new AgentPool runtime settings (vector_engine,
database_url, agent_hub_url) via /api/settings + the Settings UI, with
the same load-on-startup wiring. The file watcher's manual-edit path
is intentionally not changed — the in-process API endpoints already
update appConfig directly, so the watcher is redundant for supported
flows and a separate refactor for everything else.
15 TDD specs cover the loader behaviour (1 branding + 11 adjacent + 3
new agent-pool); 2 specs cover the persistence helpers and the
clobber-prevention contract.
Assisted-by: claude-code:claude-opus-4-7
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
|
||
|
Ettore Di Giacinto
|
4906cbad04 |
feat: add biometrics UI (#9524)
* feat(react-ui): add Face & Voice Recognition pages
Expose the face and voice biometrics endpoints
(/v1/face/*, /v1/voice/*) through the React UI. Each page has four
tabs driving the six endpoints per modality: Analyze (demographics
with bounding boxes / waveform segments), Compare (verify with a
match gauge and live threshold slider), Enrollment (register /
identify / forget with a top-K matches view), Embedding (raw
vector inspector with sparkline + copy).
MediaInput supports file upload plus live capture: webcam
snap-to-canvas for face, MediaRecorder -> AudioContext ->
16-bit PCM mono WAV transcode for voice (libsndfile on the
backend only handles WAV/FLAC/OGG natively).
Sidebar gets a new Biometrics section feature-gated on
face_recognition / voice_recognition; routes are wrapped in
<RequireFeature>. No new dependencies -- Font Awesome icons
picked from the Free set.
Assisted-by: Claude:Opus 4.7
* fix(localai): accept data URI prefixes with codec/charset params
Browser MediaRecorder produces data URIs like
data:audio/webm;codecs=opus;base64,...
so the pre-';base64,' section can carry multiple parameter
segments. The `^data:([^;]+);base64,` regex in pkg/utils/base64.go
and core/http/endpoints/localai/audio.go only matched exactly one
segment, so recordings straight from the React UI's live-capture
tab failed the strip and then tripped the base64 decoder on the
leading 'data:' literal, surfacing as
"invalid audio base64: illegal base64 data at input byte 4"
Widened both regexes to `^data:[^,]+?;base64,` so any number of
';param=value' segments between the mime type and ';base64,' are
tolerated. Added a regression test covering the MediaRecorder
shape.
Assisted-by: Claude:Opus 4.7
* fix(insightface): scope pack ONNX loading to known manifests
LocalAI's gallery extracts buffalo_* zips flat into the models
directory, which inevitably mixes with ONNX files from other
backends (opencv face engine, MiniFASNet antispoof, WeSpeaker
voice embedding) and older buffalo pack installs. Feeding those
foreign files into insightface's model_zoo.get_model() blows up
inside the router -- it assumes a 4-D NCHW input and indexes
`input_shape[2]` on tensors that aren't shaped like a face model,
raising IndexError mid-load and leaving the backend unusable.
The router's dispatch isn't amenable to per-file try/except alone
(first-file-wins picks det_10g.onnx from buffalo_l even when the
user asked for buffalo_sc -- alphabetical order happens to favour
the wrong pack). Instead, ship an explicit manifest of the
upstream v0.7 pack contents and scope the glob to that when the
requested pack is known. The manifest is small and stable; future
packs can be added alongside or fall through to the tolerance
loop, which also swallows any remaining IndexError / ValueError
from foreign files with a clear `[insightface] skipped` stderr
line for diagnostics.
Assisted-by: Claude:Opus 4.7
* fix(speaker-recognition): extract FBank features for rank-3 ONNX encoders
Pre-exported speaker-encoder ONNX graphs come in two shapes:
rank-2 [batch, samples] -- some 3D-Speaker exports,
take raw waveform directly.
rank-3 [batch, frames, n_mels] -- WeSpeaker and most Kaldi-
lineage encoders, expect
pre-computed Kaldi FBank.
OnnxDirectEngine unconditionally fed `audio.reshape(1, -1)` --
correct for rank-2, IndexError-on-input_shape[3] on rank-3, which
surfaced to the UI as
"Invalid rank for input: feats Got: 2 Expected: 3"
Detect the input rank at session init and run Kaldi FBank
(80-dim, 25ms/10ms frames, dither=0.0, per-utterance CMN) before
the forward pass when rank>=3. All knobs are configurable via
backend options for encoders that deviate from defaults.
torchaudio.compliance.kaldi is already in the backend's
requirements (SpeechBrain pulls torchaudio in), so no new
dependency.
Assisted-by: Claude:Opus 4.7
* fix(biometrics): isolate face and voice vector stores
Face (ArcFace, 512-D) and voice (ECAPA-TDNN 192-D / WeSpeaker
256-D) biometric embeddings were colliding inside a single
in-memory local-store instance. Enrolling one after the other
failed with
"Try to add key with length N when existing length is M"
because local-store correctly refuses to mix dimensions in one
keyspace.
The registries were constructed with `storeName=""`, which in
StoreBackend() is just a WithModel() call. But ModelLoader's
cache is keyed on `modelID`, not `model` -- so both registries
collapsed to the same `modelID=""` slot and reused the same
backend process despite looking isolated on paper.
Three complementary fixes:
1. application.go -- give each registry a distinct default
namespace ("localai-face-biometrics" /
"localai-voice-biometrics"). The comment claimed
isolation, now it's actually enforced.
2. stores.go -- pass the storeName as both WithModelID and
WithModel so the ModelLoader cache key separates
namespaces and the loader spawns distinct processes.
3. local-store/store.go -- drop the Load() `opts.Model != ""`
guard. It was there to prevent generic model-loading loops
from picking up local-store by accident, but that auto-load
path is being retired; the guard now just blocks legitimate
namespace isolation. opts.Model is treated as a tag; the
per-tuple process isolation upstream handles discrimination.
Assisted-by: Claude:Opus 4.7
* fix(gallery): stale-file cleanup and upgrade-tmp directory safety
Two related robustness fixes for backend install/upgrade:
pkg/downloader/uri.go
OCI downloads passed through
if filepath.Ext(filePath) != "" ...
filePath = filepath.Dir(filePath)
which was intended to redirect file-shaped download targets
into their parent directory for OCI extraction. The heuristic
misfires on directory-shaped paths with a dot-suffix --
gallery.UpgradeBackend uses
tmpPath = "<backendsPath>/<name>.upgrade-tmp"
and Go's filepath.Ext treats ".upgrade-tmp" as an extension.
The rewrite landed the extraction at "<backendsPath>/", which
then **overwrote the real install** (backends/<name>/) with a
flat-layout file and left a stray run.sh at the top level. The
tmp dir itself stayed empty, so the validation step that
checked "<tmpPath>/run.sh" predictably failed with
"upgrade validation failed: run.sh not found in new backend"
Every manual upgrade silently corrupted the backends tree this
way. Guard the rewrite behind "target isn't already an existing
directory" -- InstallBackend / UpgradeBackend both pre-create
the target as a directory, so they get the correct behaviour;
existing file-path callers with a genuine dot-extension still
get the parent redirect.
core/gallery/backends.go
InstallBackend's MkdirAll returned ENOTDIR when something at
the target path was already a file (legacy dev builds dropped
golang backend binaries directly at `<backendsPath>/<name>`
instead of nesting them under their own subdir). That
permanently blocked reinstall and upgrade for anyone carrying
that state, since every retry hit the same error. Detect a
pre-existing non-directory, warn, and remove it before the
MkdirAll so the fresh install can write the correct nested
layout with metadata.json + run.sh.
Assisted-by: Claude:Opus 4.7
* fix(galleryop): refresh upgrade cache after backend ops
UpgradeChecker caches the last upgrade-check result and only
refreshes on the 6-hour tick or after an auto-upgrade cycle.
Manual upgrades (POST /api/backends/upgrade/:name) go through
the async galleryop worker, which completes the upgrade
correctly but never tells UpgradeChecker to re-check -- so
/api/backends/upgrades continued to list a just-upgraded backend
as upgradeable, indistinguishable from a failed upgrade, for up
to six hours.
Add an optional `OnBackendOpCompleted func()` hook on
GalleryService that fires after every successful install /
upgrade / delete on the backend channel (async, so a slow
callback doesn't stall the queue). startup.go wires it to
UpgradeChecker.TriggerCheck after both services exist. Result:
the upgrade banner clears within milliseconds of the worker
finishing.
Assisted-by: Claude:Opus 4.7
* build: prepend GOPATH/bin to PATH for protogen-go
install-go-tools runs `go install` for protoc-gen-go and
protoc-gen-go-grpc, which writes them into `go env GOPATH`/bin.
That directory isn't on every dev's PATH, and protoc resolves
its code-gen plugins via PATH, so the immediately-following
protoc invocation fails with
"protoc-gen-go: program not found"
which in turn blocks `make build` and any
`make backends/%` target that depends on build.
Prepend `go env GOPATH`/bin to PATH for the protoc invocation
so the freshly-installed plugins are found without requiring a
shell-profile change.
Assisted-by: Claude:Opus 4.7
* refactor(ui-api): non-blocking backend upgrade handler with opcache
POST /api/backends/upgrade/:name used to send the ManagementOp
directly onto the unbuffered BackendGalleryChannel, which blocked
the HTTP request whenever the galleryop worker was busy with a
prior operation. The op also didn't show up in /api/operations,
so the Backends UI couldn't reflect upgrade progress on the
affected row.
Register the op in opcache immediately, wrap it in a cancellable
context, store the cancellation function on the GalleryService,
and push onto the channel from a goroutine so the handler
returns right away. Response gains a `jobID` field and a
`message` string so clients have a consistent handle regardless
of whether the op is queued or running.
Pairs with the OnBackendOpCompleted hook added in the galleryop
commit — together the UI sees the upgrade start, watches
progress via /api/operations, and drops the "upgradeable" flag
the moment the worker finishes.
Assisted-by: Claude:Opus 4.7
|
||
|
Ettore Di Giacinto
|
75a63f87d8 |
feat(distributed): sync state with frontends, better backend management reporting (#9426)
* fix(distributed): detect backend upgrades across worker nodes
Before this change `DistributedBackendManager.CheckUpgrades` delegated to the
local manager, which read backends from the frontend filesystem. In
distributed deployments the frontend has no backends installed locally —
they live on workers — so the upgrade-detection loop never ran and the UI
silently never surfaced upgrades even when the gallery advertised newer
versions or digests.
Worker-side: NATS backend.list reply now carries Version, URI and Digest
for each installed backend (read from metadata.json).
Frontend-side: DistributedBackendManager.ListBackends aggregates per-node
refs (name, status, version, digest) instead of deduping, and CheckUpgrades
feeds that aggregation into gallery.CheckUpgradesAgainst — a new entrypoint
factored out of CheckBackendUpgrades so both paths share the same core
logic.
Cluster drift policy: when per-node version/digest tuples disagree, the
backend is flagged upgradeable regardless of whether any single node
matches the gallery, and UpgradeInfo.NodeDrift enumerates the outliers so
operators can see *why* it is out of sync. The next upgrade-all realigns
the cluster.
Tests cover: drift detection, unanimous-match (no upgrade), and the
empty-installed-version path that the old distributed code silently
missed.
* feat(ui): surface backend upgrades in the System page
The System page (Manage.jsx) only showed updates as a tiny inline arrow,
so operators routinely missed them. Port the Backend Gallery's upgrade UX
so System speaks the same visual language:
- Yellow banner at the top of the Backends tab when upgrades are pending,
with an "Upgrade all" button (serial fan-out, matches the gallery) and a
"Updates only" filter toggle.
- Warning pill (↑ N) next to the tab label so the count is glanceable even
when the banner is scrolled out of view.
- Per-row labeled "Upgrade to vX.Y" button (replaces the icon-only button
that silently flipped semantics between Reinstall and Upgrade), plus an
"Update available" badge in the new Version column.
- New columns: Version (with upgrade + drift chips), Nodes (per-node
attribution badges for distributed mode, degrading to a compact
"on N nodes · M offline" chip above three nodes), Installed (relative
time).
- System backends render a "Protected" chip instead of a bare "—" so rows
still align and the reason is obvious.
- Delete uses the softer btn-danger-ghost so rows don't scream red; the
ConfirmDialog still owns the "are you sure".
The upgrade checker also needed the same per-worker fix as the previous
commit: NewUpgradeChecker now takes a BackendManager getter so its
periodic runs call the distributed CheckUpgrades (which asks workers)
instead of the empty frontend filesystem. Without this the /api/backends/
upgrades endpoint stayed empty in distributed mode even with the protocol
change in place.
New CSS primitives — .upgrade-banner, .tab-pill, .badge-row, .cell-stack,
.cell-mono, .cell-muted, .row-actions, .btn-danger-ghost — all live in
App.css so other pages can adopt them without duplicating styles.
* feat(ui): polish the Nodes page so it reads like a product
The Nodes page was the biggest visual liability in distributed mode.
Rework the main dashboard surfaces in place without changing behavior:
StatCards: uniform height (96px min), left accent bar colored by the
metric's semantic (success/warning/error/primary), icon lives in a
36x36 soft-tinted chip top-right, value is left-aligned and large.
Grid auto-fills so the row doesn't collapse on narrow viewports. This
replaces the previous thin-bordered boxes with inconsistent heights.
Table rows: expandable rows now show a chevron cue on the left (rotates
on expand) so users know rows open. Status cell became a dedicated chip
with an LED-style halo dot instead of a bare bullet. Action buttons gained
labels — "Approve", "Resume", "Drain" — so the icons aren't doing all
the semantic work; the destructive remove action uses the softer
btn-danger-ghost variant so rows don't scream red, with the ConfirmDialog
still owning the real "are you sure". Applied cell-mono/cell-muted
utility classes so label chips and addresses share one spacing/font
grammar instead of re-declaring inline styles everywhere.
Expanded drawer: empty states for Loaded Models and Installed Backends
now render as a proper drawer-empty card (dashed border, icon, one-line
hint) instead of a plain muted string that read like broken formatting.
Tabs: three inline-styled buttons became the shared .tab class so they
inherit focus ring, hover state, and the rest of the design system —
matches the System page.
"Add more workers" toggle turned into a .nodes-add-worker dashed-border
button labelled "Register a new worker" (action voice) instead of a
chevron + muted link that operators kept mistaking for broken text.
New shared CSS primitives carry over to other pages:
.stat-grid + .stat-card, .row-chevron, .node-status, .drawer-empty,
.nodes-add-worker.
* feat(distributed): durable backend fan-out + state reconciliation
Two connected problems handled together:
1) Backend delete/install/upgrade used to silently skip non-healthy nodes,
so a delete during an outage left a zombie on the offline node once it
returned. The fan-out now records intent in a new pending_backend_ops
table before attempting the NATS round-trip. Currently-healthy nodes
get an immediate attempt; everyone else is queued. Unique index on
(node_id, backend, op) means reissuing the same operation refreshes
next_retry_at instead of stacking duplicates.
2) Loaded-model state could drift from reality: a worker OOM'd, got
killed, or restarted a backend process would leave a node_models row
claiming the model was still loaded, feeding ghost entries into the
/api/nodes/models listing and the router's scheduling decisions.
The existing ReplicaReconciler gains two new passes that run under a
fresh KeyStateReconciler advisory lock (non-blocking, so one wedged
frontend doesn't freeze the cluster):
- drainPendingBackendOps: retries queued ops whose next_retry_at has
passed on currently-healthy nodes. Success deletes the row; failure
bumps attempts and pushes next_retry_at out with exponential backoff
(30s → 15m cap). ErrNoResponders also marks the node unhealthy.
- probeLoadedModels: gRPC-HealthChecks addresses the DB thinks are
loaded but hasn't seen touched in the last probeStaleAfter (2m).
Unreachable addresses are removed from the registry. A pluggable
ModelProber lets tests substitute a fake without standing up gRPC.
DistributedBackendManager exposes DeleteBackendDetailed so the HTTP
handler can surface per-node outcomes ("2 succeeded, 1 queued") to the
UI in a follow-up commit; the existing DeleteBackend still returns
error-only for callers that don't care about node breakdown.
Multi-frontend safety: the state pass uses advisorylock.TryWithLockCtx
on a new key so N frontends coordinate — the same pattern the health
monitor and replica reconciler already rely on. Single-node mode runs
both passes inline (adapter is nil, state drain is a no-op).
Tests cover the upsert semantics, backoff math, the probe removing an
unreachable model but keeping a reachable one, and filtering by
probeStaleAfter.
* feat(ui): show cluster distribution of models in the System page
When a frontend restarted in distributed mode, models that workers had
already loaded weren't visible until the operator clicked into each node
manually — the /api/models/capabilities endpoint only knew about
configs on the frontend's filesystem, not the registry-backed truth.
/api/models/capabilities now joins in ListAllLoadedModels() when the
registry is active, returning loaded_on[] with node id/name/state/status
for each model. Models that live in the registry but lack a local config
(the actual ghosts, not recovered from the frontend's file cache) still
surface with source="registry-only" so operators can see and persist
them; without that emission they'd be invisible to this frontend.
Manage → Models replaces the old Running/Idle pill with a distribution
cell that lists the first three nodes the model is loaded on as chips
colored by state (green loaded, blue loading, amber anything else). On
wider clusters the remaining count collapses into a +N chip with a
title-attribute breakdown. Disabled / single-node behavior unchanged.
Adopted models get an extra "Adopted" ghost-icon chip with hover copy
explaining what it means and how to make it permanent.
Distributed mode also enables a 10s auto-refresh and a "Last synced Xs
ago" indicator next to the Update button so ghost rows drop off within
one reconcile tick after their owning process dies. Non-distributed
mode is untouched — no polling, no cell-stack, same old Running/Idle.
* feat(ui): NodeDistributionChip — shared per-node attribution component
Large clusters were going to break the Manage → Backends Nodes column:
the old inline logic rendered every node as a badge and would shred the
layout at >10 workers, plus the Manage → Models distribution cell had
copy-pasted its own slightly-different version.
NodeDistributionChip handles any cluster size with two render modes:
- small (≤3 nodes): inline chips of node names, colored by health.
- large: a single "on N nodes · M offline · K drift" summary chip;
clicking opens a Popover with a per-node table (name, status,
version, digest for backends; name, status, state for models).
Drift counting mirrors the backend's summarizeNodeDrift so the UI
number matches UpgradeInfo.NodeDrift. Digests are truncated to the
docker-style 12-char form with the full value preserved in the title.
Popover is a new general-purpose primitive: fixed positioning anchored
to the trigger, flips above when there's no room below, closes on
outside-click or Escape, returns focus to the trigger. Uses .card as
its surface so theming is inherited. Also useful for a future
labels-editor popup and the user menu.
Manage.jsx drops its duplicated inline Nodes-column + loaded_on cell
and uses the shared chip with context="backends" / "models"
respectively. Delete code removes ~40 lines of ad-hoc logic.
* feat(ui): shared FilterBar across the System page tabs
The Backends gallery had a nice search + chip + toggle strip; the System
page had nothing, so the two surfaces felt like different apps. Lift the
pattern into a reusable FilterBar and wire both System tabs through it.
New component core/http/react-ui/src/components/FilterBar.jsx renders a
search input, a role="tablist" chip row (aria-selected for a11y), and
optional toggles / right slot. Chips support an optional `count` which
the System page uses to show "User 3", "Updates 1" etc.
System Models tab: search by id or backend; chips for
All/Running/Idle/Disabled/Pinned plus a conditional Distributed chip in
distributed mode. "Last synced" + Update button live in the right slot.
System Backends tab: search by name/alias/meta-backend-for; chips for
All/User/System/Meta plus conditional Updates / Offline-nodes chips
when relevant. The old ad-hoc "Updates only" toggle from the upgrade
banner folded into the Updates chip — one source of truth for that
filter. Offline chip only appears in distributed mode when at least
one backend has an unhealthy node, so the chip row stays quiet on
healthy clusters.
Filter state persists in URL query params (mq/mf/bq/bf) so deep links
and tab switches keep the operator's filter context instead of
resetting every time.
Also adds an "Adopted" distribution path: when a model in
/api/models/capabilities carries source="registry-only" (discovered on
a worker but not configured locally), the Models tab shows a ghost chip
labelled "Adopted" with hover copy explaining how to persist it — this
is what closes the loop on the ghost-model story end-to-end.
|