73 Commits

Include renames

Author	SHA1	Message	Date
mudler-agentandEttore Di Giacinto	4899d28e8a	feat(chats): server-side chats with steering, sub-agents and compaction (#12918) ... * Find the Sandbox guest image in the executable, not its build info The Sandbox smoke checked that the packaged executable carries the guest image by looking for -X main.guestImage in go version -m. The package is built with -trimpath, and with -trimpath Go leaves -ldflags out of the build info, so the check failed on every gvisor run, although the image was linked in. Look for the reference in the executable instead: the value of SANDBOX_GUEST_IMAGE when it is set, any digest reference otherwise. package.sh had the same check for an executable built elsewhere; it now looks for its own reference the same way. Assisted-by: Claude:claude-opus-5-5 * Answer a network card the limit refuses with one precise reason A card that waited while the administrator's limit went down was refused on the first click with 400, and every later click was a bare 409, so a stale card looked like a conflict loop. A refused card now answers 400 egress_above_ceiling every time, the same answer sent twice is one answer (204), and another answer to an answered card is 409 already_answered. The chat view marks a card the limit cannot admit with above_ceiling. When the limit is off, or below the chat's preset, the note of the turn and the block line of the proxy now say that the server's setting blocks the network and no approval changes it, so the model reports it instead of waiting or retrying. Assisted-by: Claude:claude-sonnet-5-5 * Explain a blocked Sandbox network instead of offering dead buttons A card the limit cannot admit shows why network access is off or limited, and who can change it, with a link to Settings > Tools > Sandbox network for administrators. The buttons of a card disable while an answer is sent, and a failed answer reads the card again. The Network row says why its choices are capped. Settings opens a group from ?group=. Strings are translated in all nine locales and the chat and add-on docs describe the new behaviour. Assisted-by: Claude:claude-sonnet-5-5 * Add an always-allowed hosts list to the Sandbox network The extra registries setting only widens the package registries, so a one-off host needs the approval card in every chat. The new admin setting chats_egress_allowed_hosts lists exact hosts (optionally host:port to pin a port) that a chat reaches whenever its effective preset is registries or all. The list sits in the same decision as the presets: the ceiling still caps it, the ports setting still applies, and a private, loopback, link-local or metadata address still needs the private switch and an administrator's rule or answer. Each use is audited with the reason allowed_host. Hosts are validated like the extra registries (lower case, punycode, no scheme, path or pattern, at most 64) and an IP literal is refused unless the private switch is on. get_chat_egress and the chat egress view return the list; no MCP tool writes it, as for the extra registries. Assisted-by: Claude:claude-sonnet-5-5 * Let users delete from the Trash for good A delete only set a tombstone, so the Trash grew without end and nothing could free the space of a deleted file. The workspace stores gain a Purger: it erases the deleted nodes of some change sets, or of a whole folder, with their journal rows, the change sets that nothing is left in, and the blobs that no row refers to. Both stores implement it and pass the same contract specs. A blob is removed after a grace period, and only if it is still unused then, so an upload of the same bytes in flight keeps them. DELETE /api/chats/files/trash empties the user's Trash and DELETE /api/chats/files/trash/{rev} deletes one entry. Both are scoped to the caller's home. The Trash listing now gives files, folders and bytes per entry and for the whole Trash. The file tools of a chat cannot reach any of this. Assisted-by: Claude:claude-sonnet-5-5 * Add Empty Trash and Delete permanently to the Files page The Trash tab shows what it holds, with an Empty Trash button that is off while the Trash is empty and a Delete permanently button on each item. Both ask in the page first and say how many files and folders and how much space they remove. Texts are translated in all nine locales. Assisted-by: Claude:claude-sonnet-5-5 * Fall back to Brave Search and Bing when DuckDuckGo asks for a captcha DuckDuckGo answers some servers with HTTP 202 and a bot check (the "anomaly" page) instead of results. The parser read that page as "no results it could read", so web_search failed for every query. The keyless search is now a chain: DuckDuckGo, then the result page of Brave Search, then the result page of Bing. Each page has its own tolerant parser (title, URL, snippet; Bing's /ck/a links and DDG's uddg links are unwrapped) with saved fixtures. A bot check is a typed error (ErrChallenged), and a 403 or 429 is ErrBlocked. Both wrap ErrProvider. A Fallback with a cool-down skips a challenged or blocked provider for a few minutes, so each search does not wait for it first; when every provider is cooling down, all are asked again. Fallback.SearchAttempts and TestEach report what each provider did (ok, no_results, challenged, blocked, timeout, error, skipped). The error a model reads now lists what was tried and what each provider did, then where an administrator fixes it. safehttp keeps the status code of an answer (Response.StatusCode) and of a refusal (StatusError, still ErrStatus for errors.Is). Assisted-by: Claude:claude-opus-5-5 * Use the keyless search chain in the chats and test each provider The built-in web_search now searches with DuckDuckGo, Brave Search and Bing in turn when no SearXNG instance is set. A new runtime setting, chats_web_search_providers, sets the order (searxng, duckduckgo, brave_html, bing_html) and leaves a provider out. Empty keeps the old choice: SearXNG alone when an instance is set, so its queries do not reach the public engines unless the administrator lists them. POST /api/chats/admin/web/test asks every provider of the order and returns what each did in "providers" (ok, no_results, challenged, blocked, timeout or error, with the reason), next to the first answer. Settings > Tools has a Test search on the web_search row that shows it, and the provider list is checked on the page and by the server. The cool-down of a challenged provider is kept per frontend, like the cache and the limits; web.md says why. Docs: web.md (providers, chain, bot checks, privacy, how to fix it with SearXNG or the Web add-on) and runtime-settings.md. Strings in the nine locales. Swagger regenerated. Assisted-by: Claude:claude-opus-5-5 * Try the mirror library alone in the reinstall recovery spec The spec adds a second library with the same URL and expects an install from it to keep the retained data apart. Since two enabled libraries that list one add-on are a conflict, the reinstall from the original library then failed with library_conflict, and the install from the mirror failed for that reason instead of the trusted identity. Switch the original library off while the mirror is tried, check that the mirror is refused for its identity and not for a conflict, then switch the mirror off and the original on for the reinstall. Assisted-by: Claude:claude-opus-5-5 * Pause a chat turn at the step limit and offer Continue A chat turn ran at most 10 rounds of tool calls, the default of the harness, because nothing set a limit. A multi-step task (read some files, run commands, write a result) stopped half way, and the person saw an answer that looked final. The limit is now the runtime setting chats_max_iterations (1 to 500, default 40, Settings > Tools); a model's agent.max_iterations still wins. When the tool loop reaches it, the harness sends a paused event with the limit, and the turn ends with last_turn "paused". A paused turn counts as finished: waiting messages start the next turn and a sub-agent reports back. The status line says "Paused, after 40 steps" and has a Continue button, which sends a message that asks the model to go on. A claim that found no waiting message released the chat as "done", which hid a pause. It now keeps the status the chat had. Assisted-by: Claude:claude-opus-5-5 * Edit the always-allowed Sandbox hosts in Settings > Tools Add a list editor for chats_egress_allowed_hosts next to the extra package registries: the hosts are tags with a remove button, an input adds one on Enter, comma or Add (a pasted list adds them all), and the reason an entry is refused shows under the input, with the count out of 64. The same checks stop Apply on a bad entry, and the server still has the last word. The chat's Network panel lists the hosts read only and the row says how many an administrator always allows while the chat's network is on. The help text of both settings now says how they differ. Strings are translated in all nine locales. Assisted-by: Claude:claude-sonnet-5-5 * Document the always-allowed Sandbox hosts Describe chats_egress_allowed_hosts in the runtime settings page, the network section of the add-ons page (including the allowed_host audit reason and how the list differs from extra registries) and the chats page. Assisted-by: Claude:claude-sonnet-5-5 * Let a chat edit, pause and delete the person's triggers A chat could propose a trigger and list them, but not change one: the Triggers page and the API could. list_triggers now gives each trigger a stable ref (not its id) and shows the prompt of one trigger only when asked. update_trigger and delete_trigger store a change record in the trigger table: an edit or a delete waits for Apply or Delete on a card, a pause or a resume applies at once with a receipt. An applied change can be undone for a day; a delete restores the trigger with its id. Change records are rows in the database, so every frontend shows and answers the same card. The new routes under /api/chats/triggers/changes serve the card and are gated by the triggers feature. Assisted-by: Claude:claude-opus-5-5 * Show the trigger changes a chat asks for as cards An edit lists each changed field before and after with Apply and Dismiss, a delete shows the trigger and its last run with Delete and Keep, and an applied change is a receipt with Undo. The cards sit below the message text like the proposal card, and read their state from the server so a reload shows the same answer. Assisted-by: Claude:claude-opus-5-5 * Add a Test search button to the web tools settings POST /api/chats/admin/web/test existed, but no page called it and the docs showed only curl. The web_search row of Settings > Tools now has a Test search button. It runs one search with the applied settings and shows the provider with its first results, or the error a chat would read. Assisted-by: Claude:claude-opus-5-5 * Describe undo per tool call in the chats docs The docs said the checkpoint line of a turn carries the change set of the turn. A turn checkpoint is taken only by a chat with a VM sandbox, which the service never sets up, so that rev is always 0. What ships is one change set per file tool call, with a files_change card on its tool line. The docs now say so and how to undo a whole turn. Grouping a turn's changes into one change set in the file store is left for later: it touches the file store's revisions and the undo of shared folders. Assisted-by: Claude:claude-opus-5-5 * Mount the skill folders read-only for the built-in bash The built-in pdf skill ships scripts, but the commands saw only the home and the shared folders, so the model could read the scripts and not run them. The built-in bash now mounts the three skill layers read-only, at the names the file tools already use: shared-skills, shared-builtin-skills and shared-addon-skills. A shared folder named skills moves to "shared-skills 2", so the two never collide. A read-only mount with subfolders made every command report its folders as changes the mount refused, because the host backend lists every folder of its work tree. A folder alone is no change: both stores now refuse only files, whiteouts and opaque folders there. The Sandbox add-on mounts only the home; the docs say to copy a script there first. Assisted-by: Claude:claude-opus-5-5 * Find the add-on that runs commands by what it shadows The chats knew the add-on that runs commands from a per-process cache, filled with the Sandbox's own tool name the first time a chat listed its tools. Another add-on that replaces bash did not count, so skills that need commands showed as unavailable. The admin network view read the same cache: after a restart it said no Sandbox serves until someone chatted, each frontend of a cluster could answer differently, and a removed add-on stayed listed. The tool that runs commands is now the one that shadows bash, as the spec asks. sandboxRunTool and the cache are gone. The stand-in of an add-on that cannot work carries the built-in tools it replaces, and the admin view asks the add-on host which add-on serves bash now: the pinned one or the oldest running install, and none when the pinned one is stopped. Assisted-by: Claude:claude-opus-5-5 * Show international host names in punycode in the network checks The card-spoofing defence of the sandbox network asks for host names in their ASCII form, but CheckHost passed Unicode names through. A look-alike name reached the consent card as text that reads like the real site. CheckHost now converts an international name with the IDNA lookup profile and refuses one with no valid ASCII form. The conversion runs before the address checks, so full-width digits that spell a loopback address are refused as loopback. Assisted-by: Claude:claude-opus-5-5 * Keep the seeded Studio and Talk pins until the add-ons are installed The upgrade pins Studio and Talk for existing accounts, but both are add-ons that an administrator installs later. The sweep marked a pin whose add-on is not installed as missing and deleted it after 30 days, so an administrator who installed Studio late found the seeded pins gone. The sweep now records the first time a seeded add-on is present, in the table of seed runs or the pins file. A seeded pin keeps no clock until then, and a mark that an older sweep set is cleared. After the first install the pin follows the retention rule as before. Assisted-by: Claude:claude-opus-5-5 * Fix two small labels of the Skills and Backends pages On the Skills page, the skill of an add-on that is not chosen said "Replaced by <add-on>", the label of a built-in skill that an add-on serves. It now says "Not active: <add-on> provides it", as the add-on skills spec asks. On the Backends page, Clear filters on an empty Installed search left the library filter set, so the list stayed empty. It now clears the library filter too. Assisted-by: Claude:claude-opus-5-5 * Tell the model when no network card can appear Each turn told the model that a blocked host asks the user. No card appears when the ceiling is off or no wider than the chat's preset, in an unattended run or in a sub-agent, so the model waited for an answer that could not come. The line now says that a blocked host is refused and nothing asks the user in those cases, so the model names the host it needs in its answer. Assisted-by: Claude:claude-opus-5-5 * Translate six Backends and preview labels in every locale check, reinstallBackend, deleteBackend and filterLabel of the Backends page and the label and failed text of the file preview showed in English in the eight other locales. They now have translations, and the i18n baseline goes down to match. Also fix a comment of the home composer that still named the removed Canvas chip. Assisted-by: Claude:claude-opus-5-5 * List the admin routes that have no assistant tool, with the reason The admin oversight routes of saved agents, triggers and connectors had no MCP tool and no recorded reason, and the other deliberate skips lived only in code comments. coverage_test.go now keeps them in adminRoutesWithoutTool with a reason each, and a spec checks that no route there is also wrapped by a tool. The contributor checklist names the map. The tool catalog of the assistant docs said it matched the real one but missed most tools added since. It now lists every tool, read-only and mutating, and the actions that stay in the web interface. Assisted-by: Claude:claude-opus-5-5 * Close the doc and registry gaps of the chats and add-ons - /api/agent/* answers 404 with the chats_off reason when the chats are off, as /api/chats/* and /api/workspace/* do. - The add-ons docs get an upgrade section: Studio and Talk are add-ons, nothing is preinstalled, and users need the addons feature, which the upgrade does not grant. It says how an administrator installs them (also from a private library) and grants the feature, and carries a paragraph for the release notes. - The migration page no longer lists fork, edit, the chat settings and the token counters as removed. - Runtime settings document chats_vision_model, chats_transcription_model and the skill keys of chats_tool_providers. - The docs show POST /api/chats/files/folders, GET /api/chats/files/history and the allow route of a connector sender. - The in-tree specs get amendments where the shipped design differs. - A route test of the tagged suite still set the removed Runs field. Assisted-by: Claude:claude-opus-5-5 * Look for the kit empty state in the backends regression spec The Backends page now draws the shared EmptyState of the kit (ui-empty), not the old dk-empty block, so the regression spec of the gallery suite found no empty state after a search with no match. Check the kit's no-results form and its title. Assisted-by: Claude:claude-opus-5-5 * Ignore a features request that outlives its page in the add-on stubs stubAddonRoutes answers /api/features by fetching the real flags and turning add-ons on. A spec that ends while the page polls, like the one that checks that a viewer without the feature never reads the list, left that fetch in flight, and Playwright failed the spec with "route.fetch: Target page, context or browser has been closed" after its checks had passed. Abort the request when the fetch fails because the page is gone. Assisted-by: Claude:claude-opus-5-5 * Bring the gallery-ui case inventory up to date The suite lists the node unit cases of addons.test.mjs and the Playwright specs of addons-gallery.spec.js. The inventory missed the unit cases added since it was written, and one spec was renamed when the legacy switch came in ("hub: the Add-ons tab appears only for an account that holds the feature" is now "hub: under the legacy switch the Add-ons tab appears ..."). The list comes from a local run of both parts against the UI test server. Assisted-by: Claude:claude-opus-5-5 * Read spreadsheets, documents and slides in read_document read_document said "file.xlsx has no text layer; it may be a scan" for workbooks that some Python libraries write. Their workbook gives the sheet as an absolute part name (/xl/worksheets/sheet1.xml) and the cells as inline strings. The text extractor of the harness joined "xl/" with that name, found no sheet, and returned no text; the tool then took any empty result for a scanned PDF. The new package core/services/studio/extract has one entry point, Extract(ctx, name, data), that returns the text and the structure it read, for read_document now and for a file indexer later. Office files go through the files preview reader, which resolves both kinds of part names and reads inline strings: - a workbook comes sheet by sheet, rows as tab-separated lines, up to 2000 rows and 50 columns a sheet, with a note for a cut sheet; - a Word document gives headings, list items, paragraphs and tables; - a presentation gives each slide with its title, text and notes. PDF and EPUB keep the harness extractor. Only a PDF with no text is reported as a probable scan; an empty Office file is reported as empty. The preview reader now keeps a Word table as one block with its rows, and the preview pane draws it as a table. Assisted-by: Claude:claude-opus-5-5 * Add the missing explorer, library and tools translations Eight locales had no explorer.json, library.json or tools.json. The browser asks the server for every namespace of the language it shows, so an Italian page that used the library answered 404 for /locales/it/library.json on every load and fell back to English. The 24 files now exist, translated, with every key of English. The locale gate counted a missing file as missing keys, which the baseline allowed. It now fails on a missing namespace file whatever the baseline says, and the baseline is 24 entries shorter. Assisted-by: Claude:claude-opus-5-5 * Back off while the add-on service is unavailable A page that followed an add-on upgrade read the operation every two seconds, up to 300 times. While the add-on service restarted, each read waited for its timeout and answered 503, so the log of one session had 48 failed dispatches in a row, and the page only showed the upgrade as running. The Sandbox state of a chat's Tools menu read once and, on a 503, said nothing more. The reads of an operation now back off while the service does not answer: 2, 4, 8, then up to 30 seconds. The upgrade entry says that it waits for the add-on service and when it checks again, in Activity and in the details, which also get a Check now button. The Tools menu says that the Sandbox is starting and reads its state again with the same back-off, kept for the page, so opening another chat does not ask again before the wait is over. Assisted-by: Claude:claude-opus-5-5 * Start a chat with its switches and list them before it exists The composer of a new chat could not show the Tools and Skills menus, because the lists and the switches belong to a chat. A chat started from Home therefore always began with every tool and skill on. - POST /api/chats takes switches: the add-ons, MCP servers and connectors, built-in groups and skills that are off, and the opt-in groups that are on. They are checked as the PUT endpoints check them and stored with the new record, so the chat never exists without them; a bad entry refuses the request and creates nothing. The SQL store now writes these lists on create too. - GET /api/chats/draft/tools and /api/chats/draft/addons list what a chat on a model, with a saved agent, could use, every switch in its default state. They refuse what create refuses: an unknown or forbidden model, an agent without the agents feature, another user's agent. Document both in the chats page and regenerate the swagger. Assisted-by: Claude:claude-opus-5-5 * Give Home the composer of the chat page Home drew a smaller message box than the chat page: no files, no agent, no Tools or Skills menus, no paste to attach, no rich model list, and its text was lost on a reload. A chat started there could not be set up before its first message. Move the composer wiring out of the chat page into one place that both pages use: - useChatComposer holds the draft (text and files), the model and the saved agent of a new chat, the chat models, the switches and what attaching and pasting do. - ChatComposer draws the model list (fit, vision, memory bar), the agent picker, the Tools and Skills menus, the attach buttons, dictation, the add-on slot and the vision notice over HomeComposer. - useChatSwitches has a draft mode: before the chat exists it lists from the draft endpoints, a switch changes only the screen, and the choices go with the create request. They outlive a change of model or agent. Home now creates the chat with the model, the agent and the drafted switches, sends the first message with its files and opens the chat. When the message cannot go, the new chat is removed and the text and files stay on Home. Its text is kept in the browser across a reload. Home also gets the shortcuts row, the notices of the chat page and the line that says whether the model is loaded. Its own slash actions, widgets, examples and first-run steps stay. The empty chat page gets the same draft menus. On a phone the chips of both wrap to a second row, and the Tools and agent menus open as a sheet from the bottom, as the skills picker does. Assisted-by: Claude:claude-opus-5-5 * Move add-on settings to the add-on's own page The settings.sections slot drew every add-on's settings at the end of the global Settings page, and the Sandbox and Web sections were unstyled: the add-on UIs used class names the host does not define (btn, btn-primary) and bare inputs, the SDK offered no form primitives, its muted token named a property that does not exist, and the slot root was an inline-flex span that laid a form out in a row. An add-on's settings sections now render on the Settings tab of its details on the Add-ons page (/app/addons?detail=<id>&view=settings), narrowed to that add-on, for administrators only. An add-on page gets a host Settings link to that tab. The Settings page no longer draws the slot: an Add-ons group, shown when an add-on has settings, lists those add-ons with a link each. The old #ad-section-<addon>-<id> anchor redirects to the tab once the apps list is read. Server settings of an add-on's feature stay on the Settings page; the tab shows them under "Server settings for this add-on", with a read-only summary of the Sandbox network and links to ?group=tools#settings-egress and #settings-builtin-tools. The SDK gets a settings kit in sdk.ui (Section, Field, Select, TextInput, Switch, Choice, StatusChip, Callout, List/ListRow, Code, Actions, Table, ExternalLink), drawn with host classes and tokens, and more theme tokens. The Sandbox and Web sections are rebuilt with it: backend rows with a status chip and the raw reason collapsed in a code block, a backend choice with one line per option, and the AppArmor and user-namespace hint (with a docs link) only for the gVisor "exit status 128 ... cannot read client sync file" failure. The Settings page showed "Changed, default 30, Reset" next to a 0 for the audit retention, and the same for the Sandbox network limit, preset and ports: the server stores 0 or an empty value for a setting never set. Those fields now read that value as their default, and the retention input has its unit and bounds. Assisted-by: Claude:claude-opus-5-5 * Adapt the add-on runtime specs to the new navigation Three specs of the esm-ui suite still assumed the old shell: - the Add-ons tab that lists the pages add-ons declare is part of the legacy layout, so the navigation spec found no hub subnav; it now sets the legacy switch on the server's flags for that page; - the sidebar has no "Chat" link any more, so the unsaved-edits spec could not leave the page; it leaves through Home in the primary navigation; - the shell keeps a notification stream open, so the network never goes idle and the spec for a member without the feature waited out its timeout; it waits for the server's switches, the account and the shell instead. Assisted-by: Claude:claude-opus-5-5 * Name the legacy Add-ons tab unit case as the esm unit tests do The esm-ui inventory listed "unit: registry: the Add-ons tab lists page links only when the registry has usable add-ons", but the unit test is named "the legacy Add-ons tab lists ...", so the suite could never match its inventory. Use the name the test prints. Assisted-by: Claude:claude-opus-5-5 * Say on the Collections page when the embedding model is missing At startup the server opened a collection and its first embedding call got 404 from POST /embeddings. The collections embed through this server's own API with collections_embedding_model, the default granite-embedding-107m-multilingual, and that model was not installed. Nothing told the user: uploads and searches just failed. GET /api/collections now also returns embedding_model and embedding_model_installed (a loaded configuration of that name, or a model file of that name). When it is missing, the Collections page says "No embedding model installed", names the model, and gives an administrator a button that opens the Models page with a search for it. Assisted-by: Claude:claude-opus-5-5 * Report modified times and the upload limit in folder listings The Files page could not sort by date or show when a file changed: a folder listing had no time. It also learned the upload limit only after it had sent the whole file. The default server body limit is 15 MB, lower than the 256 MiB a workspace file may have, so a large upload failed late and without a clear reason. Each listed entry now has "modified", the time of the change set of its revision. Both workspace stores fill it in List: the JSON store from its in-memory change sets, the database store with one query per batch of revisions. The listing also has "max_file_size", the smaller of the workspace limit and the server body limit, so a client can refuse a larger file before it sends any byte. The store contract suite checks the new time against the history, and the route tests check both fields. Assisted-by: Claude:claude-opus-5-5 * Add a sortable file list with thumbnails for the Files page The building blocks of the new file browser, used by the page in a later change: - fileBrowser.js: the kind of an entry from its extension, the order (folders first, natural and case-insensitive names, then size, modified time or type), the filter of a folder, dates in words with the full date for a tooltip, the Shift range, the arrow moves of a list and of a grid, type-to-find, names that do not clash, why a name cannot be used, the name of a pasted screenshot and the view settings read back from storage. - FileList: a grid with sortable column headers or a grid of tiles with image thumbnails, a check box per row, a row menu, inline rename with the reason a name is refused, rows that can be dragged onto folders, and the keyboard: arrows, Shift, Space, Enter, F2, Delete, Ctrl+A, Escape, Backspace and typing a name. - MoveDialog: a folder picker that walks the tree and refuses a move into the moving folder itself. - ActionMenu can open at a point with no trigger (a right click), and its trigger can take another icon and class (a split button). - fileSize also writes gigabytes. Assisted-by: Claude:claude-opus-5-5 * Add an upload queue with progress, cancel and retry An upload of the Files page was one fetch per file in a loop: nothing showed while it ran, one failure stopped the rest, and leaving the page lost it. - uploadQueue.js keeps every upload outside React: queued, sending with the bytes sent and a smoothed rate, done, failed with a reason and whether Try again can help, or canceled. Two files go at once. It lives for the whole app, so opening another folder or page does not stop an upload. - filesApi.uploadWithProgress sends the same PUT with XMLHttpRequest, which reports upload progress and can be aborted. The server already streams the body into the store; nothing changes there. - useUploads reads the queue and asks before the tab closes while a file is on its way. - UploadTray shows the queue at the bottom of the page: the total and the time left, each file with its progress, speed and time left, Cancel, Try again with the reason, and a done state that fades out. A live region tells a screen reader when uploads start, fail and end. - ConflictDialog asks what to do when an upload has the name of a file in the folder: replace it, keep both, or skip it. - dropFiles.js reads dropped folders with their paths. Assisted-by: Claude:claude-opus-5-5 * Rebuild the Files page as a file browser The Files page was a list of links: the store's byte order, the same icon for every file, three icon buttons on every row, no progress on an upload, no selection, and on a phone a preview that opened below the whole list. Browsing: a breadcrumb that folds a deep path, sortable columns (name, modified, type, size) with folders first, a list or a grid of thumbnails, both kept per user in the browser, a filter of the open folder with a search of every file on Enter, type icons, human sizes, dates in words with the full date on hover, and Back and Forward between folders. Uploads go through the upload queue and its tray: the Upload button or its folder menu, drops of files and folders anywhere on the page or onto a folder row or crumb, and paste. A clash asks to replace, keep both or skip; a file over the server limit is refused before it is sent. Selection and actions: click, Ctrl and Shift, check boxes, a selection bar (Download, Move to, Move to Trash; at the bottom of the screen on a phone), a row menu and a right click, inline rename and new folder or file, drag to move, the keyboard, and a toast with Undo for a delete, a move and a rename. States: a skeleton on the first load, an error with Try again, empty states that say how files get here, and a no-match state. The preview uses the side pane: docked on a wide screen, a sheet on a phone, with the same actions as the row menu. Trash, History and the Managed by LocalAI group stay as they were. Every new string is in the nine languages. The page uses design tokens only (seven raw font sizes fewer). The specs cover sorting, the filter, the views, selection and bulk actions, the keyboard, the menu, rename checks, drag to move, drops, paste, clashes, the size limit, upload progress, cancel and retry against the real server, and the error states. Assisted-by: Claude:claude-opus-5-5 * Fix two specs that fail on the integration branch The Describe spec of the egress package checked that the no-card line does not contain "asks the user", but that line says "nothing asks the user". It now checks for the sentence that promises a card. The context meter spec gave the chat a 4k context window. The tool schemas of the chat now take 3k of it, so the turn stopped before the model request. The spec uses a 16k window. Assisted-by: Claude:claude-opus-5-5 * Translate the always-allowed hosts note of the chat network row The three egress keys of the always-allowed hosts were only in English, so the locale check failed for every chats namespace. The workspace namespaces now miss two keys fewer, so their baseline goes down too. Assisted-by: Claude:claude-opus-5-5 * Bring the contracts and host-slots case inventories up to date The contracts suite has eight new specs for the check of a tool that shadows a built-in. The host-slots suite runs the unit case of the file preview slot, which the inventory did not list. Assisted-by: Claude:claude-opus-5-5 * Add the design of the file text index The search of the chats' files reads only small UTF-8 text files, so a PDF or an Office file is found by name only. The design extracts the text when a file is written, keeps it in LocalAI's own store keyed by node and hash, checks access at query time and runs one indexer per cluster under an advisory lock. Assisted-by: Claude:claude-opus-5-5 * Give extracted documents their pages, sheets and slides Extract now says where each page, sheet and slide starts in the text, so a search hit can name its page or sheet. A PDF page starts with a "## Page N" line, as sheets and slides already start with a heading. A PDF also stops at a page limit (200 by default) and at the text limit, which it ignored before, and a panic of the reader is an error. The test fixtures move to extracttest, with a real text PDF and a workbook with shared strings, so other packages can build the same documents. Assisted-by: Claude:claude-opus-5-5 * Add a store for the text of workspace documents textindex keeps one entry per workspace node: the text read from a PDF or Office file, the hash it was read from, its pages or sheets and its status (pending, ready, failed, skipped), plus the cursor of the indexer in the workspace journal. The GORM store keeps a lower-cased copy of the text, so SQL and the Go matcher agree on every script, and searches it with FTS5 and the trigram tokenizer on SQLite or a pg_trgm index on PostgreSQL. Without them it scans the column. The JSON store keeps the metadata in memory and each text in its own file, and scans the texts. One contract suite runs on JSON, SQLite and PostgreSQL. Assisted-by: Claude:claude-opus-5-5 * Index the text of new and changed workspace documents The indexer tails the workspace journal from a cursor kept in the text store. A write of a PDF, Word, Excel or PowerPoint file marks it pending (the same hash only follows the path), a delete removes the entry and a move rewrites the paths below a folder. Then it reads the pending files two at a time, with a timeout, a size cap, a page cap and a text cap, and stores ready, failed or skipped. A panic of the reader fails the file, not the pass. The same content is read once. The first pass starts at revision 0, which indexes the files that were there before, in bounded steps that resume from the cursor. Enqueue marks an upload pending at once and Forget drops the entries of nodes a purge erased. In a cluster each pass runs under a new advisory lock, so one frontend reads. Assisted-by: Claude:claude-opus-5-5 * Search and read the text of documents in the chats' files search_files and GET /api/chats/files/search now also search the text index: after the name and text-file hits come lines of PDF, Word, Excel and PowerPoint files, as path (page 3, line 40): text, with the sheet or slide where a document has them. Each entry is shown only when the caller may read its file now and the file still has the content the text came from. Documents keep up to a third of the hits. When files below the path are still pending, the answer says so instead of "Nothing found.", and the REST result has indexing. read_file returns the text of a PDF or Office file with the usual offset and limit, from the index when it matches the file and read on the spot otherwise. An upload and a write of a chat tool queue the file at once, and emptying the Trash drops the entries of the purged files. Assisted-by: Claude:claude-opus-5-5 * Start the file text index with the files and add its settings With files on, the server opens the text store next to the workspace index (the auth database, or a folder under the data path) and starts the indexer. A chat worker searches the index but runs no indexer. chats_index_files (default on) switches the reading off and chats_index_max_text_bytes (2 MiB by default, 64 KiB to 16 MiB) caps the text kept for one document. Both apply at the next pass, and both are rows of Settings > Tools. Assisted-by: Claude:claude-opus-5-5 * Show the page, sheet or slide of a document hit on the Files page Search all files now lists hits inside PDF and Office files with their page, sheet or slide instead of a line number of the extracted text, and says how many documents are still being indexed. The strings are in all nine locales. Assisted-by: Claude:claude-opus-5-5 * Test finding an uploaded PDF by its words through the routes The PDF has a generated name and is uploaded with the REST upload. The search route first reports it as being indexed, then finds it on its page after a pass of the indexer, and so does the search_files tool of a chat. Another user finds nothing. Assisted-by: Claude:claude-opus-5-5 * Document the search inside documents The chats page says what the index reads, where the text is kept, who finds it, which search engine each database uses, the two settings, the backfill and the single-runner mode in a cluster, and how search_files and read_file answer for documents. The API instructions and the PDF skill say the same in short. Assisted-by: Claude:claude-opus-5-5 * Test the indexer on the JSON workspace store A server without a database keeps the workspace and the text in JSON files. The indexer follows the uploads, a folder move and a delete in that journal as it does in the database one. Assisted-by: Claude:claude-opus-5-5 * Read all of a long document with read_file The index keeps the first chats_index_max_text_bytes of a document, so read_file used to stop where the index stopped. It now uses the index only when it holds the whole text, and reads the document again (up to 16 MiB of text) when the index cut it. Assisted-by: Claude:claude-opus-5-5 * Start bash in the user's home The Sandbox's bash and the built-in bash started in /workspace, the folder that holds the mounts, so a model that ran a script by a relative path failed and a file it saved there was lost. A call now starts in /workspace/home, the folder of the user's files: a relative path is the same file the file tools see. The tool descriptions and the commands block of every turn say where a call starts and that only what is saved there is kept. The built-in bash says that the /workspace paths do not exist on the host and that a shared folder is next to the home. Assisted-by: Claude:claude-opus-5-5 * Name files in tool results by the path the model uses write_file, edit_file and search_files named a file of the home by its path in the home ("/scripts/build.py") while the model had written /workspace/home/scripts/build.py, and a model took the two spellings for two files. The results now give the path the model uses, and move_file says where it moved the file to. The cards and the change notes keep the path in the home. Assisted-by: Claude:claude-opus-5-5 * Name the paths a command's commit did not save The result of a bash call ended with "N file(s) changed in your files meanwhile; the sandbox's version was kept as a copy" for every path the commit refused, also for a file written outside the user's files, where no copy is kept. It now has one line a path, with the path as the guest sees it, the reason, and the path of the copy when the store kept one, so the model and the user can act on it. The host returns the paths and the copy to the Sandbox add-on relative to the user's files, as its mounts are. Assisted-by: Claude:claude-opus-5-5 * Show the files a command saved under its step A document that a script wrote with bash reached the user's files, but the chat showed nothing: only the file tools put a file on their step, and the Sandbox committed for no chat, so the chat's file list, the chips under the answer and the preview pane never saw the file. The host now commits a Sandbox call for the chat that ran it (the action resource of the request names the chat). After each bash call, of the Sandbox or of this machine, the chat reads the change sets of the chat that the call made from the shared store and puts the files on the step's card, next to the card that says who served the call (session.SetToolFiles; files with path, change set and op on the card's new files field). The step lists them as chips that open the preview pane, the chat's clients are told that its files changed, and the files reach the file list, so the answer gets its chips and the pane opens on the newest one, as for write_file. Assisted-by: Claude:claude-opus-5-5 * Keep JSON out of the answer of a trigger run The workspace skill told the model to put the widget data shape and a source marker in a trigger prompt. Some proposals then asked the run to print that JSON, and every run ended its chat with a stray line of it. The skill now says source and note live inside the data file and are never the answer, and that a prompt must not ask for JSON. The runner appends one closing instruction to the first message of every trigger run: finish with one short plain sentence. The propose_trigger and update_trigger descriptions say the prompt must stand alone and must not ask for JSON. Connector chats are talks with people and do not get the line. Assisted-by: Claude:claude-sonnet-5-5 * Read the images of a page as base64 for its frame The bridge of a page reads UTF-8 text only, so the UI could not put a png or a jpg of the page's folder into the page. A read of the page file routes with encoding=base64 now returns an image as base64 with its media type. Only the extensions of an image are read this way (png, jpg, gif, webp, avif, bmp, ico, svg); any other file is 415, so this is no way to read an arbitrary binary file. The path checks, the 1 MiB limit, the viewer's rights and, for a shared page, the consent check are the same as for a text read. Assisted-by: Claude:claude-opus-5-5 * Let a page use the files next to it A page runs from a srcdoc with no network, so fetch('news.json'), <script src="app.js">, <link rel="stylesheet" href="style.css">, <img src="logo.png"> and import './x.js' all failed without a word. Models that split a page into files, or load its data with fetch, made pages that showed nothing. Before the frame loads, the UI now reads the files that the page names with relative URLs, with the same calls as the bridge, and puts them in: classic scripts in place (deferred ones at the end of the body), style sheets as <style> with their @import and url(), images as data: URLs, and ES modules as blob: URLs behind an import map, so cycles and import() work. A page with modules gets blob: in script-src; no other part of the policy changes. Inside the frame, fetch and XMLHttpRequest read relative URLs with files.read and answer with a real Response (200, 404 for a missing file, 413, 415); any other URL rejects with "network is off in pages; put the data in a file next to the page". Every path is resolved inside the page's folder: .., %2e%2e, absolute paths, backslashes and protocol-relative URLs are refused, and the query and the fragment are dropped. The files go in only after a shared page is allowed, with the viewer's rights. The files that go in are at most 4 MiB, images 2 MiB; a script or style sheet that does not load shows a notice in the page. A change of one of these files reloads the page, as a change of index.html does. The skills, the docs and the shared pages fixture say and test the same. Assisted-by: Claude:claude-opus-5-5 * Return tool descriptions from an MCP connector test A connection test listed only the tool names. A person who adds a server cannot tell from a name such as "query" what the tool does or whether to keep the connector on. The test response now also has tool_details: each tool with the description its server gives, in the order of tools. tools is unchanged, so existing clients keep working. Assisted-by: Claude:claude-opus-5-5 * Add jq, ripgrep, sqlite3 and archive tools to the Sandbox guest The guest user cannot install packages and apt is blocked by the proxy, so any tool the model needs has to be in the image. Add jq, ripgrep, sqlite3, zip, unzip, file and tree to the final stage. A new check-tools.sh runs in the existing build step and fails the build when one of these programs, Poppler, git or curl is missing or does not start. The list is one constant, GuestTools, in the shared sandbox package. The Commands block of the prompt of an isolated chat names it. The tool description (capped at 400 characters, already full) and the turn note (cut at 400 characters) have no room for it. Tests keep the Dockerfile, the check script and the Sandbox documentation in line with the constant. Assisted-by: Claude:claude-sonnet-5-5 * Keep the helper files of a chat in its work folder A chat asked for a Word document left a 19 KB build script at the top of the user's home, next to the user's own files. Chats like it leave scripts, intermediate data, logs and drafts at the top level or in folders such as scripts/, and nothing tells them apart from what the user asked for. Each chat now has a work folder, Chats/<chat folder>/work/ in the user's home, next to its attachments. The chat takes the name of its folder at its first turn (checked against the folders and the other recent chats of the user, so two chats with one title get two names) and makes the folder only when it is used: before a bash command runs, or when a file is saved there. The prompt of every turn names the folder and says what goes in it: scripts, helper code, intermediate data, logs and drafts go there; the deliverable goes where the user said or at the top of the home; no new top-level folders for scripts; a run-once command needs no file. A sub-agent is told its parent's folder. The descriptions of bash (the Sandbox's, within its 400 characters, and the built-in one) and of write_file end with one short hint, and the built-in office, apps and workspace skills say the same in one line. The steering is soft. When write_file, move_file or a command leaves a helper file outside the work folder, the result gets one line that suggests moving it, at most once a turn of a chat. One function (workspace/workfiles) decides the kind of a file: working for anything in a chat's work folder, or for a script, a log or a temporary file by its extension or a folder named scripts, tmp or scratch; deliverable for everything else, and always for pages, the LocalAI folder, the shared folders and the files the user attached. The chat's file list (kind), the files of a command's card (files[].kind) and the card of a file tool (file_kind) carry it, so the interface only renders it. The runtime setting chats_work_folder (on by default) turns the steering off. Assisted-by: Claude:claude-opus-5-5 * Stop telling the model that the Sandbox has npm The guest image has no Node, so the network hints named a tool that does not exist. Assisted-by: Claude:claude-sonnet-5-5 * Update the Sandbox description test after dropping npm Assisted-by: Claude:claude-sonnet-5-5 * Open the links of a page after the person confirms them A page runs in a frame without allow-popups or top navigation, so a click on <a href="https://..."> did nothing, and a link to #id loaded LocalAI's own address in the frame, since a srcdoc resolves it against the parent. A news page had only dead links. The bridge of the frame now takes the clicks on links (also with a modifier, the middle button or Enter) and window.open(), and sends link.open(url) to the host. The host lets through only an http or https address without a user name or password, at most 2048 characters; javascript:, data:, file:, blob:, //host and links to other files of the page are refused with a message in the console. A bar above the page shows the site in large type, the whole address, and Open and Cancel. A host in punycode also says how it reads, since it may imitate another site. Open is a click in the app, so the new tab is not blocked and opens with noopener and noreferrer. A page asks at most 5 links at once, then one every 3 seconds. A link to #id scrolls the page. The sandbox, the policy of the frame and the consent of shared pages do not change; a shared page asks the same way. Assisted-by: Claude:claude-opus-5-5 * Let the items of a Home widget link A list widget showed each item as plain text, so a digest of articles or a list of pages could not take the person anywhere. An item now has an optional url. An http or https address without a user name opens in a new tab with noopener and noreferrer, and its tooltip names the site; a path of the app (/app/pages, /app/chats, /app/files and the other screens a person reads) opens inside LocalAI. The label is the link, with a focus ring, an underline on hover and focus, and an icon for an address outside the app. The widget-data schema checks the url, and the server adds an error in words that names the item and the rule, so a trigger run that writes a bad link learns why. The skill and the docs show the field. Assisted-by: Claude:claude-opus-5-5 * Attach any file in a chat with one button The chat had three attach buttons (image, audio, file) that all add to the same list, and the file button's dialog filtered on */*, which some file dialogs read as a type filter that greys out archives such as .zip. A chat now shows one Attach file button whose dialog has no type filter. Composers without anyFile keep the three buttons. Assisted-by: Claude:claude-opus-5-5 * Accept both Node behaviours for a broken punycode label in the link test Node 22 refuses the address, Node 24 parses it. Both mean the page never opens it unwarned. Assisted-by: Claude:claude-sonnet-5-5 * Show a chat's working files apart from what it made The Files pane, the steps and the preview treated a helper script like the document the user asked for: a 19 KB build script was listed next to the document, got a chip under the answer, and the preview opened on whichever file was newest. The interface now renders the kind the server puts on each file. The pane lists what the chat made first and folds the working files into one quiet heading under the rest, "Working files · 2", with a line that says what they are; opening it shows their rows, which preview like any file. A working file opened from a step unfolds the group. The preview that opens by itself after a turn picks the newest file that is not a working file, and the chips under the answer name only those files. A step that saved working files lists them as one line, "Also saved 2 working files", that opens to their chips, and the file of a write_file step that is a working file is a quieter chip. The files the user attached are unchanged. Assisted-by: Claude:claude-opus-5-5 * Add a search field and a segmented filter to the UI kit The Agents views each drew their own toolbar: a search box capped at 320 px on the agents list, round chip filters on the triggers history, and a bare checkbox on the tasks page. Add-ons already had the look the product wants (a search that fills the row and a segmented tray), but only as add-on classes. SearchField and Segmented are that toolbar as two small primitives with ui-* classes and token values only. Segmented buttons carry their id in data-filter, so specs can press them. The README shows how to use them. Assisted-by: Claude:claude-opus-5-5 * Draw an agent task run as a history row with its output The jobs of an agent task were table rows that opened to bare <pre> blocks: no copy, parameters printed as k=v, the traces in a raw <details>, and "by manual" in the server's words. RunItem is one run of a history list: a toggle, the state chip, what it was and what happened, when and for how long, small actions, and the details that open under it. OutputBlock draws an answer or an error in mono with a Copy button. JobRow and JobDetail use them for a job: the values it ran with as labelled chips, the answer or error, the steps, the webhook outcome and a link to the chat that holds the run. Times show as words with the full date as a tooltip. The agent task helpers gain the pure rules the page needs: the jobs of a task, the run filters (waiting and running count as in progress), the on/off filter and search of the task list, the field errors of a save, how a job started, and the API call that runs a task by name. Assisted-by: Claude:claude-opus-5-5 * Create, edit and run agent tasks in a side sheet The task form opened as bare fields above the list, with a checkbox for On and a Save button that was only disabled when a field was missing. Run with parameters was a second inline form in the same place. TaskSheet is a side sheet with three labelled groups (about, who runs it, what it does), help under each field, the parameters the prompt asks for as chips while you type, and an On switch. Save checks the required fields, says next to each one what is missing, and moves the focus to the first. RunSheet asks for each value the prompt names and shows the prompt as the task will receive it. Assisted-by: Claude:claude-opus-5-5 * Show the details of an agent task in a pane with tabs A task had no details view: its prompt, its runs and the way to run it over the API were only in the lede of the page. TaskPane is the kit's side pane with three tabs. Overview shows who runs the task, the values it asks for, the prompt and a curl call that runs it by name, with Copy. Runs lists its jobs. Settings holds the On switch, the webhook count and Delete. Run and Edit stay in the footer. The tabs move with the arrow keys, Home and End. Assisted-by: Claude:claude-opus-5-5 * Rebuild the agent tasks page from the shared list pieces The tasks page looked unstyled next to Add-ons and the agents list: two raw tables under uppercase eyebrows, a bare checkbox for All users, forms that opened above the list, three text buttons on every row, and on a phone a table that broke names one syllable per line. The page now uses the same pieces as the polished pages. The header has one line and New task. A toolbar holds the search, an All/On/Off filter and, for an administrator, an All users switch. Each task is an agent row: a tile, the name, two lines of description, who runs it and what it asks for, the last run's state and time with the run strip, an On switch, Run and a menu. A task with no parameters runs at once. The name opens the details pane; New task, Edit and Run open the sheets. The run history is grouped by day, with a state filter, a task filter and Show more. A job is deleted after an in-page question. With no tasks the page explains what one is, with an example; a failed load shows the error with Try again. On a phone the controls grow to 44 px and the filters scroll sideways. New strings, and the task strings that existed only in English, are in all nine locales; the strings the page no longer uses are gone. A new stubbed spec covers the list, filters, sheets, pane and history, and the real-server spec follows the new controls. The docs describe the page. Assisted-by: Claude:claude-opus-5-5 * Rebuild the MCP connectors page on the Add-ons patterns The page read as a form: every connector was a row of raw controls (a checkbox, a Test button, red Delete text), the add form pushed the list down, a test result vanished on reload, and the frame was narrower than the pages next to it in the sidebar. The page now uses the same blocks as Add-ons and Files: - Full page width and the shared PageHeader with a one-sentence purpose and "Add connector". - The Add-ons toolbar (search and a state filter with counts) from four connectors, and the Add-ons rows: a monogram tile, a state chip (Connected, Needs sign-in, Can't connect, Not tested, Off), one line with the tools or the reason, a meta line, the kit Switch and a menu. - Details in the shared SidePane: the switch, the last test, the tools with their descriptions and a search, the connection with the query of the address hidden, and what is saved. - Add and edit in the shared SideSheet: kind first, grouped fields with inline checks, masked secrets with show and hide, then "Save and test", the result with the tools, and Turn on. Escape and Cancel ask before they drop typed changes. - An empty state that says what a connector is for. The last test of each connector is kept in this browser with a hash of what it tested, never the address or a secret, so the page shows the last known state after a reload. Strings are in all nine locales. Assisted-by: Claude:claude-opus-5-5 * Document the MCP connectors page and its states The MCP servers section now describes the add sheet (kind, checks, masked secrets, Save and test, Turn on), the states a connector shows, where the last test is kept, the search and the filter, and the details pane. The test example shows tool_details. Assisted-by: Claude:claude-opus-5-5 * Render a preloaded page without its loading placeholder A page that was preloaded still showed the route's loading placeholder when it opened: React.lazy suspends once on its first render even when the chunk is already in memory, and React keeps a fallback on screen for at least 300 ms once it shows one. Opening a chat from Home showed a blank page for that time. The page loader now remembers the module once it has loaded and hands it to React.lazy as a thenable that resolves at once, so a preloaded page renders in the same commit. A preload of a segment now warms every page registered under it: /app/chats is the chat page, its connectors and its conversion page, and before only the last one registered was preloaded. Assisted-by: Claude:claude-opus-5-5 * Move the composer from Home into the new chat in one gesture Sending the first message on Home was a hard cut. The page stood still while the chat was created, then showed a blank placeholder, the empty "new chat" screen with its composer in the middle, an empty thread with the composer at the bottom, and only then the message, while the sidebar folded and moved the whole column. Focus was lost. Now Send fades the rest of Home and keeps the composer in place while the message goes. When the send has answered, a view transition moves the composer to the dock of the chat, lifts the text out of it into the message, moves the attached files with it and folds the sidebar in place; the model's indicator comes in after the message has landed and the empty composer has the focus. The chat page draws the message from what Home already knows (a small hand-off store: the chat record, the text and the files, with the local copy of an image for its thumbnail) and keeps the same row when the server's transcript replaces it, so nothing is drawn twice or animates again. What is sent to the server does not change. A browser without view transitions gets a FLIP slide of the composer and the message (Web Animations, transform and opacity only). Less motion gets a short cross-fade, or an instant switch. A send that fails brings Home back, shakes the composer and keeps the draft; a second Enter while one send is in flight sends nothing. The draft is cleared when Home closes, so the box never shows its placeholder on the way. Focus mode folds the sidebar in a layout effect, so the fold happens in the frame the conversation appears. One motion token is added, --dur-glide (360 ms), for a part that travels to its place on the next page. Assisted-by: Claude:claude-opus-5-5 * Draw the triggers as rows and their history like the task runs The triggers page used a table with a switch column next to the agents card list, folded its list behind Hide, filtered the history with round chips, and on a phone cut each history row at the right edge. Each trigger is now an agent row: a clock or folder tile, the name, the schedule in the server's words, the agent or model and when it runs next, the last run's state and time with the run strip, an On switch, Run now and a menu. The week sentence sits under the Run history title. The history uses the segmented filter and the same run rows and day headings as the agent tasks, with Show more after 30 runs; an open run shows its reason with Copy. On a trigger page the prompt comes before the runs, in the body face, the On switch is the kit switch, and the runs stop at 10 with Show more. Assisted-by: Claude:claude-opus-5-5 * Use the shared search and the full width on the agents list The agents list stopped its search at 320 px and used the content width, while the Triggers and Tasks views next to it use the full width, so the title and the view tabs moved when you switched views. The list now uses SearchField, which fills the toolbar row as on Add-ons, and the full page width like its sibling views. Assisted-by: Claude:claude-opus-5-5 * Polish the agent task sheets and pane after a visual review Screenshots of every state showed a few rough edges: - The name placeholder looked like a filled-in value next to the error "Give the task a name". The name field has no placeholder now, and the prompt placeholder starts with "For example:". - A task run by an agent showed the agent id until the agents were read. It says "Agent" until then. - A run opened in the details pane also opened in the history, and the two copies shared an element id. The pane keeps its own open runs and ids. - A failed run in the pane drew its meta line in red; the title carries the error colour there now. - A group title that is not a legend had a double gap under it, and the kebab menus were under 44 px on a phone. Assisted-by: Claude:claude-opus-5-5 * Cover the code files of a shared page in its consent The consent to a shared page was kept against the hash of index.html and page.json. The UI also puts the scripts, style sheets and modules next to the page into it before it runs, so the author could change app.js after a viewer allowed the page and run new code without asking again. The server now finds the same files as the UI's inliner: every script src, stylesheet link, relative CSS @import and module import of the folder, followed through the module graph. Their content hashes go into the hash of the page, and a file the page names that is missing counts too, so adding it later also asks again. Data files and images stay out, so saving data does not ask again. The files are listed as code_files. What a file names is kept per content hash in a small per-frontend cache, which only saves reads. Two more checks close the gaps around it. A frame of an older version that reads a changed file of code gets 409, so a script edited while the page is open never runs under the old consent. The UI puts no text file into a shared page that is not in code_files, so if the server and the UI ever disagree, the file fails to load instead of running outside the consent. The trust sheet names the files of code when only one of them changed. Assisted-by: Claude:claude-opus-5-5 * Let a page be a tile on Home in its compact mode Home widgets could show a list, a number or the last chats, but not one of the pages a person or the assistant built. A page can now be a tile. A page opts in with a widget block in page.json: a size (small, medium or large, the height of the tile) and an optional short title. A widget of type page in LocalAI/home.json names the page, and a folder id for a page of a shared folder. The server takes at most three page widgets and says why in words, and refuses a name that is not a page name. The tile runs the page in the same sandbox and bridge as the full view. The page reads localai.page.mode ("widget" in a tile, "page" otherwise) and localai.page.size, and <html> carries data-mode and data-size, so the page decides how its tile looks instead of being shrunk. The tile reserves the height of its size, builds its frame only once it scrolls into view, tells the page through a new visibility event when it goes out of sight or the tab is hidden, takes the frame down far off screen, and keeps at most three frames alive. It has a title bar with Open page and a refresh button, a skeleton while it loads, Retry on an error, and the link question of the full view. A page that is gone or has no widget block shows a calm line with a link to Customise Home; a shared page that the viewer has not allowed shows Allow this page, which opens the same trust sheet. Customise Home lists the pages with a widget block, with the size of their tile. The task board starter has a compact view, the skills tell the assistant how to put a page on Home, and the docs describe it. Assisted-by: Claude:claude-opus-5-5 * Give the Files upload its own size limit The Files page upload was capped by the general --upload-limit (15 MB by default), although a workspace file can be 256 MiB. Make 0 the default of --upload-limit, meaning not set. When it is not set, the general routes keep the 15 MB limit and the Files upload route skips it, so the workspace file size is its limit. When the operator sets a value, it applies to the Files upload too, lower or higher (a file never exceeds the workspace size). The max_file_size of a folder listing reports the limit in force. Assisted-by: Claude:claude-sonnet-5-5 * Refuse writes of code files from a page A page loads its own scripts, modules and styles, so a page that could overwrite app.js or style.css would run new code on its next load. The page bridge now writes data files only. The server refuses a write whose final segment has a code or document extension (js, mjs, cjs, css, html, htm, svg, xml, xhtml, wasm), is named index, or is page.json, with 403 and a clear message. The check applies to the page routes only; the Files page and the assistant's file tools keep full write access. The judgement lives in pages.IsCodeFile and is mirrored by isCodeFile in the web host, which refuses early with the error code forbidden. Both run one table of cases, covering case, double extensions, trailing dots and spaces, percent-encoding, backslashes and lookalike dots. Assisted-by: Claude:claude-sonnet-5-5 * Make the trigger tools a built-in group a chat can switch off propose_trigger, list_triggers, update_trigger and delete_trigger were added to every chat a person starts, with no switch. They are now four entries of the built-in tool registry in one group, triggers, so the Tools menu, the tools panel, the draft list and the builtins switch endpoint handle them like the web group. The group is on by default. A trigger or connector run and a sub-agent still never get them. The panel lists the group as not usable, with a reason, for those chats, for a user without the triggers feature, and not at all when the server runs no triggers. The group has no runtime setting, so the registry gets an Unlisted flag and Settings > Tools skips it. The panel now builds every grouped tool to learn whether it is usable, not only the opt-in ones. Assisted-by: Claude:claude-sonnet-5-5 * Show the Triggers group in the chat Tools menu The menu rows come from the groups the server lists, so the row needs only its label in the nine locales. A Playwright spec covers the row, its switch request, Select all and Disable all, and the disabled row with its reason. Assisted-by: Claude:claude-sonnet-5-5 * Talk about settings, not the files behind them The settings cards, the shortcuts row, Home and the change notes in a chat named the files that hold a person's settings: "Defined in LocalAI/shortcuts.json", "Layout is in LocalAI/home.json", "Saved LocalAI/persona.json". A person changes their shortcuts or their Home, not a file, and the Files page still shows the files to whoever looks. The "Stored in", "Defined in" and "Layout is in" lines are gone; the history line keeps Versions, Revert and the Invalid badge. The notes say what changed ("Saved your shortcuts.", "Created the page Task board."), the errors name the setting or the widget instead of a path, and the Source view says "The settings as text" instead of JSON and schema. A failed save now says what happened in one plain sentence and keeps the server's own words under a collapsed Details. The drafts that "Edit with the assistant" puts in the composer no longer carry the path; the built-in workspace skill already maps shortcuts and Home to their files. All nine locales are translated; only the changed keys moved, and four keys that nothing reads any more are removed. Assisted-by: Claude:claude-opus-5-5 * Drop internal names from the skills, add-ons and tasks copy Other pages used words that only make sense to someone who knows how LocalAI is built: - Skills said "a folder with a SKILL.md" and "the file workspace". It now says what a skill is for, and that skills need Files. - The add-on notes blamed "the owner", the internal name of the server that runs add-ons. They now say what happened to the action, or that the add-on service is not available; a private library names a credential "an administrator configured on this server". - The Tasks header listed an API route, and the first-run wizard spoke of a "host bash fallback". The memory line of Settings no longer mentions "the resources endpoint". All locales that have these keys are translated. The skill delete dialog, the unused skills intro and the Tasks header exist only in English and change only there. Assisted-by: Claude:claude-opus-5-5 * Fail the unit tests when UI copy names implementation details A new test reads every string of every locale and fails when one names a settings or data file (.json, .yaml, LocalAI/), an API route or HTTP method, a setting key (chats_*), an environment variable, a format word (JSON, YAML, schema) or a path shown as code. Plain words such as "schema" and "endpoint" are checked in English only, since other languages use them for other things. Operator pages may still name the model configuration, the metrics route or a startup setting that the docs name. Those keys are listed in ui-copy-allowlist.json, each with the audience and the reason. The list only shrinks: an entry whose string no longer needs it, or whose key is gone, fails too. Assisted-by: Claude:claude-opus-5-5 * Name the triggers group in the chat tools routes The routes that list and switch the built-in groups of a chat said only "a name such as web". They now name all three: web, triggers and saved_agents, with the tools of each. Assisted-by: Claude:claude-opus-5-5 * Make the built-in skills answer in plain words The skills told the model to report file names, folders and formats after a change. Add one rule near the top of localai-workspace and localai-apps: do the work with files, but say what changed, and name a file only when the user did or asks. Replace the example replies that carried paths, keep every path that says where to write, and pin the rule with a test. Assisted-by: Claude:claude-sonnet-5-5 * Apply the Go 1.26 modernizers to the latest Studio code go fix suggested slices.Contains, strings.SplitSeq and range-over-int on lines added since the first pass. ParseSearchProviders had the same hand-written membership loop, so it uses slices.Contains too. Only lines that the branch added since then are changed. Assisted-by: Claude:claude-opus-5-5 * Share the blob wait and the counting of the two workspace purges The JSON and GORM stores each had the 2-minute default for the blob grace, the same switch that reads the BlobGrace option, and the same loop that counts files, folders and bytes per revision. They now call blob.Grace and PurgeResult.Add, both with specs. The SQL and the in-memory walk that pick the nodes stay in each store. Assisted-by: Claude:claude-opus-5-5 * Read the upper layer of a sandbox in one place for both stores The JSON and GORM workspace stores had the same readUpper, place, joinRel, depth and sortedKeys, with the same upperSet types. They move to workspace/internal/upperset, which has its own specs. The commit algorithm that applies the set to the nodes stays in each store; it needs a storage interface first. Assisted-by: Claude:claude-opus-5-5 * Let chat attachments use the 1 GiB message limit The docs promise 1 GiB for a message with attachments, but the general body limit (15 MB by default) ran first, because only the Files upload skipped it. Skip it also for a multipart POST to the messages route when the operator set no --upload-limit. A JSON message keeps the general limit, and an explicit --upload-limit still wins. A message that goes over the 1 GiB limit now gets 413, not 400. Assisted-by: Claude:claude-opus-5-5 * Name the read_file tool in the skills index header A weaker model read a built-in skill with the wrong path and then tried the Sandbox, which cannot see skill folders. The header now says which tool reads a skill and that the shown path is the one to use, before it mentions where the user's own skills go. Assisted-by: Claude:claude-sonnet-5-5 * Tell the model that skill folders are not in the Sandbox The Commands block of an isolated chat now says that skills are read with read_file. The Sandbox tool description and the per-turn note are at their size caps, so the sentence goes in the system prompt. Assisted-by: Claude:claude-sonnet-5-5 * Point read_document at read_file for text files read_document reads PDF, Word, Excel, PowerPoint and EPUB files. For a Markdown or text file it said only that the file is not a document it can read. The error now names read_file. Assisted-by: Claude:claude-sonnet-5-5 * Tidy the text index stores and the indexer pool The two stores wrote the default limits 20 and 100 as numbers in five places; they are now named constants, so the stores cannot drift. The JSON store checks a missing file with errors.Is. The indexer starts its readers with wg.Go and keeps the first error inline, which drops the cmpErr helper. Assisted-by: Claude:claude-opus-5-5 * Refuse the server's public address in the Sandbox proxy web_fetch refuses the addresses of the server: its interfaces, loopback and the host of the external base URL. The Sandbox proxy runs in the add-on process and listed only the interfaces of its own machine. Under "All public sites", an always-allowed host or an Allow card, a command could reach the LocalAI API through the public name or address of the server, a NAT address or a proxy in front of it. The host now keeps one list for both. The application builds it once, gives it to web_fetch, and sends it as CIDR strings in the new self field of every egress.check allow. The proxy keeps the last list it got and refuses it on each dial, besides its own interfaces, also for an allow it cached. safehttp gets the codec and a join of deny lists, so both sides read the list the same way. Assisted-by: Claude:claude-opus-5-5 * Check the TLS server name on every port of the Sandbox proxy The proxy checked the ClientHello only on CONNECT to port 443. An always-allowed host pinned to another port, or any port added to chats_egress_ports, was a raw tunnel: a command could open TLS to another tenant of the same front through it. The docs said the server name must still match. On other ports the proxy still waits for nothing, since the protocol may be anything and the server may speak first. But when the first bytes from the guest are a TLS handshake record, it reads the whole ClientHello and passes it on only when the server name is the CONNECT host; otherwise it closes the tunnel and audits the reason sni. Assisted-by: Claude:claude-opus-5-5 * Say what sent each notification The inbox only knew whether a notification was about a run or needed attention, so a trigger that was switched off and a connector that was switched off looked the same. Each notification now carries a source: trigger, connector or system (the conversion of old data). The list endpoint returns it, and items stored before have none. Assisted-by: Claude:claude-opus-5-5 * Say where a skill is when a read of it fails When read_file, read_document or list_files fail with not found for a path that names a skill of the chat (a wrong skills folder, or just <name>/SKILL.md), the error now gives the skill's real path and says to read it with read_file. The skills come from the chat's own resolver, so every layer, add-on and per-user switch is respected, and nothing is looked up unless a read fails. Assisted-by: Claude:claude-sonnet-5-5 * Keep the last valid Sandbox network lists when a value is bad The settings endpoint refuses a bad chats_egress_allowed_hosts with the reason, and the runtime settings file keeps the value in force. A bad value that reached the configuration another way was dropped with no log line: the always-allowed hosts and the extra registries became empty, and a bad port list fell back to 80 and 443, which can widen a narrower list. Each of the three lists now keeps the last value that parsed, logs a bad value once, and uses the default only when the process never saw a valid one. The parse runs again only when the value changes, not on every egress.check. Assisted-by: Claude:claude-opus-5-5 * Use the keyless chain of the built-in search in the Web add-on The add-on replaces the built-in web_search, but its keyless provider was DuckDuckGo alone. DuckDuckGo often answers data-center addresses with a bot check, so installing the add-on without a key made search worse than the built-in, which falls back to the Brave Search and Bing result pages. The keyless entry of the add-on's order (saved name duckduckgo) is now the built-in chain from pkg/web/search: DuckDuckGo, then the Brave Search page, then the Bing page, with the same cool-down. The status tool reports the providers in their cool-down, and the Settings tab shows them with the reason. Test the providers gives one row for each page of the chain. Assisted-by: Claude:claude-opus-5-5 * Fix the GitHub hosts in the Q1 row of the egress design The Q1 row still listed raw.githubusercontent.com on the registries list. The owner decision below it and registries.json exclude it, since it serves any user content, and add release-assets.githubusercontent.com. The row now says the same. Assisted-by: Claude:claude-opus-5-5 * Keep one private-address rule for the network cards A card asked whether its host is a private IP literal by calling egress.Admits with a made-up ceiling, which re-derived the rule that egress already has. egress.PrivateLiteral is now exported (with a spec) and the card uses it. The two String methods of an allowed host write host:port with net.JoinHostPort, which brackets an IPv6 literal the same way the hand-written code did. Assisted-by: Claude:claude-opus-5-5 * Read a unique constraint violation in one place The jobs and triggers stores had the same isUniqueViolation, and the saved agents store had the same test inline before it looked for the name index. They now call dberr.IsUniqueViolation, which has specs for the SQLite and PostgreSQL error texts. The workspace store keeps its own stricter match. Assisted-by: Claude:claude-opus-5-5 * Ask before a Home widget link opens another site A list widget turned an http or https item into a plain link to a new tab. Its data file is written by trigger runs and the assistant, which read web text, so a planted link opened with one click, and only a tooltip showed the host. A link of a page asks first. An external item link now opens the confirmation bar of page links in the widget: the site, the full address, the punycode warning, and Open or Cancel. A link inside the app still navigates directly. The bar takes a label, so the widget says that it wants to open the link. Assisted-by: Claude:claude-opus-5-5 * Prune the audit of the commands run on this machine The audit of the built-in bash had no retention: the database table grew without bound in a cluster, with commands of up to 4 KiB each. The Sandbox network audit had chats_egress_audit_days and a daily prune under an advisory lock. Both audits record what the commands of the chats did, so the host command audit now uses the same retention and is pruned in the same daily pass: one frontend of a cluster, under the same lock; a standalone process alone. The GORM and file stores get Prune. No new setting. Assisted-by: Claude:claude-opus-5-5 * Save the Studio JSON stores with atomicfile.WriteJSON Seven stores (agents, connectors, jobs, MCP connections, notifications, push subscriptions and triggers) encoded their state and then called atomicfile.Write with the same four lines. atomicfile.WriteJSON does both and has specs. The load side stays in each store, since the error messages differ. Assisted-by: Claude:claude-opus-5-5 * Build the result page walkers of the keyless search on one walk hasChallenge and firstByClass each had their own recursive walk that stopped at the first match. They are now predicates passed to firstElement, which already did that walk. The order of the walk and the element tested first are the same. Assisted-by: Claude:claude-opus-5-5 * Expire network cards that a dead frontend left pending Pending network cards expired only on the turn-end path of the process that ran the turn. When that frontend died, its cards stayed pending in the chat record, counted toward the cap of three, and gave the next turn's connections cards-full blocks. The UI showed them as waiting. A card now names the turn that asked it. One rule, shared with the command card in a later change, says whether a card still has a waiter: its turn holds the chat and its wait has not run out. A card that fails it reads as expired, takes no place under the cap, and is expired in the record when the next card is asked, when the next turn starts, or when the reaper ends the dead turn. Assisted-by: Claude:claude-opus-5-5 * Reuse ChooseProvider for the expected add-on of a tool shadowers.expected repeated ChooseProvider line for line: the pinned add-on when it is installed, else the oldest install. It now calls it. commandFiles kept an order field and sorted the nodes of a map by it; it now appends each node to a list when it first sees it, which gives the same order without the sort. Assisted-by: Claude:claude-opus-5-5 * Do not print a nil error when a key or entry is missing Three checks joined "the read failed" and "the read found nothing" in one branch, so the second case wrote "<nil>" into the message: the VAPID keys of push notifications, the sealing key of connectors and a rendered catalog entry. Each case now has its own message. The message of a failed read does not change. Assisted-by: Claude:claude-opus-5-5 * Check the close errors and rename the tool variables golangci-lint failed on the new studio code. Five deferred Close calls and one Fprintf into a hash dropped their error, and two locals named t made forbidigo read `t.Run` as a testing call. Discard the errors explicitly and name the locals tool. Assisted-by: Claude:claude-opus-5-5 * Make the file text index forget purged files and say its limits Three gaps of the text index of documents: - A purge dropped the index entries of what it erased in a goroutine that only logged its errors, and the purge had already erased the journal rows. A crash or a store error left the text in the database for good. Each pass of the indexer now also checks a batch of entries (200 every 30 s, round all of them) and deletes those whose node is gone, so what a failed Forget leaves is removed. - A document search stopped at 20000 files or 64 MiB of text without a word, and the "still being indexed" count stopped at 50. The search now sets Limited (search_files says "results limited", the REST answer has "limited", the Files page shows a note), and the count is the number of pending entries below the path. - read_file extracted a long document again for every page, up to 50 MiB and 60 s each time. The first read keeps the whole text in the index store by content hash (a new documents table, or files in the JSON store), and the next pages come from it. The text goes away with the last entry of that content, on delete and on purge. Search reads the text without its folded copy, which only the database needs. Assisted-by: Claude:claude-opus-5-5 * Stop the PDF reader when the extraction gives up The PDF and EPUB reader runs in a goroutine, and the call returns after 60 seconds when it is slow. The goroutine went on reading every page, since its context ended only with the caller's. It now gets a context that ends when the call returns, so it stops at the next page. What the call returns does not change. Assisted-by: Claude:claude-opus-5-5 * Compare the code files of a shared page as the store does The guard that refuses new code to a frame of an older version compared the name it reads with the names of the page's code exactly. The workspace folds case, so an old frame that read Index.html or JS/App.js got the new code with no 409. Compare names in the store's form (workspace.FoldName), as the write check already does, and test the case variants. Assisted-by: Claude:claude-opus-5-5 * Expire the command card of a turn that is over The card of the built-in bash was cleared only by a defer in the call that asked. When that frontend died, the card stayed in the chat record. The chat hid it only while no turn ran, so the next turn showed it again, and an answer to it reached no one but still got 204. The card now names its turn and when it was asked, and follows the same rule as the network cards: it waits only while that turn holds the chat and its five minutes have not run out. GET /api/chats/{id} shows only such a card. The next turn and the reaper clear a card left behind, and the waiting call clears only its own card, so it never removes a newer one. An answer to a card with nobody waiting is 410 consent_expired, and the chat shows the card as not answered, with no buttons. The store gets UpdateHostConsent, an atomic change of the card. Assisted-by: Claude:claude-opus-5-5 * Answer a trigger change once across frontends ApplyChange, UndoChange and DismissChange checked the outcome of the change record under a mutex of one process, then wrote the store. Two frontends could both see a pending change and both apply it, or both undo it. Add Store.SetOutcome, a conditional update of the outcome (only from the outcome it names), to the GORM and JSON stores and to the store suite. The three answers now claim the record with it before they write, give the outcome back when the change itself fails, and answer ErrProposalAnswered (409) when another request won. A test runs the two answers on two frontends at the moment between the read and the write, for an edit, its undo, and a delete. Assisted-by: Claude:claude-opus-5-5 * Add the shadowers spec to the tables-tools inventory The tool adapter gained a spec that lists the add-ons that replace a built-in in any state. The tables-tools suite runs every spec of that package, so verify.sh found one case more than the inventory holds. Assisted-by: Claude:claude-opus-5-5 * Offer Clear filters when the backend Catalog shows nothing The Installed view had a Clear filters button in its no-results state; the Catalog view did not, although the spec asks for it in both. Move the button to one place and show it in the Catalog when a search, a type filter or a library filter leaves no row. Add Playwright specs for both views, which the earlier Installed fix lacked. Assisted-by: Claude:claude-opus-5-5 * Give the chat settings without an assistant tool their reason chats_max_iterations, chats_work_folder, chats_index_files and chats_index_max_text_bytes had no MCP tool and no entry in adminRoutesWithoutTool, and the older chats_saved_agents, chats_suggestions, chats_vision_model and chats_transcription_model were unlisted too. Add each with the reason it stays a person's choice in Settings: they apply to every user's chats, and the step limit is one a model must not raise for itself. Assisted-by: Claude:claude-opus-5-5 * Remove the stray allowed-hosts keys from the chats inbox locales allowedHosts, allowedHostsOff and allowedHostsNote sat twice in eight locales: under egress, where English has them and the UI reads them, and again under inbox, where nothing reads them. In de and ko the two copies had drifted apart. Delete the inbox copies and keep the egress ones. Assisted-by: Claude:claude-opus-5-5 * Bring three design specs in line with what shipped - web tools and shadowing: the status still said phase 2 waits; both phases shipped. Note that the Web add-on's keyless entry is now the built-in keyless chain. - Home front door: Home now creates the chat and sends the first message itself, with a short-lived hand-off to the chat page, and its composer takes attachments. Both were described the old way or as out of scope. - chat footer: the meter bar is 96 px with the tokens left, and a phone keeps the bar and the tokens left, not the percentage only. Assisted-by: Claude:claude-opus-5-5 * Store the command card's answer and read it back every two seconds The answer to the command card of the built-in bash went only as a bus message. When the message was lost, the call waited the full five minutes and ended as not answered, although the person had clicked. The network card stores its answer first and re-reads the card every two seconds. The command card now does the same, through one shared pair of helpers: listenConsent subscribes before the card is stored, and waitConsent wakes on the bus message or every two seconds and reads the record, which holds the truth. The network card uses the same helpers. HostExecDecide writes the decision to the card atomically before it sends the message: the same answer twice is 204, the other answer 409 already_answered, and an answered card is no longer shown as waiting. Assisted-by: Claude:claude-opus-5-5 * Load skills by name with a load_skill tool A weaker model failed to read a skill at the path the index showed. The chat now has a built-in load_skill tool that takes a skill's name and returns the instructions of its SKILL.md, without the frontmatter. The tool reads the same resolved skill set as the index: the nearest skill of each name, the chat's switches, the add-ons the chat can use now and the grants of its agent. It reads through the user's rights. When the skill's folder holds other files, the answer starts with the base directory and names up to 20 of them for read_file; a chat without read_file gets the instructions only. A skill that needs commands, in a chat that cannot run them, is refused with the reason. An unknown name is an error that gives near names first and then every name the chat has. Every chat with at least one skill gets the tool, and a sub-agent gets it with its parent's tools. It has no switch and Settings > Tools does not list it. The index of skills now lists names and descriptions only and tells the model to call load_skill. The Sandbox sentence and the error for a wrong skill path now point at load_skill too. Assisted-by: Claude:claude-opus-5-5 * Tell the built-in skills to load other skills by name localai-apps told the model to read the localai-workspace skill, and writing-skills said a description is what the agent sees before it opens the file. Both now speak of loading a skill with load_skill. localai-workspace keeps the path of its schemas folder and says to read it with read_file, since those files are not a skill. A test checks that no built-in skill sends the model to another skill by "read" or by the path of its SKILL.md. Assisted-by: Claude:claude-opus-5-5 * Document load_skill and the skills index The chats page now says that the prompt lists skills by name and description, that load_skill returns a skill's instructions and names the files of its folder, what it refuses and who gets it. It no longer says the model reads a skill by its path. The add-ons page says how a chat loads an add-on's skill, and the saved agents page says that the skills grant limits load_skill too. Assisted-by: Claude:claude-opus-5-5 * Add the proposal action spec to the tables-acceptance inventory The spec that an editor cannot confirm a schema change as a write ran in the suite but was never in the inventory, so the evidence manifest check failed. CI did not reach that check before, because the burst spec failed first. Assisted-by: Claude:claude-opus-5-5 * Give a chat a goal that it works toward by itself A chat can now hold one goal. While it is active, every turn's prompt names it, the model gets goal_done and goal_blocked, and when a turn ends by itself the frontend that ran it starts the next one with "Continue toward the goal". The person's waiting messages still go first, and a turn they start does not count as an automatic one. The goal pauses when the person presses Stop, when a turn fails or is cut short, and when the automatic turns or the model steps reach their limits. Resume gives it fresh budgets. A sub-agent, and a chat that a trigger, a connector, a job or a page started, has no goal and no goal tools. The goal lives with the chat record in the store (UpdateGoal on the JSON and GORM stores, with contract tests and a column migration), so every frontend reads the same one. The next turn goes through the same turn claim as any other, so only one frontend continues it; a reaped turn, or a JSON store that reloads after a crash, leaves the goal paused instead of continued twice. The transcript keeps goal lines (set, replaced, continue, paused, resumed, done, blocked, cleared) as their own role, so an automatic message never reads as typed by the person. Assisted-by: Claude:claude-opus-5-5 * Add a store for the loops of the chats A loop repeats a prompt in one chat, at an interval or when the model asks for the next run. The new loops package keeps the loops, the daily budget of runs of each owner, and which chats confirmed a loop once. JSON and GORM stores share one contract suite. Add checks the limits per chat and per owner in the same step as the insert, and TakeRun never hands out the last run of a day twice, also across frontends. A user message that a loop sent now carries a loop mark (loop ID, run number, self-paced or not). Both chat stores keep it. The model reads a short line before the prompt that says the message repeats by itself, and, for a self-paced loop, how to ask for the next run. Assisted-by: Claude:claude-opus-5-5 * Add the file preview slot specs to the host-slots inventory The file preview slot came with two Go specs in the host-slots package. verify.sh found them in the run but not in the inventory. Assisted-by: Claude:claude-opus-5-5 * Check the Settings page and the add-ons with the UI copy guard The guard read the locale files only. The Settings page keeps its English labels and descriptions in settingsSchema.js, and several add-ons carry their own English bundle, so neither was checked. The guard now also reads the label, description and note of every setting and the label and hint of every group, and the English bundles of the add-ons (an en.js module or an English JSON file). The allow-list takes settings:<key>.<what> and addon:<id>:<key> entries, each with its reason; the operator ones (the SearXNG JSON format, the fine-tuning file types, the API route of a voice, the schema of the Databases add-on) are listed. What it found in the voice library is fixed in plain words: "model YAML is not modified" and "Existing model YAML is not changed" now say that the models' settings are not changed, in the voice add-on, the Studio add-on and the host locales that have the strings (en, fr, pt-BR). The two host allow-list entries for them go. Assisted-by: Claude:claude-opus-5-5 * Add the update digest specs to the updates inventory Two checker specs pin that an update names its candidate digest and that a stale cached notice stays readable. The updates suite runs the whole checker package, so the inventory needs both. Assisted-by: Claude:claude-opus-5-5 * Draw the Add-ons search and state filter with the kit pieces The Add-ons page had its own copy of the search box and the segmented filter (ad-search, ad-seg), the same shapes and styles as the kit's SearchField and Segmented that the Agents pages use. Use the kit pieces and drop the copy; the narrow-screen rules now target the kit classes inside the toolbar. The search keeps its test id. Assisted-by: Claude:claude-opus-5-5 * Add the limits of a chat goal to the runtime settings chats_goal_max_turns (1 to 200, default 20) is the most turns a chat with a goal starts by itself, and chats_goal_max_steps (1 to 10000, default 400) the most model steps of all the turns of one goal, before the goal pauses. Both are read when a turn ends, so they apply without a restart. They sit in Settings > Tools next to the step limit of one turn, and the settings page documents them. Assisted-by: Claude:claude-opus-5-5 * Add the goal routes of a chat to the API GET, PUT and DELETE /api/chats/{id}/goal read, set and clear a chat's goal, and POST /api/chats/{id}/goal/resume resumes a paused or blocked one. They belong to the chat's owner like the other chat routes and take the chats feature. A chat view carries the goal with the limits that apply now, the transcript gives goal lines with goal_event and goal_turn, and the event stream has a goal event so that other tabs read the chat again. The routes are in the route feature registry, the swagger file and the chats entry of /api/instructions. They manage the person's own chat, not the server, so they have no MCP tool, as the other chat routes. The chats page documents goals, their states and limits, and how several frontends share them. Assisted-by: Claude:claude-opus-5-5 * Redesign the notification inbox The inbox panel was a plain list: every row had the same weight, a failed run looked like a finished one, the panel ran off the screen on a phone, and a failed load showed nothing. The panel now has a header with the unread count, a menu for Clear read and Clear all, Mark all read, and an All / Unread / Failed filter. Items are grouped under Today, Yesterday and Earlier. Each item shows what sent it (trigger, connector or LocalAI) with a tinted icon, the title, two lines of text or the reason of a failure, and when it happened, with the exact time on hover. A click marks it read and opens its chat; a connector that was switched off opens the connectors page. Mark read and Delete show on hover or focus, and stay in view on a touch screen. The panel has a loading skeleton, an error with Try again, an empty state that says what shows up here, and a caught-up state. A new notification slides in, is highlighted for a few seconds and is announced once. The badge stops at 99+. Focus moves into the panel when it opens; the arrow keys, Home, End and Delete work on the items, and Esc returns focus to the button. On a phone the panel is a sheet over the bottom of the screen with 44 px targets. Assisted-by: Claude:claude-opus-5-5 * Explain the reads of the text index and the provider requests to gosec gosec flagged three new lines. The text index reads files from the directory the host gave it, with names it listed there. A search provider sends its request to the endpoint fixed at construction, never to a URL from a tool call. Mark the lines with the reason. Assisted-by: Claude:claude-opus-5-5 * Keep the add-on details open when Escape closes a dialog Escape in the restore dialog of the recovery panel also closed the details pane around it. The pane handles Escape in React, and the dialog listens on the document, so the pane ran first and saw no preventDefault. The pane now registers as a modal layer and closes only when it is the top layer, as the side panes do. Assisted-by: Claude:claude-opus-5-5 * Add skill and agent messages, compact and clear to the chats API A message can now carry a skill of the chat or one of the user's saved agents. The server checks the name against the chat's resolved skill set (switches, the agent's grants, shadowing, add-ons) or the saved agents the chat may call, and refuses an unknown one with a 400 that lists what the chat has. A skill message gives the model the text of the skill's SKILL.md before the user's words, for that turn only; an agent message asks the model to start the agent with spawn_agent. The transcript keeps the words and the command (Entry.Invoked), not the instructions. A regenerate or an edit runs the command again. Such a message does not wait for a running turn: it answers 409. POST /api/chats/{id}/compact runs the summary the loop makes when the context fills up, through a new optional session.Compactor that the loop harness implements with its own history compaction. The compacted state is saved for the next turn and a "compacted" line with the sizes goes in the transcript. POST /api/chats/{id}/clear writes a "cleared" line and drops the saved state. The model then sees nothing above the line: the rebuilt state and the transcript summary start after the last such line, so a fork from above it keeps what the model saw there. Export, import and the Markdown export carry both lines. Both calls take the turn claim, so no frontend starts a turn on the chat meanwhile; messages sent during them start a turn after. No MCP tool: they act on one chat of the caller and are not admin calls. Assisted-by: Claude:claude-opus-5-5 * Give each network card spec its own settings A check that a card spec starts can still wait on its card when the next spec begins. It read the settings through a closure over a variable that the next BeforeEach replaced, which the race detector reported. Each service now reads the settings holder it was built with, under a lock, and the specs change the settings through it. Assisted-by: Claude:claude-opus-5-5 * Show the effective audit retention in the settings snapshot The settings snapshot returned 0 for chats_egress_audit_days while the setting was unset. Any page that saves the whole snapshot back, such as the Traces settings strip, then posted 0 and the server rejected it with "Invalid chats_egress_audit_days", so saving tracing settings failed on a fresh install. Report the retention that applies (30 days by default), the same way chats_max_iterations does. Assisted-by: Claude:claude-sonnet-5-5 * Run the Sandbox smoke test in a short scratch folder The smoke test step was reached for the first time and both isolated backends refused to start: their state folder under the checkout's cache made the socket paths 126 and 132 bytes, over the 107 a unix socket allows. Point SANDBOX_SMOKE_WORK at /tmp/sandbox-smoke, which leaves room for the sockets. Assisted-by: Claude:claude-opus-5-5 * Use paths from the home folder in the Sandbox command specs bash now starts in the user's home folder, so the end-to-end specs that read and wrote home/<name> found nothing. Use names relative to the home folder, as the Sandbox specs already do. Assisted-by: Claude:claude-opus-5-5 * Show the goal of a chat on the chat page A bar above the composer shows the goal with its state: Pause (which stops the running turn and so pauses the goal) while it works, Resume when it paused or needs the person, and Clear. A second line says why it paused, what the model needs, or what it did. The status line shows "Goal · turn 3 of 20" while the goal is active, and the Continue button of a paused turn stays away then, since the next turn starts by itself. The transcript shows the goal lines as quiet rows: set, replaced, continuing (with its turn), paused with the reason in plain words, resumed, reached with the summary, blocked and cleared. A goal event, also from another tab, reads the chat again, and a hidden live region says each change to a screen reader once. The strings are in all nine locales. Unit tests cover the helpers and the thread rows; a Playwright spec covers the bar, the progress, the lines, Pause, Resume, Clear and the event. Assisted-by: Claude:claude-opus-5-5 * Run the loops of the chats The loop host fires the loops that are due. A run sends the prompt to the chat as a new message only when the chat is idle: a busy chat skips the run, and an interval loop moves to its next slot instead of piling missed runs up. A self-paced loop gets two tools while it is active: schedule_next (clamped to the minimum interval and one day) and loop_done. A run in which the model calls neither ends the loop with a note. A loop ends after its run limit, pauses after three failed runs in a row, and waits for the next UTC day when the owner's daily budget of runs is used up. One frontend fires the loops, the holder of a new advisory lock. A run is claimed in the store first, and the turn claim keeps one turn per chat, so a short overlap of two leaders starts no run twice. A new leader finishes the runs the old one watched. Deleting a chat removes its loops; a due loop whose chat is gone removes itself. Five runtime settings bound the loops: chats_loop_min_interval, chats_loop_max_runs, chats_loop_max_per_chat, chats_loop_max_per_user and chats_loop_daily_runs. Assisted-by: Claude:claude-opus-5-5 * Add the loops API of the chats GET, POST and DELETE /api/chats/{id}/loops list, start and stop the loops of a chat; PUT and DELETE /api/chats/{id}/loops/{loopId} pause, resume, confirm or stop one. GET /api/chats/{id} carries the loops, the transcript marks the user lines a loop sent with loop {id, run}, and a loops event tells every open tab that the list changed. The routes are gated by the chats feature and documented in the swagger spec and the API instructions. They act on the caller's own chats only, so they get no assistant tool. The chats page gets a loop section, with when to use a loop and when a trigger, and the cluster behaviour; the runtime settings page lists the five loop settings. Assisted-by: Claude:claude-opus-5-5 * Show the loops of a chat above the message box Each loop of the open chat shows as one line over the message box, such as "Loop · every 5 min · run 4 · next at 09:20", with Pause or Resume and Stop. A loop that paused itself or ended says why; an ended one has Dismiss. The first loop of a chat shows a card instead: "Run this every 5 minutes until you stop it?" with Start and Cancel. The server keeps the pending loop, so a reload shows the card again. A message that a loop sent carries a small "Loop run 4" label. The page reads the loops again on the loops event, so every open tab follows. The answer to /loop list is ready for the slash command. The strings are in all nine locales. Assisted-by: Claude:claude-opus-5-5 * Build the chat slash menu on one command registry The slash commands of Home and of a chat now come from one registry, src/utils/slashCommands.js. A command has a kind (navigate, insert or server), the surfaces that offer it, a needs check and, optionally, an argument provider: args(query, ctx) returns the items the menu lists when the box holds "/cmd <query>". Every existing command and test id stays; the registry adds: - one command per skill of the chat or the draft (/review), marked built-in, add-on, shared or yours, and /skill <name>, which also reaches a skill named like a command; - /agent <name> <task>, which lists the saved agents, or offers to turn them on for the chat when they are off; - /model and /models with a model name, which switch at once; - /compact and /clear, which show their progress in the status line and leave a marker in the thread; the context meter drops; - /attach and /attach clear, /help (a card that lists the commands) and /about (a card with the model, its state, the tools, skills, where commands run and the context size). Tab completes a command or an argument, Enter runs it, and the first row whose name starts with what was typed is selected first. A message typed as a skill or agent command is sent with its marker, not as text; the transcript shows the words and a chip. Home offers the same set without the commands that need a thread. The words are in all nine locales. The registry has unit tests, and a Playwright spec covers the menu, the completion, the marker, the chip, compact, clear and the cards with the routes stubbed. The docs page gains a slash commands section. Assisted-by: Claude:claude-opus-5-5 * Add the /goal command to the chat /goal shows a card with the goal, its state, the automatic turns and steps it used out of the limits, and when it was set. /goal <text> sets a new goal at once, replacing the one the chat had without a question, and the goal bar says what it replaced. /goal clear and /goal resume act on the goal; typing "/goal " lists resume and clear for the goal the chat has. None of these is sent as a message. In a new chat, /goal <text> creates the chat first. The command sits in the Context group of the shared command registry, on the chat surface only, and not in a chat that a trigger or another run started. A skill named goal is reached through /skill. Unit tests cover the parser and the menu items; the Playwright spec covers the card, setting, replacing, resuming and clearing from the box. Assisted-by: Claude:claude-opus-5-5 * Add /loop to the chat /loop [interval] <prompt> starts a loop in the chat; /loop list shows the loops of the chat with the code that /loop stop takes, and /loop stop [id] stops one or all. The command is in the slash registry: after "/loop " the menu offers list and stop, and after "stop " the loops of the chat. A loop prompt that invokes a skill (/review the diff, or /skill review the diff) runs as the person's skill command would: the model gets the skill's text, and the transcript shows the words, the skill's chip and the loop label. The server checks the skill when the loop starts and at each run; any other command is refused with the reason. Assisted-by: Claude:claude-opus-5-5 * List the loop settings on the Settings page The five limits of the loops of the chats show under Tools with the other chat settings: the shortest interval, the runs per loop, the loops per chat and per user, and the loop runs per day. All apply at once. Assisted-by: Claude:claude-opus-5-5 * Clear a sent Home draft when the page goes away too Home clears the draft of a sent message when it closes, so the text stays visible while the composer moves into the new chat. While the chat page still loads, a reload or a closed tab could come first, and the sent text came back on Home. Leaving the page now clears it too. Assisted-by: Claude:claude-opus-5-5 * Check the close error of a previewed file golangci-lint (errcheck) flagged the deferred Close of the file the preview endpoint reads. Discard its error explicitly, as the other studio file handlers do. Assisted-by: Claude:claude-opus-5-5 * Explain the document read of the text index to gosec gosec reported the read of a stored document text as a file inclusion through a variable. The path is the configured index directory joined with a hash that documentPath refuses when it holds a path separator or a dot, so it cannot leave the documents folder. Assisted-by: Claude:claude-opus-5-5 * Build the darwin/arm64 add-on packages on macos-15 GitHub retires the macos-14 runner image on 2026-11-02 and runs scheduled brownouts until then: every darwin/arm64 package job failed at once with the brownout notice. macos-15 is the Apple Silicon image that replaces it. Assisted-by: Claude:claude-opus-5-5 * Expect the chat's own load_skill tool in the agent prompt spec The spec that keeps the agent engine's text out of the prompt also required that no load_skill tool or mention reach the model. Chats now offer their own load_skill tool and the skills index tells the model to call it, so the spec failed. Keep the checks on the engine's own skill and command text, and check that the chat's tool is offered. Assisted-by: Claude:claude-opus-5-5 * Read the user's scripted tool call in the mock model The chat prompt now names the chat's work folder, and the folder name comes from the chat title, which holds a cut copy of the user message. The mock model read the first ADDON_TOOL_CALL marker, the cut one in the system prompt, and called the tool with broken arguments over and over. Read the last marker, the one of the user message. Assisted-by: Claude:claude-opus-5-5 * Keep the phone targets of the inbox at 44 pixels in every build On a phone the inbox close button and the item actions must be 44 pixels wide. Their rule had the same weight as the kit's .dk-btn min-width, so the order of the stylesheets decided. The single-chunk coverage build puts the kit later and the close button shrank to 28 pixels. Scope both rules to their container so they win in any order. Assisted-by: Claude:claude-opus-5-5 * Let the about-you spec accept the sidebar's own seen date When the user has pages, the sidebar writes the date of its New label under seen.new in user.json. In the whole suite other specs make pages first, so the saved record held that date as well and the exact comparison failed. Check that the uploaded seen record is kept. Assisted-by: Claude:claude-opus-5-5 * Give the typed-while-working chat spec time for two turns The spec waits up to 30 seconds for each step, but two turns of the mock model with its tool steps take about 28 seconds on their own, so the default 30 second test limit cut it under load. Set its limit to 90 seconds. Assisted-by: Claude:claude-opus-5-5 * Scope the foundation spec's state query to the sheet header The details pane now repeats the observed state under Health, so a role query for "State: Off" inside the add-on sheet matched two elements and failed in strict mode. The administrator-manages spec, and the two specs after it, did not run. Query the chip in the header (.ad-pane__meta), which is the state the administrator sees next to the add-on's name, through one helper used for all three assertions. Assisted-by: Claude:claude-sonnet-5-5 * Wait for the page to read each apps answer in the registry polling spec The spec moved the page clock as soon as the stand-in saw a request. The registry sets its next timer only after it reads the answer, so on a slow runner the clock could pass the due time of a timer that did not exist yet, and the next read never came. With the CPU throttled ten times, the old spec failed in 3 of 3 runs. The spec now counts the answers the page has read and lets queued work finish before it moves the clock. It pauses the clock before it leaves the page, and it tries the pause again when a slow machine passed the target time. The assertions do not change: one read at once, one read per interval and no faster, and no read after the disable. Assisted-by: Claude:claude-opus-5-5 * Close the egress test listener only after it accepts The dial helper of the egress proxy specs closed its listener when it returned, while the goroutine that accepts the connection could still be waiting to run. On a loaded runner the listener sometimes closed first. The kernel then reset the connection that was still in the accept queue, the spec's write consumed the reset error, which the spec ignores, and the next read saw EOF. The spec failed with "unexpected EOF" before the proxy got the request. Wait for Accept to return before the listener closes, then start the proxy handler on the accepted connection. Assisted-by: Claude:claude-opus-5-5 * Run host commands in a shell that reads no startup file The host backend promises a fixed environment, but bash -lc reads /etc/profile and the user's dotfiles. On macOS path_helper rewrote PATH there, so the command did not get the fixed PATH. Turn the login shell into 'bash --noprofile --norc -c' in the host backend only, and make the test assert the exact fixed PATH. Assisted-by: Claude:claude-sonnet-5-5 * Answer the command card in the platform turns and 400 a bad answer The tool-list spec of the platform suite timed out: without accounts the one person is an administrator, so the chat has the built-in bash in ask mode. The tool list is sorted and bash is first, and the mock model calls the first tool, so every turn waited five minutes for a yes on the command card. The turn helper now refuses each card it sees, as a person who clicks deny, and the turn goes on to its step limit. bash stays in the checked list. An administrator who posts a consent with no card id or an unknown decision got 403 "only an administrator can allow commands". That is a bad request, not a refusal: it now answers 400, as the swagger already says, and the route check for administrators passes. Assisted-by: Claude:claude-opus-5-5 * Let a chat switch off its shell, file, media, collection and sub-agent tools The Tools menu of a chat switched only three groups of built-in tools: web, saved_agents and triggers. A person could not keep a chat away from the shell, the files, the helper models, the collections or the sub-agents. Add five groups to the registry of built-in tools, all on until the chat switches them off: - shell: bash, whoever serves it. The Sandbox add-on's bash and the built-in bash on this machine both go, and the add-on list reports commands switched_off. The row stays listed when nothing can run commands, not usable, with the reason. - files: the file tools and read_document. - media: describe_image and transcribe_audio. - collections: list_collections and search_collection, usable only for a user with the collections feature. - subagents: spawn_agent. list_agents and /agent follow it. The registry gets two flags. External marks a tool that its own code builds for the turn (with the wrappers it needs); the registry lists it in its group and Hidden tells the builder what the chat switched off. Covers makes a group's switch also hide the add-on tool that replaces the built-in one. PUT .../builtins, the panel, the draft list and the fallbacks use the existing group code. A sub-agent follows the switches of its parent chat. The turn prompt is built from the final tool list: no file tool names, no skill writing hint, no work folder and no rules pointer when the chat cannot use files, no describe_image when the helper is off, and a commands block that says the person switched commands off. Assisted-by: Claude:claude-opus-5-5 * Queue the edited chat message after the first step starts The spec queued its message as soon as the chat showed a busy state. The turn is busy before its first model request starts, and that request takes every message that waits. With the chat's tools and skills, the first request now starts a few hundred milliseconds after the turn, so the message went with it, and the edit got 409 "already delivered" (5 of 6 local runs). Wait until the answer streams, as the Stop spec does: the request is running then, and the queued message waits for the next step. Assisted-by: Claude:claude-opus-5-5 * Give the reload chat spec a turn that outlasts the reload The spec reloaded the page during a 3 second turn and expected the page to show the turn still running. A reload of the coverage build on a busy runner takes 4 to 5 seconds (measured with a 6x CPU throttle), so the turn had ended before the page could show it. With the 30 second turn the page always showed it running, so the resumed state works and only the window was short. Add MOCK_LONG_STREAM_MEDIUM to the mock model, a turn of about 10 seconds, and use it in the spec with a 60 second limit. Assisted-by: Claude:claude-opus-5-5 * Wait for the chat page after the Home hand-off in the specs Send on Home changes the address inside a view transition, and the chat page mounts about 35 ms later. Specs that waited only for the address acted on Home in between: - The title spec pressed Ctrl+K at once, because the server names a chat after its first message as it arrives, so the title poll ends at once too. The key reached the page before the chat page listened, and the menu did not open (3 of 3 presses at 0 ms failed, all at 300 ms and later passed). - The Back spec went back before the chat page was drawn. Home was never closed, so its draft of the sent message stayed in the box. startChat now waits for the chat page, the title spec for the title in the chat header, and the Back spec for the chat page before Back. Assisted-by: Claude:claude-opus-5-5 * Accept a resolved user.json conflict in the quiet page specs The sidebar notes in the seen record of user.json the first day it labels Pages as new, and saves it from the version it read. All specs share the test server's one user, so tabs of other workers save the same record at the same time: one gets 409 and useSeen reads the file again and saves once more. Three tabs that open /app/files at once give 200, 409, 200. The quiet check of the shell pages counted the 409 as a problem. Ignore a 409 on PUT /api/chats/wellknown/user in that check. Assisted-by: Claude:claude-opus-5-5 * Check only the spec's own items in the Trash after the reload After Empty Trash and a reload, the spec expected the whole Trash to be empty. The Trash belongs to the test server's one user, and specs of other workers move files to it meanwhile, so CI saw "In the Trash: 3 files, 356 B." after the reload. The checks right after Empty Trash stay. After the reload, check that the server's Trash, the Trash tab and Files do not hold the spec's items, once each list has loaded. Assisted-by: Claude:claude-opus-5-5 * Show the new tool groups in the chat Tools menu The Tools menu lists the groups of built-in tools that the server sends. The five new ones need words and an order, and the shell row needs to say who runs the commands. Name the groups in all nine locales (Run commands, Your files, Images and audio, Collections, Sub-agents) and order the rows by what they reach: the shell and the files first, the triggers last. The shell row carries a chip with the provider (Sandbox or This machine) and a line under it that says the commands run isolated, or a warning when they run on this machine. When nothing can run commands the row is disabled with the words of the commands notice. A switched-off sub-agents group also turns /agent off in the composer. The specs cover the rows, the off and on requests, the bulk actions, the provider chip, the disabled reasons, the draft on Home and a real server round trip. Assisted-by: Claude:claude-opus-5-5 * Document the groups of built-in tools a chat can switch off The Tools menu section described only the web, triggers and saved agents groups. List every group with its tools, say which groups the server may leave out and why a row can be not usable, how Run commands covers both providers, what the model is told when a group is off, and that a sub-agent follows its parent's switches. Assisted-by: Claude:claude-opus-5-5 * Fix the host command consent spec for the 400 contract The consent route answers 400 for a body without a card id or with a decision other than once or deny, whoever asks. The spec still expected 403 for always from an administrator. Cover the malformed answers for an administrator and a user, each in their own chat, keep another user's chat at 404, and say it in the chats guide. Assisted-by: Claude:claude-sonnet-5-5 * Find the web tool group by name in the route spec The tools panel lists the shell group before the web group, so the spec that read the first entry compared the wrong group. Look the group up by name, which does not depend on the registry order. Assisted-by: Claude:claude-sonnet-5-5 * Give Skills its own sidebar item Skills work in every chat, but the sidebar filed the page under Agents and only agent skill chips linked to it. Add a Skills item right after Memory, show it in the legacy list too, and stop lighting up Agents on the skills pages. Assisted-by: Claude:claude-sonnet-5-5 * Add Manage skills to the skills picker and a /skills command The picker now ends with a Manage skills link, and /skills opens the Skills page from the composer on Home and in a chat. The picker keeps the height of its status line, so a refused change does not move the rows. Words are added in all nine locales. Assisted-by: Claude:claude-sonnet-5-5 * Say on the Skills page that every chat can use skills The page subtitle named only agents as users of skills. Say that all chats use them, translate it, and document where to manage skills: the sidebar item, the picker link and /skills. Assisted-by: Claude:claude-sonnet-5-5 * Keep Home on screen while its widgets load their strings A Home widget reads the pages strings, and i18next loads a namespace the first time a component asks for it, by suspending. Home had no Suspense boundary of its own, so the route's boundary hid all of Home (display: none) and showed the route fallback until the strings came. A widget shows only after home.json is read, so this happened after the composer was already drawn. Focus given to the composer in that window went nowhere, and a key typed then was lost. The slash menu spec failed in CI this way when another spec had written widgets to the shared home.json: the box was empty and the body had the focus. With widgets in home.json the composer was hidden right after it appeared on 5 of 5 loads, and the spec failed 1 or 2 times in 5. The examples and the widgets now have their own boundary with no fallback, so only they wait. A new spec holds the pages strings while it types in the composer; it fails without the change (the composer never takes the focus) and passes with it. Assisted-by: Claude:claude-opus-5-5 * Send the stubbed file change after the Files page read the folder The change feed spec answered the first stream request with the change at once. The app shell (the appearance sync) opens the shared stream when the app starts, before the Files page chunk has loaded, so on a slow runner the change reached a tab where nobody watched the folder yet and was dropped. The page then never read the folder again, and the spec failed on all three CI attempts. A CPU throttle of 8 makes it fail 3 of 3 times locally. On a real server the page's own first read would already show such a change. The stub now holds the change until the page has read the folder; the page subscribes in the same commit as that read, so the change arrives with a subscriber in place. The resume check is kept. Assisted-by: Claude:claude-opus-5-5 * Give the oversized 3D input as a sparse file, not a buffer The spec built a 32 MiB buffer and passed it to setInputFiles. Playwright sends a buffer to the browser over the protocol: that step alone took 5.5 s on an idle machine, and on the loaded CI runner the test hit the 30 s timeout on all three attempts. It now writes a sparse file of the same size to the test output folder and passes its path, which the local browser opens as is. The page still checks the size before it reads the file, so the toast and the no-request assertion are unchanged. The step takes milliseconds. Assisted-by: Claude:claude-opus-5-5 * Check the tool line, not the tool name, in the tool call journey The spec looked for /list_files/ in the transcript. The user's own message holds that name, so the check could pass before any tool ran. Once the model's call streamed, a second element matched, and the strict locator failed (2 matches on the first attempt, 5 on the retry). It now waits for the activity line that the tool run adds ("Listed /"), which only shows after the tool ran. Assisted-by: Claude:claude-opus-5-5 * Measure the proposal card and its neighbours in one frame The layout spec read the boxes of the message text, the card and the message actions with three boundingBox calls. They are viewport positions taken in three frames, and the transcript scrolls to its end while the card fills in. On the phone the actions were read after a scroll and came out 156 px above the bottom of the card. The three boxes now come from one page.evaluate, so they are read in the same frame. The order assertions are unchanged. Assisted-by: Claude:claude-opus-5-5 * Read the page fixture time when the page is served The task board fixture of the discovery and Pages specs set its change time once, when the file loaded. A worker loads it once and keeps running specs, so a spec that ran two minutes later read "Changed 2 minutes ago by the assistant" where it expects "just now". The change time is now a getter, read when the fixture is serialized into a response. Assisted-by: Claude:claude-opus-5-5 * Check a background process without /proc in the host test The alive helper read /proc/<pid>/stat, which macOS does not have, so it reported every process as dead and the spec failed there with "did not survive the suspend". Use signal 0 for existence and ps (or /proc where present) to tell a zombie from a live process. Assisted-by: Claude:claude-sonnet-5-5 * Keep the composer switch menus at their own height The base rule of the dropdown menu caps it at 320px and lets it scroll. The switch menus set their own height with a rule of the same weight in a lazy sheet, so which one won depended on the order the sheets loaded in. When the base rule won, the picker showed a few rows, and a click on a row at the edge of the list scrolled the list by a few pixels. Double the class on the switch menu rules so they win in any order, and add a test that loads the base rule last. Assisted-by: Claude:claude-sonnet-5-5 * Show what applies in the chat Network switch The Network row highlighted the choice stored with the chat, so a chat that asked for All public sites under an administrator limit of Off showed All highlighted although it was disabled and not in effect. Highlight the effective choice, the lower of the chat's choice and the limit, which the server already computes. When the chat asks for more than the limit, say so in one line instead of highlighting it. Add a line that the limit is set by the administrator, and tell the host network line that limits need the Sandbox add-on. Assisted-by: Claude:claude-sonnet-5-5 * Design the host components that add-ons reference The host UI names the Sandbox and the Web add-ons to place their server settings, and the chat's Network row links to the global Settings page. Describe a closed registry of host components that an add-on references by ID in its manifest, entitled by the existing host operation grants and tool shadows, so the controls live on the add-on's own page and a second add-on can reuse them with no host change. Assisted-by: Claude:claude-opus-5-5 * Let an add-on ask for host controls in its manifest An add-on had no way to say that a host-owned control belongs on its page, so the host UI placed those controls by add-on name. Add ui.controls, a list of IDs from a closed registry of host controls: network-policy (the network of the commands an add-on runs) and tool-providers (the rows of the built-in tools it replaces). The host decides which controls it draws: EntitledControls keeps a control only when the add-on holds its host operation grant and replaces its built-in tool. The field is optional, so every existing manifest stays valid. Assisted-by: Claude:claude-opus-5-5 * List the host controls an add-on is entitled to The apps list now carries ui.controls with the host controls the manifest declares and the host setting entitles the add-on to: the manager passes each add-on's host operation grants to the list. GET /api/chats/{id}/egress gains addon, the add-on that runs the chat's commands when it has the network-policy control, so the chat can draw its Network row from that add-on and link to its settings. applies and applies_reason keep their meaning: the policy holds with or without the control. The model and the 400 error no longer send an administrator to Settings > Tools for the limit. Assisted-by: Claude:claude-opus-5-5 * Draw the host controls on the add-on's own page The host UI placed the Sandbox network and the tool rows by add-on name (SERVER_SETTINGS), showed a read-only summary on the add-on's Settings tab, and linked the chat's Network row to the global Settings page. Add a registry of host controls keyed by the IDs of the contract. The Settings tab of an add-on draws, below its own sections, the controls the apps list gives it, in one form with the pending bar of the Settings page: network-policy reuses the network section and its setting rows, tool-providers the rows of the built-in tools the add-on replaces. The chat draws its Network row only when the server names an add-on with network-policy, and the administrator link opens that add-on's tab. The row itself is unchanged. The Settings page keeps the settings of features that exist without an add-on, points to the add-on where one serves them, and no longer lists the network settings. Its old address opens the add-on's tab. Assisted-by: Claude:claude-opus-5-5 * Ask for the host controls in the Sandbox and Web manifests The Sandbox asks for network-policy and tool-providers, so the network of its commands and the bash row are drawn on its own Settings tab and its chats get the Network row. The Web add-on asks for tool-providers only: it holds no egress grant. The manifest tests check that the default grants entitle each add-on to what it asks for. The proxy's refusal now sends the model to the Sandbox Settings tab instead of Settings > Tools. Assisted-by: Claude:claude-opus-5-5 * Keep the button of the network pointer row at its size The row used the wide layout of the tool rows, so its Open settings button stretched across the page. Use the plain setting row, which puts the button on the right like the other controls. Assisted-by: Claude:claude-opus-5-5 * Fix the add-on contract inventory and the apps Swagger fixture The contracts case inventory missed the six host control specs, so verify.sh rejected the Ginkgo report. List them. The Swagger spec test never saw `controls` in the emitted UIView JSON because its fixture left the omitempty field empty. Set it there. Assisted-by: Claude:claude-sonnet-5-5 * Add the host controls spec to the apps inventory The apps suite ran one more spec than scripts/addons/suites.json listed, so verify.sh rejected the apps report as an inventory mismatch. List the new spec in the report inventory and in the suite's aggregate case list. Every other Ginkgo suite was re-run and matches its inventory. Assisted-by: Claude:claude-sonnet-5-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-11-01 12:08:23 +01:00
mudler-agentandEttore Di Giacinto	2cadf319a0	feat(realtime): LFM3-Audio speech-to-speech on llama.cpp, remove liquid-audio (#12958) ... * chore(backends): remove the liquid-audio Python backend The backend targets liquid-audio, which loads LFM2 checkpoints only. Remove its directory, the gallery file and the LFM2.5-Audio model entries, the library and generated backend gallery entries, the backend matrix and CI job, the Makefile targets and the golden test fixtures that listed it. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(config): drop liquid-audio from capabilities, importers and UI With the liquid-audio backend gone, remove its capability entry, its importer, its artifact-gate entry and the realtime_audio heuristic that only matched that backend. realtime_audio is now set only through known_usecases. Update the Talk and fine-tuning pages, the docs and the swagger text that named the backend. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): bump llama.cpp to e486a4b5 Move the llama.cpp pin to e486a4b559617dc88bf79978996f6c670b6bbc9d, the master head of 2026-10-28. It contains the LFM3-Audio support (ggml-org/llama.cpp#31257) and the realtime server split (ggml-org/llama.cpp#31269). The score task patch is regenerated against the new pin. It applies with git apply and with patch -p1, with no offset and no fuzz. Its hunks in server-task.h and server-context.cpp were rebased by hand next to SERVER_TASK_TYPE_REALTIME and the control dispatch. The C++ backend was built on CPU for this change: the llama, mtmd and server-context libraries and the grpc-server binary link. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): compile server-realtime.cpp into grpc-server server-context.cpp includes server-realtime.h, and its realtime slot type is defined in server-realtime.cpp. Upstream compiles that file into the server library. The gRPC server is one translation unit, so it must include the file too, or the link fails. The include is guarded by __has_include. Forks that predate the realtime split do not ship the file, so they keep building as before. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): probe realtime and TTS voice support in prepare.sh prepare.sh writes llama_compat.h from probes on the checkout. Add two: LOCALAI_HAS_REALTIME (SERVER_TASK_TYPE_REALTIME in server-task.h) and LOCALAI_HAS_TTS_VOICE (set_voice in mtmd-helper.h). The probes let the gRPC server choose the realtime and voice code paths per checkout, so forks without them keep compiling. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): advertise speech recognition and speech-to-speech llama-cpp now registers the AudioTranscription and AudioToAudioStream RPCs, with the transcript and realtime_audio usecases. The default usecase stays chat, so a plain GGUF is still a chat model. The backend implements the RPCs in a follow-up change; until then the calls return UNIMPLEMENTED. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(config): add pipeline.any_to_any for speech-to-speech models A realtime_audio model that serves the whole speech loop itself sets pipeline.any_to_any. The flag is off by default, so chained pipelines and existing realtime_audio configs are unchanged. The realtime_audio heuristic stays explicit: a llama-cpp model is never a Talk-page model unless known_usecases says so. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(config): read the audio encoder and speech-to-speech projector flags Add HasAudioEncoderProjector for clip.has_audio_encoder and HasSpeechToSpeechProjector, which requires the LFM3 audio encoder family together with the gen-audio decoder. Qwen3-TTS sets both encoder flags too, so the flags alone cannot tell it apart from LFM3. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(importer): import LFM3-Audio projectors as speech-to-speech An mmproj that carries the LFM3 audio encoder and the gen-audio decoder now imports with known_usecases realtime_audio, chat, transcript and tts, pipeline.any_to_any: true, the parallel:2 option and the tokenizer template. Other gen-audio projectors keep the TTS-only result, Qwen3-TTS included. The probe runs against a local HTTP server in the tests. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): named-voice TTS and the LFM3 ASR prompt Models without a speaker reference (LFM3-Audio) take a named voice through their "tts" chat template instead of a reference file. The request is validated with needs_reference=false, the prompt is rendered with common_chat_get_tts_prompt(), and the voice is set on the gen-audio input. Qwen3-TTS Base keeps the reference-file rule. The branch is guarded by LOCALAI_HAS_TTS_VOICE, so forks without named voices keep the old behaviour. Transcription takes its prompt from common_chat_get_asr_prompt(). LFM3 gets a system turn that names the language and an empty user turn. Qwen3-ASR's preset is unchanged: fixed user text, no system turn, and a caller prompt is still appended after a newline. The preset call is guarded by a new LOCALAI_HAS_ASR_PRESET probe. The pure parts sit in tts_request_options.h and transcription_prompt.h, with standalone tests run by backend/cpp/run-unit-tests.sh. Add the test-extra-backend-llama-cpp-lfm3 target. It runs the tts, ttsstream and transcription specs against LFM3-Audio GGUF weights. It is a manual run and is not wired into CI. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): mock AudioToAudioStream and pending realtime S2S specs The mock backend serves AudioToAudioStream deterministically: session created, commit with RMS, a tool call on the first turn with tools, then transcript pieces and paced 24 kHz tone frames, with response.cancel ending a reply as cancelled. The realtime S2S specs cover the mock and real-weights variants. They skip until the /v1/realtime S2S path lands, so the mock suite stays green in CI. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): speech-to-speech AudioToAudioStream on the realtime slot Implement AudioToAudioStream on upstream's realtime slot, so a model that answers after the audio (LFM3-Audio) can hold a conversation over the existing duplex RPC. The proto schema does not change: tools travel in the config, tool calls leave as function_call_arguments.done and tool results arrive as a function_call_output item. realtime_session.h keeps what needs no model: the turn text derived from the chat template (with a tool turn), the conversation state, a filter that keeps <tool_call> blocks out of the transcript and drops audio produced inside them, and the translation of slot results into events. grpc-server.cpp wires it to the slot with control tasks and parses calls with the parser the chat template selects. The unit test uses a fake renderer. A ctest-only check renders the real LFM3 template and compares the slot text with the full render. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add the LFM3-Audio-2B entries Two llama-cpp entries for LiquidAI/LFM3-Audio-2B-GGUF, the Q4_K_M default (lfm3-audio-2b) and the Q8_0 build (lfm3-audio-2b-q8), both with the Q8_0 audio projector. Each declares pipeline.any_to_any and the realtime_audio, chat, transcript and tts usecases the importer writes for an LFM3 projector, with parallel:2 and the tokenizer template. The Q4 entry lists the Q8 one as a variant. File sizes and sha256 come from the Hugging Face API for the pinned repo revision. Speaker reference files and the VAD model are not part of the entries: VAD stays optional. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: describe LFM3-Audio speech-to-speech and its gallery entries Adds a speech-to-speech section to the Realtime API page (any_to_any configuration, VAD versus manual commit, tools, transcripts, barge-in, limits), a pipeline.any_to_any row, a llama.cpp speech-to-speech row in the compatibility table, and LFM3-Audio mentions on the TTS and ASR pages. The feature is marked experimental: the behaviour follows the backend session contract and has not been run with real weights. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): run any-to-any models on AudioToAudioStream A realtime_audio model with pipeline.any_to_any now opens one duplex stream to the backend and keeps it for the session, replacing the "any-to-any models not implemented" branch. The chained pipeline is unchanged. The endpoint forwards input audio (with a pre-roll from speech start when a VAD model is set, otherwise every chunk and a client commit), sends response.create, and turns the backend's events into the usual realtime events: audio and transcript deltas, tool calls through the existing function_call items, and a single response.done. A function call output is forwarded before the next response. Barge-in sends response.cancel ahead of the new speech and stops forwarding audio at once. The user's speech is transcribed on the side and reported on the user item; a failure there does not touch the response. Server VAD is on only when pipeline.vad is set. Classifier mode, compaction, streaming.llm, voice gating, sound detection and semantic_vad are rejected for these models. The mock backend now matches the llama.cpp contract (session.updated, the cancel error when idle) and its VAD reports the loud stretches of the audio, so the S2S e2e specs run a VAD turn. The specs are enabled. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): add the real-weights realtime S2S variant and the gRPC s2s spec The realtime S2S specs now check reply audio on real models: duration, loudness, clipping, and keywords found in the model transcript or in a separate ASR of the output audio. run-realtime-lfm3.sh stages the llama-cpp and sherpa-onnx backends, the LFM3-Audio files and the sherpa VAD, ASR and TTS models, then runs the real variant. It is manual and never part of CI: `make test-realtime` still filters out real-models. The backend suite gains an s2s capability (spoken turn, tool call, tool result, spoken answer over AudioToAudioStream) and optional TTS duration and RMS bounds plus a transcript keyword check, all enabled for test-extra-backend-llama-cpp-lfm3. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): expect the idle-cancel error from the mock S2S The mock now answers a response.cancel with no reply running with response_cancel_not_active, as the llama.cpp backend does. The cancel spec still treated that cancel as a no-op and read the error as extra data after the stream ended. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(talk): show any-to-any models as one speech-to-speech model /api/pipeline-models lists a pipeline.any_to_any model with any_to_any set and its optional VAD. A realtime_audio model that is not any-to-any is listed only when it has the four pipeline slots, since the session takes the chained path for it. The Talk page shows a speech-to-speech model as one model: no four-slot view, the VAD slot only when a VAD model is set, and the female and male voices offered. Without a VAD model the server does not detect turns, so the page offers Done speaking, which sends input_audio_buffer.commit and then response.create. Text-only replies stream through response.output_text.delta and .done. self_contained stays in the response for clients built before any_to_any. Assisted-by: Claude:claude-haiku-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): refuse a llama-cpp realtime model without pipeline.any_to_any llama.cpp has no VAD, transcription or TTS RPCs, so filling the empty pipeline slots with the model name failed later with UNIMPLEMENTED. Fail at session start and name the missing setting instead. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): hold manual-commit S2S audio until the commit In manual-commit mode the endpoint streamed every appended chunk to the backend, so input_audio_buffer.clear only emptied the Go buffer while the model kept the audio, and idle audio filled its context between turns. Keep the audio in the input buffer and send it, behind pending text and tool-result items, when the client commits or asks for a response. Frames and controls now go through one ordered write queue, so no gRPC send happens under the audio buffer lock. A tool result created while a VAD turn is open is refused with a clear error. When the stream ends outside a response the session reports it and closes, and a bare EOF from Send reports the status that Recv holds. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): drop the S2S readiness gate and stale comments The gate was always true. Some comments referred to work items that readers of the repository cannot see. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(realtime): add the liquid-audio migration note and the clear semantics Say that the Python liquid-audio backend is gone, which gallery entries replace it and that LFM2.5-Audio has no replacement. Describe what input_audio_buffer.clear drops for speech-to-speech sessions now that manual-commit audio is held back until the commit. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(talk): do not promise turn detection when no VAD model is set The idle note said the server notices when you stop talking. A speech-to-speech model without a VAD model needs Done speaking, so the note now says that. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): make the mock S2S backend keep the llama-cpp session limits The mock accepted what the real backend refuses and hid two endpoint bugs. It now treats output_modalities from the config as a fixed session mode and picks the default per response from the tools set at that time. A fixed text plus audio mode with tools reads the tool call out loud. After the first commit or message, session.update of instructions, voice or tools is an error that does not belong to a response, committing an empty buffer is an error, and response.create while a reply runs is refused. Errors of a refused response.create carry response_scoped in their meta. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): leave the S2S output mode to the backend unless the client sets one The stream config sent text plus audio when the session had no tools yet. The llama-cpp backend keeps a mode from the config as a fixed session mode, and tools added later by session.update never cleared it. The model then read tool calls out loud. Send output_modalities only when the client chose them. The backend picks the default of each response from the tools it has when the response starts. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): mark the realtime errors that answer a response.create The error event now carries response_scoped: true when it ends the request for a response (a refused response.create, a failed commit made by it, or a stream failure). Other errors, such as a refused session.update or a commit of an empty buffer, are not about a response and one that runs keeps generating. The client needs the difference to know whether a response.done is still coming. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): end an S2S response only on the errors that belong to it Every backend error ended the response in flight, although the backend also reports errors about other things (a refused session.update, a commit of an empty buffer) and goes on generating. The rest of the reply was lost and the next response.create met either a refusal or the leftover events of the old reply. End the response only on an error marked response_scoped, forward the others to the client and keep waiting for response.done. Drop the events buffered by an earlier response when a new one starts, and skip the commit of a turn that forwarded no audio. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): run the S2S input transcription detached from the commit slot The commit waited for the side transcription of the user audio while it held the commit slot, and response.create waits for that slot. The ASR latency was added before every reply, and on a backend with few slots a busy ASR task could stall the reply. The model already has the audio, so run the transcription detached under the session context and release the slot at once. The transcript is still reported on the user item. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): do not pin the place of the input transcript in the S2S order The transcript of the user audio is reported on the side and may land before or after the first response events. Check the order of the response events without it. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): stop an S2S reader leak and a cancel that waits on the write queue The reader waited on a channel it closes itself when the events buffer was full, so it outlived the session after a turn ended on an error or timed out. It now also stops when the session closes. Stopping a response sent the cancel through the write queue and waited for it before the grace timer started. With backpressure on the stream the wait never ended and nothing read the events meanwhile. The timer now starts first and the cancel is only queued. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): let a chained pipeline run without a VAD model A pipeline that names no pipeline.vad failed to build, because the VAD slot was always loaded, and a new session still started with server_vad. The slot is now optional for chained pipelines as it already is for any-to-any models: the session starts with turn detection off and the client commits each turn. Turning server_vad on for such a session is refused, and a switch to a pipeline without a VAD drops turn detection. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add LFM3-Audio VAD and chained-pipeline entries Add lfm3-audio-2b-vad and -q8-vad (any-to-any with silero-vad-ggml as the turn detector), and lfm3-audio-2b-pipeline, -pipeline-vad and the -q8 forms (the model chained through its own transcription, chat and speech slots, with and without the VAD). The gallery has no dependencies between models, so the VAD entries state that silero-vad-ggml must be installed first. Drop the variants list from lfm3-audio-2b: it let the installer pick the larger Q8 build on any host with enough memory, so a plain install of the Q4 entry did not get Q4. All entries are now independent. Tests cover the entry shapes, the single-build install and their acceptance by the realtime endpoint. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): run the realtime specs for any-to-any and chained shapes The realtime specs covered one any-to-any model with a VAD. They now run for any-to-any and for a chained pipeline, each with and without a VAD, on the mock backend in CI. VAD-less shapes also check that turn detection starts off and that server_vad is refused. The manual real-weights script writes one config per shape that mirrors the gallery entries and runs the four shapes in turn (LFM3_MATRIX picks a subset). Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(realtime): document the LFM3-Audio VAD and chained-pipeline entries List the eight LFM3-Audio gallery entries, explain that silero-vad-ggml must be installed before the -vad entries, describe the chained pipeline on one model and the behaviour of a pipeline without a VAD, and describe the shape matrix of the e2e specs. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): wait for the detached input transcription in the S2S specs The input transcription runs in its own goroutine, so the transcription completed and failed events can arrive after the response ends. Poll for them instead of reading the event list right after the response. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): release the S2S stream before running server-side tool calls A response held the stream slot until it returned, and it ran the tool calls from inside that scope. A server-side tool starts a follow-up response on the same stream, which then waited for the slot its parent still held, so the client never got response.done. Run the tool calls once the slot is free; the event order is unchanged. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): end an S2S session whose stream dies, and stop queuing after close A stream that died during a response left the session open: later requests failed one by one and VAD frames kept being forwarded. The response now ends the session through the transport, and enqueue drops requests once the session is closed so the write queue cannot grow. The writer is now the only caller of Send and of CloseSend. close() used to call CloseSend while the writer could be inside Send, which gRPC does not allow. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): close the open S2S turn when the VAD stops Switching turn detection to manual in the middle of a VAD turn left the turn forwarding, and the next client commit sent the whole buffer a second time. Stopping the VAD now commits the forwarded audio on the backend and drops it from the input buffer. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): keep a tool result pending while an S2S user turn is open The check that refuses a tool result during a user turn ran when the item arrived, but the item was forwarded later, and the result was marked as sent before the backend took it. If the user spoke in between, the backend refused it and it was never retried. The result is now claimed and queued under the audio lock only when no turn is open, and otherwise stays pending for the next commit or response. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(talk): empty the input buffer for models without a VAD Without voice detection the microphone fills the server's input buffer for the whole session, so Done speaking committed idle silence and the echo of the previous reply. Clear the buffer when the session starts and when a reply ends. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(importer): keep an existing parallel option on LFM3-Audio imports The speech-to-speech import added parallel:2 even when the options already carried another parallel value. Use the same check as the hardware defaults, which also covers n_parallel. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(talk): list chained pipelines that have no VAD /api/pipeline-models required a VAD for a chained pipeline, so the VAD-less LFM3-Audio pipeline entries never showed up on the Talk page. Any pipeline with transcription, LLM and speech slots now qualifies. The Talk page treats every model without a VAD as manual-turn, and the session sheet says so for chained pipelines too. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(llama-cpp): bump llama.cpp to 0ac1a7fd Move the llama.cpp pin to 0ac1a7fd35c127a18c0cbc6c57d12e256a6a385b, the master head of 2026-10-29. It contains tool calls in the realtime server (ggml-org/llama.cpp#31287). The score task patch is regenerated so that it applies with no offset and no fuzz. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(functions): keep a null tool argument null A tagged parameter whose text was "null" came out as an empty string: unmarshalling JSON null into a string succeeds and leaves it empty, so the value was taken for a string. Check the first byte of the JSON value instead. A parameter with the text "null" now becomes JSON null, as in llama.cpp. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(functions): add an lfm3 XML format preset The LFM3 chat template writes a call as <tool_call>, <function=NAME>, one <parameter=KEY> block per argument, then the closing tags. The qwen3-coder format has the same shape and auto-detection already found it, but xml_format_preset: lfm3 was not a known name. Add the preset, with tests for one call, a call without arguments, two calls, string and non-string values (numbers, booleans, null, arrays, objects), a call between text, and unfinished input. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): parse tool calls with the chat template, as upstream Since ggml-org/llama.cpp#31287 the realtime server handles tools with the parser and grammar of the chat template. Do the same in the speech-to-speech stream instead of a hard-coded tag filter and a parse of the whole text at the end. - Each response gets a common_chat_session with the tools of the session and parallel calls. Its grammar and preserved tokens go into the sampling of the response, so the slot keeps the call tokens in the text and constrains the call. - The text of the response is fed to the session piece by piece (task_result_state::update_chat_msg). Content deltas become the transcript; tool-call deltas are collected into one function_call_arguments.done per call at the end. Audio after the start of a call is dropped. - A tool_choice of auto, none or required is accepted in the params of the config, in session.update and in response.create, with the response overriding the session. - The tool turn is derived from a conversation in which the assistant called a tool, like upstream, not from a plain reply. - response.done carries raw_text, everything the model wrote with the call markup, for a caller that parses it another way. If the parser throws, the rest of the response is content and the raw text is the way out. ToolCallFilter is no longer used and goes in the next commit. The unit tests cover the splitter contract (content, call deltas, the final piece, whole characters only). realtime_chat_check feeds LFM3-format text through the session in pieces of 1, 7 and all bytes and checks the call, the transcript and the grammar. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(realtime): drop the tool-call text filter The parser of the chat template now splits content from tool calls, so ToolCallFilter, which looked for the literal <tool_call> tags, has no caller. Remove it with its tests. Every case it covered (a call between text, split tags, several calls, an unfinished call, text after a call) is handled by the parser, and realtime_chat_check runs the LFM3 format through it in pieces of different sizes. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): fall back to the raw text for S2S tool calls For an any-to-any model, the calls the backend parsed still win. When there are none and the request had tools, parse the raw text that response.done now carries the way the chained pipeline parses the text of its LLM (CleanupLLMResult, ParseFunctionCall, ParseTextContent), so a call format the backend parser does not know can still be served with function.xml_format_preset or the other function settings of the model. The content around the calls replaces the streamed transcript only if the config captures it, as in the chained path. Also pass tool_choice to the backend: the session mode with the stream config and with session.update, and the mode of a response.create override with that event. A named function cannot be forced, so it is not sent. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): carry raw_text in the mock S2S response.done Match the llama-cpp session: with tools set, response.done has raw_text, the call markup of the LFM3 template for a tool turn and the transcript for any other reply. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(functions): parse tool blocks that have text between them The iterative XML parser stopped at the first scope block that was preceded by text, so "<tool_call>A</tool_call> then <tool_call>B" gave one call. Skip text between blocks once a call was found; text before the first block still ends the parse as before. Blocks that follow each other, or only whitespace, already worked. Tests cover three calls back to back and with text between them, with the lfm3 preset and with auto-detection. They also pin the current reading of argument values: the parser has no tool schema, so a value such as 42 or true is read as JSON even when the schema types it as a string. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): read LFM3-Audio tool calls with the lfm3 preset Set function.xml_format_preset: lfm3 on the eight LFM3-Audio-2B entries and in the config the importer writes for a speech-to-speech projector, so the server parses the call format of the LFM3 chat template when the backend returns no call (and in the chained pipeline entries, which use the regular chat path). gallery/index.yaml is regenerated from the library. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(realtime): describe how S2S tool calls are read Replace the description of the hard-coded tag filter with the parser and grammar of the chat template, tool_choice, and the raw-text fallback in the server. Document function.xml_format_preset, and set it in the real-weights e2e model configs like the gallery entries do. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(functions): keep earlier calls when a later block does not parse After a valid call, text that holds the opening tag again (a prose mention, a markdown example, a malformed second block) made the parser treat it as a new block, fail, and return no calls at all. Remember the state after each complete block and fall back to it when a later one fails: the calls found so far are kept, parsing stops there and the remaining text stays in the content. Text skipped between two valid blocks now goes to the content instead of being dropped, so StripToolCallMarkup keeps it. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): do not drop audio after a tool call started In mixed mode the speech for text written before a call can still be pending when the call starts, so dropping every audio frame after the first call delta cut the speech while its transcript was sent. The llama.cpp server never drops audio when it reads a result; do the same. If a model speaks the call markup aloud, that is model behaviour to observe on real weights, not something to filter here. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): flush held-back text when a response is cancelled The parser got is_partial=true on a cancelled response, so text it held back (a possible start of a tag) never reached the transcript. The llama.cpp server parses a cancelled response with is_partial=false; do the same. A cancelled response still makes no tool call. The path where the parser throws already sends the text as content and needs no change. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): stop the fake S2S stream from sending after Close The barge-in spec failed now and then with a data race and "send on closed channel": its script pushes audio from a goroutine, and the session closes the stream while that goroutine still runs. A push now does nothing once the stream is closed, and Close waits for a push in flight. The production close order is right: the writer closes the stream only after it stops sending. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): align the S2S tool-call fallback with the backend Four gaps in how the Go side reads the tool calls of an any-to-any reply: - With tool_choice none (the response value wins over the session value) the backend builds no tool parser, yet the raw-text fallback still turned a call the model wrote into a function_call item. Skip it. - Without a capture regex the fallback never cleaned the transcript, so the stored item and transcript.done kept the raw call markup. When the call is XML, use the text left after the markup is cut out; a reply that is only a call now has an empty transcript. - A named function as tool_choice mapped to nothing, so the backend kept the previous mode. Refuse it with an error in session.update and response.create. - A response-level tools list changed what the fallback looked for, but the backend only has the tools of the session. Use those in both. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(realtime): say how S2S tool_choice and response tools behave Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(e2e): use the real llama-cpp backend in the real realtime run The suite always registered the mock backend under the name llama-cpp, so a run with REALTIME_S2S_MODEL set never reached the llama-cpp backend found in REALTIME_BACKENDS_PATH. Register the mock only when that run is off. The real run converts the input audio with ffmpeg. Check for it in the suite and in run-realtime-lfm3.sh before any download or backend start, with a message that says what to install, and list it as a prerequisite in the script header and the Makefile target. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): cover the tools path of a client session.update to the S2S backend A tool call that never came back from an any-to-any model raised the question whether the tools of the session reach the backend at all. Add a spec that sends the session.update of the real-weights e2e (flat Responses API tool, no session type) and checks that the backend gets the tool in the chat-completions form. Log the number of tools sent when the stream opens and when a session.update carries them, so a run log shows it without reading the wire. The "No tools received" line of the llama-cpp backend does not show it: the server prints it for the input transcription request, which has no tools by design. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): interrupt the chained pipeline reply while it is in flight With unary stages, the default of the lfm3 pipeline gallery entries, the chained pipeline synthesizes the whole reply and sends it as one audio delta right before response.done. The barge-in spec waited for that delta and then rendered the interruption with the TTS, so the first response was already done when the speech started and ended completed instead of cancelled. Real runs on a GPU showed it with both quantizations. For a pipeline, start the interruption at response.created, which comes before the LLM and TTS stages. Render the interruption audio before the first utterance is sent, so nothing sits between the reply and the interruption. The any-to-any shape still waits for the first audio delta, which that backend streams while it generates. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): accept a number word and its homophone in the spoken-answer check The check that the reply audio says what the transcript says needs a shared word of 4+ letters. A short answer failed it: the model wrote "Four" and the ASR heard "for". Compare numbers in one spoken form, so digits, number words and their homophones match each other. A homophone only counts when the other text has the number itself, so "for" on both sides still shares nothing and the check does not pass on a stop word. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(llama-cpp): ask LFM3-Audio for function calls when a realtime session has tools The default system prompt of a realtime session was "Respond with interleaved text and audio." whatever the tools. Under that prompt LFM3-Audio answers in words ("I'll check the weather ...") and never writes the call. The model card, and the llama.cpp realtime example client, use "Respond with function calls, and interleaved text and audio." when the session has tools, and the model then calls the tool. Pick the default by the tools of the session, and treat the plain default that the LocalAI server sends back as no choice: it follows the tools too, also when they arrive with a session.update before the first audio. A prompt the client wrote is kept as it is. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): transcribe the input of an any-to-any turn between responses The input transcript of a speech-to-speech turn ran at the same time as the reply, on the same model. llama.cpp serves both from one context: the realtime slot wants embeddings on, the chat slot of the transcript wants them off, and each flip makes it reserve its compute buffers again. In a GB10 run that was two reserves of about 175 ms per step, the reply fell to 2.6 tokens per second and the transcript to 380 ms per token, until the transcript was done. When pipeline.transcription is the model of the stream, run the transcript after the response, or before one that is not started yet, and keep responses out while it runs. A separate transcription model still runs in parallel. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): send the input transcript of an any-to-any turn before its reply The previous change moved the same-model transcript after the response. A client that reads until response.done then never saw conversation.item.input_audio_transcription.completed: the transcript was made (the log shows it) but sent after the end of the window. Run it in line with the commit instead. A VAD turn transcribes and then responds. A client response.create already waits for the commit slot, so it starts once the transcript is out. The transcript event now comes before any output of the response of its turn, for a manual commit and for a VAD turn, and never overlaps the response on the llama.cpp context. A transcription model of its own still runs in parallel. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): transcribe the input of an any-to-any model only when asked The stream answers from the audio, so the transcript of the input is for the client only. On the model of the stream it is costly: llama.cpp serves the transcript request and the realtime stream from one context and rebuilds its compute buffers on every step while both are active. On a GB10 that was 26 to 29 reserves of about 175 ms per turn, against 0 to 1 on the llama.cpp server, and the reply ran at a few tokens per second. An any-to-any model no longer fills pipeline.transcription with its own name. Without it the session does not transcribe the input; with another model it does, in parallel with the reply; with the model itself named it does, before the reply. The reply transcript always comes from the stream. The gallery entries set none, so they get the fast path. The mock configs and run-realtime-lfm3.sh name a transcription model, so the real-weights spec still covers the input transcript. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(realtime): end a response as cancelled when a barge-in lands during the LLM call The chained pipeline cancels the response context on a barge-in. When that happens while the model is generating, the backend call fails with the context error, and the response ended as failed: an error event and no response.done. A client that waits for the terminal of the interrupted response then waits forever. On a GB10 the q4 model answered before the barge-in arrived, so only the slower q8 model showed it: the spec read events for 30 s and timed out. When the context is cancelled, a failed Predict call or a failed result is a cancel, as the TTS step and the streamed LLM path already treat it. The response ends with response.done{cancelled} and no error. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): wait for the input transcript that comes from a model of its own An any-to-any model now gets its input transcript from a separate ASR that runs next to the reply. The reply is often over first, so a spec that stops reading at response.done saw no transcript in the VAD shape. Keep reading, up to 30 s, until the transcript arrives. The contract is that it comes, not when. Add a guard spec for what the input transcription is handed: the whole turn, in a WAV that says the rate it was recorded at (the session rate for a client commit, 16 kHz for a VAD turn). Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): check the errors of Close in the realtime specs golangci-lint (errcheck) flagged the unchecked Close of the websocket connections, the response body and the fake stream. Use the `defer func() { _ = x.Close() }()` form the other e2e specs use. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): write the mock S2S specs with Ginkgo and drop the env read forbidigo only allows Ginkgo/Gomega specs and no os.Getenv outside the sanctioned places. Convert the mock S2S tests to Describe/It with a suite bootstrap (the package had none), with the same 11 cases. The mock no longer reads MOCK_S2S_FRAMES: only these specs set it, so the reply length is a package variable they change per spec and restore afterwards. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-10-29 22:06:41 +01:00
mudler-agentandEttore Di Giacinto	e45dc5e9f7	feat(addons): installable add-ons foundation and the Tables add-on (#12702) ... * feat(addons): add the add-on runtime, navigation and first-party add-ons Add the add-on runtime to LocalAI: manifests, the host operation broker, admin gating by account role, the add-on UI bundle builder and the SDK for add-on pages. Move the navigation to Chat, Models, Add-ons and Settings, with pinned add-ons in the sidebar and an Add-ons drawer. Ship the first-party add-ons (voice, face, fine-tuning, quantization) as packages for linux/amd64, linux/arm64 and darwin/arm64. Intel macOS (darwin/amd64) is enabled in the platform files. Squashed from the review branch. The tree is unchanged; the pre-squash history is kept on a local backup branch. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gosec): tighten file modes and annotate false positives Write job state, configs and imported models with mode 0600, clean the paths that are read from disk, and mark the route tables, the header name and the retry jitter as reviewed false positives. Assisted-by: Claude Code:claude-sonnet-5-5 gosec Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): keep API keys off recovery mutations and same-site cookies POST /api/addons/dispatch forwarded the recovery and update mutations to the owner for any administrator key, which skipped the browser-session rule of their own routes. Refuse those actions there. Also apply the same-origin rule to the legacy token cookie, not only the session cookie, because the global CSRF check lets same-site requests through. Do not fail the start of LocalAI when an add-on library cannot be refreshed; log it and keep the installed add-ons up. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): refuse a reinstall that would migrate retained data unbacked An upgrade takes a verified backup before the new package migrates the data. A reinstall over retained data has no running generation to stop, so the new package would migrate it without one. Refuse a reinstall whose data format differs from the retained one. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): release the manager lock while a release is fetched Submit held the state lock across the catalog lookup and the package download, so the apps list and status reads stalled for the whole install. Serialize Submit and Reconcile with their own lock and drop the state lock around the fetch only. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): kill an add-on only after three missed health probes One slow 100 ms probe killed a busy add-on and dropped the work it was serving. Require three misses in a row. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): move settings under the operate hub The Operate hub already has Settings as a tab, so the sidebar no longer shows a separate Settings entry. The /app/settings route stays, and the hub tab still reaches it. The navigation test now expects Chat, Models and Add-ons as the primary items. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): accept same-host browser requests that carry no fetch metadata Browsers do not send Sec-Fetch-* headers over plain HTTP on an address that is not localhost, so every add-on request from a LAN address was refused as cross-site. Keep refusing a Sec-Fetch-Site value other than same-origin. When the header is absent, accept the request only if its Origin, or failing that its Referer, names the same host and port as the request. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): show the server's reason when an add-on request is refused A 403 or 400 answer showed a generic message and hid the reason in the response body, so a refusal by the browser guard could not be told from a missing permission. Show error.message from the body as the server sent it, and keep the generic text when the body has none. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): run the add-on owner in the normal binary The owner was only enabled when an embedding host set the add-on config, so a normal run answered every add-on request as unavailable. Add --addons-enabled (default true) and --addons-ownership-mode (default named-user, or local-owner; anything else stops startup). The node is its own owner: state lives under the data path, and the carrier uses loopback keys made at each start and kept in memory. --addons-enabled=false turns the owner off completely. Named-user mode needs the account database. With authentication off the owner stays off and a warning says how to use local-owner. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): keep the loopback owner address stable across restarts The single-node loopback transport took a new ephemeral port at each start. The owner stores its address in its identity file and refuses to start when the address differs, so every start after the first failed with "invalid addon owner configuration". Choose the port once, keep it in the add-on state root (loopback-port, mode 0600) and reuse it. If the port is taken at a later start, stop with an error that names the port and --addons-enabled=false, because another port would break the identity check. The keys and certificate stay in memory. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): let add-on pages keep the host modules they were written against A page written for the web UI imports React, the router, i18next and a few shared components. The add-on builder refuses a bundle that holds a host file, so moving such a page into an add-on meant rewriting it. Add a compat mode to the shared builder. It answers those imports from a directory of stand-ins that read the SDK, bundles the dynamic import of the page code into the one output file, and collects the style sheets the page imports. A relative import that is not in the add-on's own source resolves to the file at the same path under the compat directory. Widen the SDK for these pages, without a version change: more host components (spinner, modal, lightbox, model picker) and hooks (models, resources, voice profiles, operations, MCP client), a style on icons, the icon names the pages draw, sdk.navigation.openApp to open a page of the app (only addresses under /app, and the way back as the only state) and sdk.portal. The shell passes the query string to the page. The stand-ins cover the router (the tab in the path, the model in the query), the translator over the host locale files, the core API calls through sdk.core.fetch, and byte copies of the pure helpers, with tests that keep the copies equal to the originals. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): ship Studio and Talk as add-ons instead of built-in pages Studio and Talk were compiled into the web bundle and always listed in the Add-ons drawer. They are add-ons now, like Voice, Face, fine-tuning and quantization: a person installs and removes them, the drawer lists them only while they are installed, and they are packaged the same way. Move the pages, their components, hooks and helpers into addons/studio and addons/talk. Each has a manifest, a UI bundle, a release platform matrix and a catalog template, and is built by the compat mode of the shared builder. The core API stays where it is: the pages call it with the signed-in person's session. The history keeps its browser storage keys. The unit tests of the moved helpers move with them. In the core, drop the two pages, the Create group of the sidebar (Chat joins Home and Models in the old layout), the built-in catalog entries (the catalog is empty) and the talk locale namespace. The old addresses (/app/studio, /app/talk and the generator links) redirect to the add-on pages, with the model as ?model=. The Home /studio command is offered only while the Studio add-on is installed and opens it. The release workflow and its tests now list seven add-ons. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): run the Studio and Talk specs against the add-on bundles The pages are add-ons now, so a spec that draws one installs the real built bundle first: the apps list names it and its module and style sheet are served from the build output, the way the add-on owner serves them. studio-addon.js holds that helper, and the Studio and Talk fixtures call it. Update the specs for the new addresses (/app/addons/studio/studio, the model as ?model=), the session permission the add-on route needs, the legacy redirects, the Home /studio command that appears only while the add-on is installed, and the sidebar without a Create group. Add specs for the add-ons themselves: every tab draws with no script error, the shell titles the page once, the old addresses say so when the add-on is not installed, a button of the page reaches a core page with the way back, and the drawer lists both only after they are installed. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): centre the icon of a built-in tile in its icon box The drawer drew the icon of a built-in tile (Studio's palette was one) in a bare 40px box. Only the placeholder box was laid out as a centred grid with the 20px size, so the real icon sat in the top-left corner at the size of the page text. Give the built-in tile the same classes as the placeholder box, and check in a browser that the icon is centred. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(addons): describe Studio and Talk as add-ons Replace the built-in list (now empty) with the Studio and Talk add-on sections, the new targets of the old links, the Home command, and the seven packaged add-ons. Describe the SDK additions the pages use: components, hooks, icons, openApp and portal. Say on the Studio page that it is installed from the drawer. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): click the shell title in the Studio slash-key spec The add-on shell titles the page, and the Studio overview no longer draws a second title, but the generator tabs keep theirs, so a bare h1 locator matched two elements. Click the first, which is the shell's. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): read the loopback port file through os.Root The persisted port file was read with a path built from the add-on root, which gosec reports as a file inclusion through a variable. Open it through os.Root so the name cannot leave the root, even by a symbolic link, and require a regular file with no group or other access. A looser file now fails closed instead of being trusted. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): expect Settings under the Operate hub, not the sidebar Settings moved out of the sidebar into the Operate hub. The navigation spec still expected a sidebar link. Assert that the sidebar has no Settings entry, and that the Operate hub opens Settings as a tab. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): drop Settings from the 404 places Settings is a tab of the Operate hub and not a destination of its own, so the 404 page does not list it. Assert that it is absent. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): drive the media history and TTS specs through the Studio add-on The image, video, sound and speech generators are pages of the Studio add-on now, and the old addresses redirect to it. Install the add-on with the Studio fixtures, open its pages and use its compose card, so the specs run against the page that exists. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): count the first-run steps, not every list item The recommended models are a list inside the second step, so counting all list items read 3 only while the models were still loading. Count the step elements, which does not depend on the load order. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): fail the node refresh by state, not by call count The spec served the list to the first request and failed the rest. The first request is not always the one from the page, so the page could get the failure first and show no list. Serve the list until the page shows it, then fail the refreshes. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): default to local-owner when auth is off A server without authentication has no accounts, so named-user left the add-on owner off. Resolve the unset mode in one place: local-owner when authentication is off, named-user when it is on. A flag or environment value still wins, and an unknown value is still refused. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): register an official add-on library by default Add gallery/addons/index.yaml, a YAML index of the four first-party add-ons (fine-tuning, quantization, voice, face) in the library format. The entries come from addonpkg catalog over locally built linux/amd64 packages, so the digests pin those images. On the first start, when the owner has no sources.json, register the raw GitHub URL of that index on master as the "official" library and write the file. The file is then the record: a library the admin removes stays removed, and an existing file is never changed. --addons-official-library=false skips the default. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(addons): document the ownership default and the official library Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): list Studio and Talk in the first-party index Add the Studio and Talk entries to gallery/addons/index.yaml next to the four existing ones. They come from addonpkg catalog over locally built linux/amd64 packages, so the digests pin those images. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): expect Studio and Talk in the official index The official index now lists six add-ons. Update the test that decodes it so it accepts Studio and Talk. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): add a real icon to each first-party add-on Each icon is a 256 by 256 PNG: a rounded square in one colour of the UI kit palette with a white outline glyph from the kit sprite. Fine-tuning uses sliders, quantization layers, voice waveform, face user, studio palette, talk phone and tables database. They were rendered with headless Chrome from the sprite paths. Assisted-by: Claude Code:claude-sonnet-5-5 chrome Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): ship a committed icon.png instead of the generated square The shared UI builder and the Tables builder copy ui/icon.png from the add-on source tree to the package unchanged when the file exists. They write the generated placeholder only when it does not. Add a fixture with an icon and a test for both cases. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): list the tables add-on in the first-party index Add the Tables entry to gallery/addons/index.yaml next to the six existing ones. It comes from addonpkg catalog over a locally built linux/amd64 package, so the digest pins that image. The official index test now expects seven add-ons. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): make the Voice add-on one page with tabs The Voice add-on had four pages: recognition, library, create and speech, and the drawer listed two of them. It now has one page, "voice", and one drawer entry. Recognition, Library, Text to speech and From a recording are tabs of that page, drawn with the same tab strip the recognition page already used. The tab is the first path segment (voice/library), as in the Studio add-on, and the form that adds a voice is a view of the Library tab (voice/library/new). The old page names stay declared in the manifest. Each one redirects to its tab and keeps the query string, so saved links still open. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): send the old voice routes to the Voice tabs /app/voice, /app/voice-library and /app/voice-library/new now open the matching tab of the Voice add-on page while the add-on is installed and running, and the core pages otherwise. The moved-page helper compares the first segment of the target with the declared pages, so a target can name a tab. The docs and the navigation spec list the new addresses. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): cover the Voice tabs and the old page redirects The add-on unit tests render the tabs instead of the four pages. The Playwright spec checks the tab strip, the address of each tab, the missing Library tab for a member, and that the old page addresses of the add-on redirect to their tab with the query string kept. The migration table test follows the new targets. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(studio): keep the model menu inside a narrow screen The Home model picker sizes its menu from the viewport, and assumes the picker sits at the page gutter. In the Studio composer card the field is further in, so on a 390px screen the menu of the Sound expansion picker ended 7px past the right edge. Size the menu from the field it opens under instead. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): build the add-on bundles once for the whole run Each worker built the Studio and Talk bundles into the same dist directory, and the build empties that directory first. A worker that started, or restarted after a failure, removed the files another worker was serving, so a Studio spec saw "the style sheet of the add-on could not be loaded". Build once in a global setup before the workers start. A run without the setup still builds per worker. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): register the worker when the add-node spec says so The mocked roster added the worker on the third read of /api/nodes. The shell and the page both read it, and which one asks first depends on load, so the page could take the new worker as part of the roster it opened with and never show it as found. The worker now joins after the page shows it is listening. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): hold the gallery answer until the skeleton is seen The spec delayed the answer by a fixed 800 ms and then looked for the skeleton. The assertion polls at growing intervals, so on a slow machine it could step over that window. The answer now waits until the skeleton has been seen, and the spec then checks that the skeleton goes away. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): give the PDF tag the time the other PDF specs have pdf.js and its worker load on first use. The other specs in this file wait up to ten seconds for the result of that load. The first one waited five, and missed the tag on a busy machine. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): build the Face and Fine-tuning bundles in the global setup The navigation spec serves the built Fine-tuning bundle, but nothing built it, so two specs failed on a fresh checkout when the dist folder was missing. The spec also built the Face bundle in a hook that every worker runs, and a build rewrites the folder the other workers serve. The global setup now builds Studio, Talk, Face and Fine-tuning once, before the workers start. The spec builds the two bundles itself only when it runs without the global setup. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): fix the Home answers in the stale pin spec The spec asserts that the page shows no alert. The Home page raises its own alert when the model list fails or a staging operation has failed, and both answers come from the shared test server. A slow reply or a failed install left there counted as an alert about the pin. The spec now fixes both answers, so the alert count tells only about the pin. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): read the server address in the add-node spec from the page The spec expected the register address to carry port 8089, so it failed on any other port. It now expects the origin the page was opened from. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): give the navigation routes spec its own catalog No page is built in today, so the catalog is empty and the three specs that expect the studio and talk entries failed on the list and on the pin seed. They passed only while Studio and Talk were built in. The spec now sets a catalog of two ungated entries for each example and restores the old one afterwards. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(cli): report a failed write of the MLX hostfile The tunnel callback ignored the error of os.WriteFile and set the hostfile variable even when the file was not written. It now logs the error and leaves the variable alone. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): discard the close error of the held listener errcheck flags the deferred Close of the listener that keeps the port busy. The test has no use for that error, so the defer drops it explicitly. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): move the add-on tool tests to Ginkgo The fine-tuning, quantization and add-on host library tests used t.Errorf, t.Fatal and t.Run, which the forbidigo rule rejects. They now use Describe, It and Gomega, with a suite bootstrap in each package. The cases and the assertions are the same. The tables in the host client and the tool schema tests keep one entry per case. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): move the host operation tests to Ginkgo The hostops tests used the testing package calls that forbidigo rejects. They now run as Ginkgo specs under one suite bootstrap. Every check and every per-operation case is kept. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): move the pin store tests to Ginkgo The pin service, file store, sweep and shared store contract tests used the testing package calls that forbidigo rejects. The store contract is now a function that registers its specs for each store, so the file store, the memory-only store and the database store run the same cases. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): move the remaining add-on tests to Ginkgo The built-in catalog, tool arguments, loopback carrier, host operation contract, packaging, application, config, CLI and endpoint tests used the testing package calls that forbidigo rejects. They now run as Ginkgo specs. The endpoint tests also drop the unchecked Close errors that errcheck reports when the auth tag is on. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): wait for the targeted Reinstall button before the install click The "generic deletion blocks targeted install at confirmation" spec dispatches a popstate to set the target node, then clicks Reinstall with a script click. The click can run before the targeted page has rendered or while the button is still disabled, so no install request is sent and the poll on the install count times out. It failed 1 run in 10 with two workers and on every retry in CI. The spec now waits for the button to be enabled first. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(addons): refresh the voice package digest in the first-party index The Voice add-on became one page with tabs and got a new icon, so the digest in gallery/addons/index.yaml no longer matched the build. Render the entry again with addonpkg catalog over a locally built linux/amd64 package; the main page is now voice. The other entries are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 go node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): wait for the target node's Reinstall button after the popstate The "generic deletion blocks targeted install at confirmation" spec dispatches a popstate to set the target node, then waits for a button named Reinstall and clicks it. Until React renders the targeted page, the page still draws the generic Reinstall button. It is enabled and has the same name, so the wait passed on it, the click started a global install, and no node install request was sent. The poll on the install count timed out with the confirm dialog still open. With the CPU throttled to 8x, the spec failed on 20 of 20 runs, and the button it clicked had the title "Reinstall". The spec now waits for the button titled "Reinstall on <node>". With the throttle it passes 30 of 30 runs. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): give the Studio hand-off spec a budget for the cold bundle load The "a result made through a hand-off records which result it came from" spec opens the Studio add-on straight from an address, twice. Each is a cold load: the page shows "Loading" until the add-on module has run, and the next assertion had the default five seconds. The load takes about two seconds on an idle run and seven to ten under CPU load, so the spec failed on a loaded machine with the hand-off note not yet drawn. With the CPU throttled to 6x and eight busy loops running, it failed on 21 of 30 runs. The first assertion after each load now waits up to 20 seconds, and the spec then passed 30 of 30 under the same load. The assertions are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): carry the identity page styles in the Face add-on The Face page is drawn with the shared identity views, but its style sheet held only the clip and error rules. The page, card, model list, known-faces table, enrol sheet and distance scale classes had no rules. They looked right only when the host had already loaded the style sheet of its own Faces page, which a visit through the add-on drawer does not do. Opened that way, the cards had no gap, the model list showed bullets, and the install buttons ran into the names. The Face sheet now holds the same identity rules as the core sheet, so the page looks the same whichever way it is opened. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): scope the narrow-screen identity rules of the Voice add-on The rules for narrow screens (single-column pairs, smaller tool padding, compact rank rows) were copied into the Voice style sheet without their media query, so they applied at every width. They now sit in a max-width 720px query, and the reduced-motion rule in its own query, as in the core sheet. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): drop the repeated title from the Voice and Face pages The add-on shell already titles the page, so the recognition pages showed the name twice: "Voice" above "Voices", "Face" above "Faces". Keep the supporting line and the model picker, as the Studio page does, and remove the unused title strings. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): replace the Pinned sidebar section with an Apps entry The sidebar no longer lists pinned add-ons. It shows one Apps entry that opens the add-ons drawer, with a Show all link below it that opens the same drawer. Pins are used from the drawer, where a pinned tile keeps its pin and move buttons. The server pins API is unchanged. Remove the PinnedNav component and its locale title, rename the items.addons label to Apps, and update the e2e specs and the add-ons documentation to match. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): assert the single shell title on the Voice and Face pages The add-on shell now shows the one title, "Voice" or "Face", and the page no longer repeats it. The tests still looked for the h1 "Voices" and "Faces". Assert that the add-on page has exactly one h1 and that it is the shell title. The core pages keep their own "Voices" and "Faces" titles. Assisted-by: Claude Code:claude-sonnet-5-5 playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): drop the repeated title from the Voice Library and Text to speech pages The add-on shell already titles the page "Voice", so these two tabs showed a second heading below it ("Speech voices", "Speak text"). Keep the supporting line, as the Studio and recognition pages do, and remove the unused title strings. The e2e spec now checks that each tab has exactly one h1, the shell title. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): list pinned apps under Apps in the sidebar A pinned app now shows as an indented child of the Apps entry, with its icon and title, in pin order. Show all stays the last child. The pins come from GET /api/addons/pins through the navigation context, and the apps from the add-on registry, which the drawer feeds on every list read, so the links follow the drawer. A pin without an installed app is not drawn, nothing extra shows when no app is pinned, and the children are hidden when the sidebar is collapsed to the rail. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(addons): say that pinned apps are listed under Apps in the sidebar The sidebar lists the pinned apps between Apps and Show all, in pin order. The pins section and the navigation section said that pins do not appear in the sidebar. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): remove the Show all link under Apps in the sidebar The link repeated the Apps entry, which opens the same drawer, and looked redundant when no app was pinned. Apps is now the only add-on entry; pinned apps still list under it in pin order. The link, its locale key and the style comment are gone, and the e2e spec and the add-ons docs now say that Apps is the only entry. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): show a first-run card on Apps when nothing is installed An administrator who opens Apps with no add-on installed now sees a short card with three steps: choose a library, browse and install, start the add-on. Its button opens the Browse tab. The card is hidden once an add-on is installed and for anyone who cannot install. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): hide the pinned apps in the icon rail When the window is narrower than 1024 px the sidebar shrinks to the icon rail by CSS alone, without the collapsed class. The pinned app links were still drawn there, with the indent meant for the full sidebar, so their icons sat to the right of the rail and were cut off. Hide the pinned app links in both rail states. The Apps entry keeps its centred icon. The e2e spec checks that the rail shows no pinned app and that every icon is centred at 1280 px collapsed and at 1000 px. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): stop the backend redirect test racing the installed list The test reads the redirected URL right after the path changes. The Backends page then loads the installed list and drops a selected backend that is not installed, so the backend parameter was sometimes already gone when the test read it. The mock server has no llama-cpp, and about 15 of 40 repeated runs failed on the backend assertion. The redirect itself was correct, including show_all. Serve the selected backend in the installed list, so the page keeps the selection and the assertions stay as strict as before. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): accept a same-host Referer or Origin for human confirmation Browsers send no Sec-Fetch-* headers over plain HTTP on an address that is not localhost, so creating a resource from a LAN address failed with "Creating a resource requires your own browser session". When Sec-Fetch-Site is present it must still be same-origin with an empty destination, as before. When it is absent, Origin or, failing that, Referer must name the request host and port. A destination or mode that is present must still be the expected one. This matches the rule in BrowserProtection. The credential, token cookie, API key and tool-context checks are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): keep hub entries off the Apps route in the sidebar The Build hub still owned /app/addons through its legacy Add-ons tab, so the sidebar lit both Apps and Build on the Apps page. A tab whose gate is closed now owns no route when the sidebar asks, so only the Apps entry is active there. The hubs keep their own routes, and the legacy navigation still gives /app/addons to the Build hub. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): label the pin control on app tiles and in the detail view The pin was a small icon-only button and hard to find. Each app tile now shows a button reading Pin, or Pinned with a filled pin, and the detail view has the same control in its header. Both toggle the same stored pin. The Apps grid lists pinned apps first, in pin order, then the others. The sidebar list and the server pins API are unchanged. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): check the Voice From a recording tab has one title The Library and Text to speech tabs already assert that the add-on shell title is the only h1. Do the same for the From a recording tab, so a heading added there later fails the navigation spec. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): stop the add-on shell drawing a title over add-on pages The app decides what its page shows. The shell drew the add-on name as a header above every add-on page, so an add-on could not choose its own title. The shell now draws no header on a live, loading or frozen add-on page; each add-on titles its own pages. The unavailable and missing-page states keep the name, because no add-on page is drawn there. The runtime and shell specs now check that the shell adds no page title. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): let each add-on draw its own page title The shell no longer titles add-on pages, so each add-on draws one h1 per page. Voice titles its tabs "Voices" (Recognition and From a recording), "Speech voices" (Library) and "Speak text" (Text to speech); "Add a speech voice" already had its own. Face draws "Faces", Studio "Studio" and Talk "Talk". Tables promotes the database list and database name from h2 to h1. Fine-tuning and Quantization already drew a title. The add-on unit tests and the navigation and Studio specs now assert exactly one h1 per page, with the page's own title. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): add a kind to library entries and read unknown kinds tolerantly A library entry now has an optional kind: addon (the default when it is absent), backend or model. Every index written before kinds existed stays valid and its entries are add-ons. A reader keeps the entries of the kinds it knows and skips the others without reading their fields, so a newer kind never makes an older host refuse the whole index. The refresh result and the library state report how many entries were skipped and of which kinds, and the refresh logs a warning. A kind that is not a plain kind name still makes the index malformed. Search lists add-ons unless the caller asks for a kind, and Resolve returns add-ons only, so an add-on install cannot be pointed at a backend image. A library policy may name the signer with an anchored identityRegex instead of an exact identity, so builds from master and from tags both verify. An add-on install refuses a policy that carries one. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backends): install a backend from a library entry A library can now list backends. An entry of kind backend names the image of the backend gallery by digest, and POST /backends/library/install installs it on the local node. The gallery stays the source of truth and its install path is the one used: the library resolves the entry, takes the signature policy from the library, pins the image by digest and calls the same install as the gallery. Installing a name that came from a gallery, another library or by hand is refused and changes nothing. The signature policy of a backend library names the signer with an anchored identityRegex, so images signed by the merge workflow on master and on tags both verify. A pattern that is not anchored is refused. An install is bounded by a size limit per kind: add-ons keep 512 MiB and a backend image defaults to 8 GiB, set with --addons-backend-max-bytes. The limit reads the manifest after the signature check and before any layer is fetched. The metadata of an installed backend records the library it came from. scripts/backends/cmd/backendcatalog renders entries from backend/index.yaml and pins them against the registry. The pilot is the CPU build of local-store, in gallery/backends/index.yaml. The library is registered on the first start only with --addons-backend-library, which is off by default. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): show the kind of library entries and install backends from them The Browse tab lists the entries of every kind this host reads and marks each one as an add-on or a backend. A backend installs through the backend route and not through the add-on lifecycle: the confirmation says that the backend gallery is not changed and that a backend runs native code, and a refusal of the host, such as a name installed from another origin, is shown with its own reason. The details give the kind, the alias and the signer pattern. The library editor can name the signer with a pattern. Without this a save would drop the pattern of a backend library. The Libraries tab reports the entries a refresh skipped because their kind is not read. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(addons): describe backends in the library Document the entry kinds, how a reader skips a kind it does not know, the backend entry format, the signer identity pattern, the install route, the size limit of each kind, the pilot library and the command that renders it. The backend page links to it. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(mcp): install a backend from a library entry The backend library route installs a backend from an add-on library entry, and the admin REST surface and the MCP catalog must stay in step. Add the install_library_backend tool with an in-process and an HTTP client, the route map entry, the safety-rule listing and a skill recipe. Listing library entries stays on the add-on routes, which the assistant does not expose, so no list tool is added. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): carry the add-on library keys in every locale The add-on library and backend library strings existed only in English, so the other locales had no value for them. Add the missing keys of the addons namespace to de, es, id, it, ko, pt-BR and zh-CN. The values are the English text until a translation is contributed. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Revert "feat(mcp): install a backend from a library entry" This reverts commit 7493cff727. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Revert "docs(addons): describe backends in the library" This reverts commit 57816ed7e6. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Revert "feat(ui): show the kind of library entries and install backends from them" This reverts commit 49b2b151b7. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Revert "feat(backends): install a backend from a library entry" This reverts commit 1d8d3e3421. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Revert "feat(addons): add a kind to library entries and read unknown kinds tolerantly" This reverts commit 484813f856. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): drop the backend library keys from the locales Revert the part of 39f77ae29 that added strings for backend libraries: the kind labels, the backend install texts, the signer identity pattern and the skipped entries notice. The add-on library keys stay. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): add an add-ons control to the chat header Add-on tools were on or off only through the /addons command, with a toast and no state a person could see. The chat header now has an Add-ons button in the style of the MCP dropdown. It lists each running add-on that ships tools with a switch, and a badge shows how many are on. The chat keeps the ids that are on (addonToolIds) next to the older boolean, and both are saved with the chat. A chat that has only the boolean means every add-on. The request sends addon_tools alone when every add-on is on, and adds addon_tool_ids when only some are. /addons turns all on or all off and uses the same state. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): let a chat request name the add-ons whose tools it uses addon_tools=true still offers the tools of every add-on the caller may use. A new addon_tool_ids metadata key, a comma-separated list of add-on ids, narrows that to some add-ons. The list counts only with the opt-in, and a list that names nothing offers no tool rather than all of them. The executor drops the tools of add-ons that are not listed and refuses a call to one as an unknown tool, so a model cannot reach a tool the person turned off. Both keys are consumed so they do not reach the backend. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): cover the per-add-on tool list and the chat header control Go tests cover the list form (the list filters the tools offered and the calls allowed) and the unchanged boolean form. Unit tests cover the chat add-on rules. Playwright specs cover the header control: listing only add-ons that ship tools, toggling each one, the count, /addons in step with the switches, and the request metadata. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): check each add-on page has one title when not enabled The shell names an add-on only where the add-on page cannot mount (not enabled, failed, no UI, bundle not loaded, no such page). Once the page mounts it draws its own title, including when it says it is not enabled on this server, and the shell draws none. The shell code already works this way; this change states the rule beside the unavailable state and pins it in the specs. The shell spec now expects exactly one title in the unavailable states. The navigation spec checks the Face page without a face model (the add-on title only) and the Face page of a stopped add-on (the shell title only). Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(addons): upgrade add-ons on their own when the setting is on A new auto_upgrade_addons setting (flag LOCALAI_AUTO_UPGRADE_ADDONS, default off) lets the owner upgrade an installed add-on when its library lists a newer version. It follows the backend setting: a startup flag, a runtime setting, and the same schedule as the backend update check (a first pass shortly after start, then every six hours). Each pass fetches the library of every installed add-on, and for each available update builds the same pinned upgrade command an administrator sends and runs it through the same lifecycle path. The channel rules, the signer identity, the digest checks, the SDK compatibility check and the backup before the upgrade all apply unchanged. The only difference is the gate: the owner acts on its own setting, so no administrator is asked. A refused or failed upgrade changes nothing and is logged; a candidate that failed is not retried until the library lists another digest. With the setting off, nothing runs and upgrades stay manual. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): add the Upgrade add-ons automatically toggle to Settings The toggle sits beside Upgrade backends automatically and is off by default. It saves the auto_upgrade_addons runtime setting. The English locale gets the label key, and the Settings test fixture lists the new field. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): cover the automatic upgrade setting and document it The setting defaults to off in the config, the runtime settings and the flag, and is read back from the saved settings file. With it on, a real owner upgrades an installed add-on through the manager; with it off, nothing changes, and a change of the setting at run time takes effect on the next pass. The manager specs show the upgrade goes through the same pinned path as the manual one, and that a corrupted package, a failed signature policy, an incompatible release, a missing update checker and a caller who is not an administrator each leave the installed version in place. The new specs are added to the lifecycle case inventory. The add-ons page describes the setting, its schedule and its default. Assisted-by: Claude Code:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(addons): name the library entry identifier id The library entry of an add-on used addonId while the shared envelope and the model entries use id. Release now has the field ID, written as id, and every writer emits it: the index encoder, the package workflow templates, the catalog generator and gallery/addons/index.yaml. The digests in the index did not change. DecodeIndex still reads addonId as a legacy alias for one release, so an existing library and a library served by an older tool keep working. An entry that holds both names is refused. The Apps UI reads id and falls back to addonId. Assisted-by: Claude Code:claude-sonnet-5-5 go node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(addons): use the id field in the auto-upgrade test release The auto-upgrade test arrived after the identifier rename and still named the release field AddonID, so the application tests did not build. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(addons): annotate the auto-upgrade actor label for gosec The actor label looks like a credential to gosec G101. It is a fixed name for upgrades the owner starts itself. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(schema): persist the per-chat add-on tool fields The chat header persists addonTools and addonToolIds, but Conversation had no fields for them, so the server dropped them on save. Add both and update the generated swagger files by hand because swag is not installed here. Assisted-by: Claude Code:claude-sonnet-5-5 go Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): count the add-ons auto-upgrade setting in the settings groups The setting sits in the Backends and galleries group, once. The page now lists 54 settings plus the three branding images, so the spec expects 57. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): pause the fake clock ahead of now in the clear-all spec The fake clock runs from install, so pausing at the current time failed with 'Cannot fast-forward to the past' on a slow runner. The save itself is synchronous and was not lost. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(ui): hold the Studio capabilities reply longer in the skeleton spec 800 ms was shorter than a slow runner needs to render the page, so the skeleton was gone before the check. The product shows it. Assisted-by: Claude Code:claude-sonnet-5-5 node playwright Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(i18n): add the French add-ons locale Master added the fr locale after this branch forked. The add-ons unit test requires every locale to carry the add-ons namespace, so fr needs its own addons.json. Assisted-by: Claude Code:claude-sonnet-5-5 node Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-10-22 17:57:19 +02:00
localai-org-maint-botandEttore Di Giacinto	6b794651a4	feat(diarization): return sound events with include_sounds (#12544) ... * feat(diarization): return sound events with include_sounds A client that wants text, speakers, voice prints and sound events had to make a diarization call and a separate sound call. Add an include_sounds request field to /v1/audio/diarization that adds a sounds array of closed events {start, end, label, confidence}, in seconds. The parakeet-cpp backend runs a tagger-only scene stream over the clip, the same stream and thresholds the live path uses, so a clip gives the same events offline and live. A model with no sound_model companion, or a backend that does not report sound events, fails with 501 and the stable code include_sounds_unsupported instead of an empty list. The proto carries sounds_included so an empty list still means "nothing heard". The localai-proxy backend forwards the field. Swagger, docs and the e2e mock backend are updated. Assisted-by: Claude:claude-sonnet-5-5 [protoc swag go] * feat(gallery): add parakeet-cpp-multilingual-diarization-speakers-sounds Same as parakeet-cpp-multilingual-diarization-speakers (TDT 0.6B v3, Nemotron-3-Diarization, WeSpeaker) plus a CED-Tiny sound_model, so one model name serves /v1/audio/diarization with include_text, include_speaker_profiles and include_sounds. It declares the sound_classification usecase like the realtime scene entries. Assisted-by: Claude:claude-sonnet-5-5 --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-10-07 14:10:45 +02:00
mudler-agentandEttore Di Giacinto	99043b442c	feat(router): route with native decision models (#12449) ... * fix(schema): preserve SystemOne image inputs Assisted-by: OpenAI * test(schema): follow Ginkgo conventions for decision inputs Assisted-by: OpenAI * feat(llama-cpp): dispatch native decisions through Score Upgrade the stock dependency and reconcile Score/TTS patches. Reuse native decision parsing, tasks, formatting and response-reader cleanup; preserve ordinary scoring admission and guard older dependencies. Assisted-by: OpenAI * refactor(systemone): share request and model validation Assisted-by: OpenAI:gpt-5 * fix(systemone): preserve HTTP wire-byte validation limit Keep structural validation separate from the serialized internal request bound so HTML escaping cannot reject valid HTTP payloads. Assisted-by: OpenAI:gpt-5 * feat(systemone): bound images and account native decisions Preserve public wire limits independently from router serialization. Reject unsupported NER images, map native request/capability errors, and stamp explicit usage once. Advertise decisions for stock llama-cpp. Assisted-by: OpenAI * fix(systemone): record usage on registered native route Exercise real registration and billing with a mock native backend. Reject empty native responses, malformed image URLs, trailing JSON, and wire overflow including whitespace. Assisted-by: OpenAI * feat(router): add lazy native decision transport Bind named models through internal ModelSystemOne calls with shared validation and bounded abandoned operations. Remove request and echoed-error contents from decision traces. Assisted-by: OpenAI:gpt-5 * feat(router): classify overlapping policies with native decisions Ask independent noul questions, validate probabilities and preserve first-superset routing. Wire the central factory with config-sensitive invalidation and cancellation-safe resolution. Document native framing and bounded operation limits. Assisted-by: OpenAI:gpt-5 * feat(gallery): add pinned Julia-1 native decision model Add a separate text-only llama-cpp Q8 entry with pinned Apache-2.0 source provenance and checksum. Installed using the gallery installer and exercised choice, score and noul on CPU. Assisted-by: OpenAI * test(router): verify native decisions through central factory Add an opt-in real-model Ginkgo integration covering the native Go loader and C++ transport, token usage, independent overlapping labels, and candidate selection. Document owned-server execution and the intentionally non-quality threshold. Assisted-by: Codex:gpt-5 * fix(llama-cpp): align upstream pin and preserve decision signatures Advance to bed0a856 without losing the automated upstream bump. Detect full-request fill_task support at compile time and forward every question for Nimble framing while retaining the earlier native signature. Preserve reconciled SCORE/TTS patches; add standalone compatibility coverage. Assisted-by: Codex:gpt-5 * feat(gallery): add native decision family defaults Pin Laya, Kev-4B, lev, OpenJev and Nimble artifacts. Verify Laya/Kev/lev gallery installs and CPU contracts on both native pins; clearly mark OpenJev/Nimble runtime validation pending and their noncommercial licenses. Assisted-by: OpenAI * docs(decisions): clarify integrated Nimble prerequisite Record the exact combined backend pin while retaining pending OpenJev and Nimble installation/runtime validation status. Assisted-by: Codex:gpt-5 * fix(gallery): indent native decision model sequences Match repository yamllint indentation for Laya, Kev, lev and OpenJev list fields. Parsed gallery data is unchanged; reproduce CI gallery lint failure before the whitespace-only fix and pass the same command afterward. Assisted-by: Codex:gpt-5 * docs(decisions): record OpenJev and Nimble CPU validation Record gallery installation, checksum/metadata verification and multiquestion native smoke results on bed0a856. Retain noncommercial and text-only limitations without accuracy or deterministic-output claims. Assisted-by: OpenAI * fix(ui): expose native Decisions router classifiers Select classifier models using metadata-driven capability routing, retain tuned thresholds, and validate native decision selections before saving. Cover both native backends and create/save/reopen in the real React editor. Assisted-by: Codex:gpt-5 * fix(router): exclude aliases from native decision discovery Check the originally named config before advertising native Decisions eligibility. Retain target capability inheritance for ordinary generation aliases. Exercise the actual capabilities endpoint with native models on both backends, aliases, and disabled models. Assisted-by: Codex:gpt-5 * feat(systemone): share bounded multimodal input validation Preserve text wire limits while admitting bounded PNG/JPEG decision input. Share collection and header validation across internal and public callers and keep the native runner response budget independent. Assisted-by: OpenAI:API-assistant * fix(systemone): bound admission lifetimes and validate complete images Retain shared admission leases through actual work completion, including abandoned internal operations. Decode bounded image pixels, cap public native responses before usage stamping, and preserve oversized malformed text status precedence. Assisted-by: OpenAI:API-assistant * fix(router): classify images before media fetching Preserve ordered structured probes for native decisions. Defer OpenAI media preparation until routing selects the served model, so rejected decision URLs cannot trigger downloads before shared validation. Guard direct image collection with context-aware shared admission. Keep text classifiers and embedding caches from discarding image input. Retain fail-closed classifier configuration and runtime fallback policy. Add middleware, typed-content, admission, cancellation and cache tests. Assisted-by: OpenAI:API-assistant * fix(router): bound extraction before serialization Check probe budgets before copying text or marshaling message state. Count JSON escaping so oversized internal inputs fail before allocation. Preserve typed Anthropic blocks through selected-model conversion and fallback. Keep retry coverage in Ginkgo without global test registration. Assisted-by: OpenAI * fix(router): bound supported probe serialization Arbitrary structs can bypass the probe budget through pointer marshalers, string tags, and promoted fields. Accept concrete chat schema types and plain JSON values instead of emulating arbitrary struct serialization. Budget escaped direct prompts before marshaling so raw length cannot hide serialized expansion. Preserve runtime fallback and reject oversized input before invoking the decision runner. Add Ginkgo allocation, boundary, and marshaler invocation regressions. Six-package tests, three-package race tests, and full-T2 delta lint pass. Assisted-by: OpenAI:GPT-5 golangci-lint * feat(decisions): enable bounded OpenJev images Validate native decision images before permissive media parsing and pixel allocation. Require both decision image support and a vision projector; missing or audio-only projectors cannot silently become text decisions. Pin the OpenJev Q8 projector and document its license and disk footprint. Add native safety tests, canonical limit parity, gallery and load-option checks, and a reproducible CPU direct-RPC contrasting-image smoke. Assisted-by: OpenAI:GPT-5 * fix(decisions): reject incomplete image streams stb accepts corrupt PNG Adler checksums and truncated JPEG scans. Use bounded zlib validation and strict libjpeg decoding before parsing. Keep dimension and aggregate pixel checks ahead of decoder allocations. Wire decoder dependencies into native builds and runtime packaging. Add regressions for appended EOI and embedded marker bypasses. Assisted-by: OpenAI:GPT-5 * fix(ci): gate native decision image validation Run the decoder security tests outside the stdlib-only native suite. Fetch vendor headers at the backend pin and provision decoder dependencies. Gate Go limit parity and production CMake wiring without model downloads. Assisted-by: OpenAI:GPT-5 * test(decisions): cover multimodal public API paths Exercise shared image contracts through the registered HTTP routes and external mock backend. Add opt-in cached gallery installation and real OpenJev image decisions through SystemOne and both routing APIs. Assisted-by: Codex:gpt-5 * test(decisions): assert isolation and cache bypass Observe external RPC calls and compare complete classifier history. Winner-only and cache-miss checks could hide dropped history or cache use. Give real inference its own application and model directory so shared backend mappings and loaded processes cannot affect mixed suite order. Assisted-by: OpenAI:ChatGPT * test(decisions): isolate fixture globals Disable optional global services in the isolated HTTP fixture and register cleanup before setup assertions. Verify meter provider identity survives fixture creation and destruction. Snapshot observed usage before assertions so failures cannot retain the mutex. Require a successful usage stamp before checking error responses. Assisted-by: Codex:gpt-5 golangci-lint * fix(application): honor optional telemetry controls Skip failover gauge registration when metrics are disabled. Register against the application meter rather than looking up the global provider. Allow embedders to retain the bounded routing log without billing stats. Keep the existing default when stats are disabled. The isolated HTTP fixture uses this option without losing its native router assertions. Assisted-by: Codex:gpt-5 golangci-lint --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-10-04 09:34:21 +02:00
mudler-agentandEttore Di Giacinto	e4fa051ee3	refactor(distributed): put the NATS-only paths behind interfaces (#12395) ... * feat(messaging): add shared subject rules Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(messaging): cover BroadcastRoots, ControlRoots and SubjectRoot Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(messaging): add Broadcaster and enforce subject rules in every carrier Broadcaster is the fan-out half of MessagingClient. The NATS client and the in-memory FakeBus now refuse a subject outside the served roots and any wildcard other than a whole single token, and FakeBus shares MatchSubject instead of its own copy. FakeBus Unsubscribe now removes its own subscription instead of the first one with the same subject. A shared conformance suite in messagingtest runs against both carriers. The distributed e2e specs that used invented test.* subjects, and the one that subscribed with a > filter, now use subjects from subjects.go. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: depend on Broadcaster where only publish and subscribe are used Narrowed to messaging.Broadcaster: nodes/staging_progress.go, nodes/install_progress_publisher.go, galleryop/operation.go, galleryop/service.go, agentpool/user_services.go, agentpool/agent_jobs.go, openresponses/store.go, openresponses/sync.go, syncstate/syncstate.go, finetune/service.go, quantization/service.go and failover/distsync/distsync.go. SubscribeJSON now takes a Broadcaster because it only calls Subscribe, which lets the narrowed consumers use it. Stayed wide: worker/supervisor.go, because its client field also serves the SubscribeReply handlers in worker/lifecycle.go. The request/reply, queue and wiring files (nodes/unloader.go, nodes/file_stager_s3.go, jobs/dispatcher.go, agents/dispatcher.go, agents/events.go, worker/file_staging.go, cli/agent_worker.go, http/app.go) are unchanged by design. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): name the no-route condition and confine the carrier error Consumers matched nats.ErrNoResponders, which names an absence, to demote a node. They now match ErrNoRoute, the control path maps the carrier's failure onto it, and timeouts and worker refusals are pinned as not being no-route. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(nodes): state which FileStager implementations return ErrNoRoute Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): build backend clients through one node-aware seam Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: describe the distributed transport seams Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: correct comments that overclaim after the seams refactor Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(agent-worker): refuse an unserved LOCALAI_AGENT_SUBJECT at startup The messaging client now refuses a subject whose root no carrier serves. An agent worker started with a custom LOCALAI_AGENT_SUBJECT such as tenant-a.agent.execute used to start and then wait on a subject the frontend never publishes to. After the subject rules landed it exited at subscribe time with an error that did not name the setting. Behaviour change: the worker now checks LOCALAI_AGENT_SUBJECT before it registers or connects, and exits with an error that names the variable and says to use a served subject under the agent root, for example agent.execute. The served roots are not widened: a custom root was never delivered by the frontend, and a wider set would reopen the drift the subject rules exist to close. The flag help and the agent worker docs state the constraint. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(nodes): pin the reactions to ErrNoRoute Three callers react to ErrNoRoute and had no spec: the reconciler's upgrade drain falls back to the legacy forced install, the reconciler marks the node unhealthy when a pending op has no route, and the backend-op fan-out marks the node unhealthy. Each spec drives the real caller with a scripted no-responders reply and reads the result from the registry or the recorded requests. A fourth spec pins the other side: a pending op that times out leaves the node healthy and only counts the attempt, so mapping timeouts onto ErrNoRoute would fail here. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(messaging): pin client subject checks and fail the carrier suite in CI Add specs that call Publish, Request, Subscribe, QueueSubscribe, SubscribeReply and QueueSubscribeReply on a client with no connection. Each call must return ErrUnservedSubject for bogus.thing and ErrUnsupportedWildcard for jobs.>. This proves that the subject check runs before the connection is used, and needs no server. The NATS conformance suite is the only check that runs the subject rules against a real carrier. Before this change it skipped without output when Docker was missing. Now it fails when CI is set, so a Linux runner without Docker cannot hide it. It still skips on local runs and on macOS CI, which has no Docker. Add SubjectNodeBackendInstallProgress to the list of constructors that must build served subjects, and ask contributors to extend the list. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: state what ErrNoRoute may change, and group the distributed guides The seams note said MarkUnhealthy was the only state change allowed on ErrNoRoute. A pending backend op still records the failed attempt, counts toward the reconciler's retry limit and is dead-lettered after the maximum attempts. The note now says that MarkUnhealthy is the only change to the node's own state, and that the per-op accounting is not a verdict about the node. The note also documents that the NATS conformance run fails under CI when Docker is missing. The distributed-seams row moves next to the distributed-state row in the topics table. The liveness ping spec header now says no route is a reason to skip the worker, not proof that the worker is gone. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): give the backend client factory the node id Mechanical: the method gains a nodeID parameter and the eight test fakes are updated. No behaviour change. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): drop the optional node-aware factory The node id is now in the main method, so the optional interface and its helper had no behaviour of their own. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): dial backend probes through the client factory Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): dial workers' file servers through a per-node dialer Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(http): proxy backend logs through the per-node worker dialer The admin backend-logs proxy (list, lines and the WebSocket stream) now reaches a worker through the same per-node dialer as the HTTP file stager, so every frontend-to-worker dial goes through one seam. The shared direct dialer keeps alive for 15s where the proxy used 30s. Harmless for requests bounded at 15s. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(http): keep the backend-logs proxy independent of the admin connection The proxy request had no context before the dialer change and is bounded only by its 15s timeout. Keep it that way. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: move the worker control payloads to workerctl Mechanical move of the request and reply structs, the install progress event and the file payloads out of messaging. The verbs no longer belong to one carrier. No alias is left behind. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(worker): serve the lifecycle verbs through a controlServer The worker registers one handler per verb and a NATS server maps each verb to its subject. Registration errors now name the verb. node.stop is served with SubscribeReply, which is identical on the wire because the handler never replies. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(worker): report install progress through the control sink Install and upgrade now emit download progress through the sink the control server hands them. The debounce and the terminal flush stay in the handler path, built over that sink by the new nodes.NewDebouncedInstallProgressSink, which replaces NewDebouncedInstallProgressPublisher. The subject and payload on the wire are unchanged. The supervisor no longer holds the bus, and installFn and upgradeFn let specs drive both verbs without a gallery. The malformed-request log lines are restored for install, upgrade, backend.delete, model.unload, model.stop and model.delete, with the reply bytes unchanged. The signal adapter is renamed noReply, which also lets worker.go import os/signal without an alias again. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(worker): serve the file-staging verbs through a controlServer An empty list-dir answer is now {} rather than {"files":null}, because the typed reply omits an empty Files slice. The frontend decodes both to a nil slice in nodes/file_stager_s3.go. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(messaging): add WorkQueue and the NATS producer This is the producer side of the competing-consumer seam. The work kinds map one to one to today's subjects and queue groups: task to jobs.new and mcp-ci to jobs.mcp-ci.new (both in group workers), agent-run to agent.execute (group agent-workers). Enqueue publishes the payload as Publish does today, with one JSON marshal. FakeBus now records queue groups and keeps reply handlers so later specs can pin and drive them. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(messaging): add the NATS WorkConsumer An in-flight limit of one runs the handler inline on the delivery goroutine, as the MCP CI consumer does today. Any other limit spawns per delivery, as the agent consumer does. Queue groups are unchanged. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: publish queued work through WorkQueue The job dispatcher, the agent pool and the agent scheduler enqueue through messaging.WorkQueue; the NATS implementation publishes to the same subjects as before. DistributedServices builds the queue next to the NATS client and hands it to the dispatcher and the agent pool, whose distributed mode switch now reads a non-nil WorkQueue. The unused AgentPoolService.SetNATSClient is removed. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: consume queued work through WorkConsumer The agent dispatcher and the MCP CI consumer register through messaging.WorkConsumer. The NATS implementation keeps the inline one-at-a-time model for MCP CI and the per-delivery model for agent runs. handleMCPCIJob reports on the events publisher the carrier hands it instead of a captured client. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: delete the consumers nothing in production reached jobs.new has a producer and no production consumer, and the agent dispatcher's Dispatch was only called from tests. Publishing jobs.new is unchanged. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(mcp): send MCP requests to agent workers through AgentControl Timeouts still honour only the deadline, not cancellation, exactly as today. The NATS no-responders error maps to ErrNoRoute and a timeout does not. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(agent-worker): serve MCP requests and backend.stop through agentRPCServer The agent worker's MCP tool and discovery reply subscriptions and its backend stop listener move behind an unexported agentRPCServer interface, served on NATS by nodes.NATSAgentRPCServer. The handlers become typed mcp.ToolHandler and mcp.DiscoveryHandler values that answer every failure with a reply carrying Error. Queue group (agent-workers), inline execution on the delivery goroutine, the background handler context, the unmarshal error reply texts and the reply-less backend stop subscription are unchanged. The backend stop handler takes the decoded backend name, so it can still close that backend's MCP sessions. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(messaging): remove helpers that only tests used BroadcastRoots, ControlRoots and SubjectRoot had no production caller. The roots spec now asserts every served root through ValidateSubject instead. MatchSubject moves back into the test support package, the only place that used it, with its table. NATSAgentRPCServer drops the subscription list it stored and never read, and NewNATSAgentRPCServer gets a doc comment. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(mcp): round trip the agent RPC server over a real NATS server One spec sends a tool request and a discovery request through NATSAgentControl to NATSAgentRPCServer and checks that the handlers see the decoded requests and the replies come back. It also puts an undecodable body on the tool subject and checks the server answers with an unmarshal error instead of leaving the requester to time out. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: describe the distributed transport seams The developer note now lists the final seams: fan-out, queues, both halves of the control verbs and of agent RPC, and the dial. It records the open items a second carrier has to handle. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test: pin the in-flight limit each queue consumer asks for The work queue specs pin what Consume does for a given limit, but nothing pinned which limit each production consumer passes. Changing the agent worker's MCP CI limit from 1 to 0 would have let MCP CI jobs run concurrently on each worker with every test green. Move the MCP CI Consume call into startMCPCIConsumer with the same wiring and pin that it asks for (WorkMCPCI, 1). Pin that NATSDispatcher.Start asks for (WorkAgentRun, maxConcurrent) for several limits. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor: remove helpers the branch left without a caller SubjectJobCancelWildcard lost its last subscriber when the frontend stopped listening on jobs.*.cancel; the NATS permissions and conformance suite spell the subject out, so nothing reads the constant. decodeBackendStopRequest returned a stopAll flag that production dropped and only a test read. decodeBackendStop is now the single decoder with the same semantics: an empty body is stop-all, an empty Backend is stop-all, malformed JSON is an error. stopBackends still derives stop-all from Backend, so no reply changes. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(messaging): keep an explicitly empty agent queue a plain subscription Before the work queue seam the agent worker passed LOCALAI_AGENT_QUEUE straight to QueueSubscribe, so an explicitly empty value made a plain subscription and every agent worker ran every agent run. WithAgentRunRoute replaced an empty queue with agent-workers, which silently changed that. Keep the queue as given once the option is applied. An empty subject still falls back to agent.execute, since it never had a meaning of its own. The flag default stays agent-workers, so only an explicitly empty value reaches this. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: correct comments and record the PR B notes Fix the recordingFactory comment (it also records the parallel flag), document that a negative maxInFlight is unbounded and that Unsubscribe from a handler deadlocks, and say a permanently undecodable payload returns nil. Record controlHandler's undecodable return as a kept exception, and add the second carrier notes to the developer note: the reconciler has no ClientFactory option, the logs proxy honours HTTP_PROXY, verbs one carrier serves need an opt-out, terminal replies come from the result event, and agent runs publish through the NATS-bound EventBridge, which is not an additive change. Assisted-by: Claude:claude-sonnet-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-10-02 23:58:41 +02:00
Ettore Di Giacinto	9e95ef0dbd	fix: resolve gosec alerts in files this change touches ... Code scanning reports alerts on every line of a touched file. Mark the gRPC auth env var name and the mock backend's staged-path reads as reviewed, and log the error when evicting a model after its connection fails instead of dropping it. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-09-27 08:42:12 +00:00
Ettore Di Giacinto	5591687d9c	fix(e2e): check the realtime-failover websocket close result ... errcheck flagged tests/e2e/realtime_ws_test.go:307 as new on this branch (the localai-proxy realtime failover spec) once the e2e lint scope widened. Same house style the maintainer bot used for the other close results in this file (19f6a18b8): wrap in a deferred closure so the error is explicitly discarded rather than silently ignored. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-09-27 07:42:21 +00:00
Ettore Di Giacinto	55f7d5bfdf	fix(e2e): wire proxy api_key_env lookup into the e2e in-process app ... 19f6a18b8 moved credential-env resolution behind an explicit lookup (ApplicationConfig.ProxyAPIKeyEnvLookup / config.WithProxyAPIKeyEnvLookup), wired only at the CLI boundary (core/cli/run.go). The e2e suite builds its Application in-process without that option, so the failover prober could never resolve a remote target's api_key_env, remote liveness never passed, and "fails over ... and fails back" hung waiting for chain-remote to recover. Pass config.WithProxyAPIKeyEnvLookup(os.Getenv) there too, same as the CLI. worker/federated commands don't serve proxy/failover configs and tests/e2e-ui never sets api_key_env, so neither needs the lookup. Also make the misconfiguration itself easier to diagnose: the prober now logs a one-time xlog.Warn per api_key_env when a remote target sets it but no lookup is configured, instead of only surfacing it as a per-probe "is unset" error indistinguishable from a genuinely empty env var. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-09-27 07:42:21 +00:00
Ettore Di Giacinto	4a2a6180a2	test(localai-proxy): proxy APIs and realtime stages end to end ... The e2e suite now registers the localai-proxy binary and points proxy models back at the test server itself, so a request leaves LocalAI through the backend, returns over REST and is answered by a mock model. Chat, embeddings, TTS and transcription through the proxy return the upstream model's answer; a chain whose proxy target's upstream model fails to load serves from the local target; and a realtime pipeline whose LLM stage is a chain on a remote target completes a turn, then switches to the local target with a localai.model.failover trip event when a gate in front of the upstream starts answering 503. The docs describe the localai-proxy backend next to cloud-proxy and add a per-stage remote LocalAI example to the failover page. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-09-27 07:42:21 +00:00
localai-org-maint-bot	e62854c340	fix(failover): pass credential lookup from CLI ... Pass the API key environment lookup through ApplicationConfig to satisfy core configuration lint. Keep credential resolution dynamic and exclude the callback from serialization. Handle the five close results reported by errcheck. Assisted-by: Codex:gpt-6 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-09-27 07:42:20 +00:00
Ettore Di Giacinto	8dbe7a0e3f	chore(e2e): ignore the cloud-proxy backend binary ... make build-cloud-proxy-backend writes it next to the mock backend, which is ignored; the cloud-proxy binary showed up as untracked. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-09-27 07:42:20 +00:00
Ettore Di Giacinto	2ecbaab010	fix(failover): send remote targets their own upstream model ... A chain request reached a cloud-proxy target with the client's model, the chain name, whenever the target set no upstream_model: passthrough forwards the body's model and translate falls back to it. The upstream answered 404, which neither retries nor trips, while the liveness probe, which checks the target's own name, kept passing. PrepareTarget now sets the upstream model of a remote target to proxy.upstream_model or the target name, the same name the probe uses. The request pipeline and realtime chain stages both call it. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-09-27 07:42:20 +00:00
Ettore Di Giacinto	ea55e2ffc3	feat(failover): switch realtime pipeline stages per call ... A stage that names a chain is resolved on every call, so a switch keeps the session and its conversation. Clients get localai.model.failover events at session start and on every switch. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-09-27 07:42:20 +00:00
Ettore Di Giacinto	3e8ca016ee	fix(failover): stop judging cold local targets by their model file ... A missing file is not a reliable signal: models download on first use, some backends need no file, and dotted names like Phi-3.5-mini look like paths. Marking such a fallback down removed the retry a chain exists for. Cold targets are now judged only by real requests. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-09-27 07:42:20 +00:00
Ettore Di Giacinto	74775a820b	test(failover): cover retry per endpoint family and remote fail-back ... Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-09-27 07:42:20 +00:00
3 people	636c43baa5	fix(openai): return HTTP errors for pre-stream failures and report per-slot context (#12204) ... * fix(capabilities): report the per-request context with split KV slots With parallel slots and kv_unified:false, llama.cpp gives each slot n_ctx/n_parallel, padded up to a multiple of 256. /models/capabilities still reported the full n_ctx. A client that budgets a request against context_size then overflows at a fraction of it. EffectiveRequestContextSize returns the per-slot size in that case and the full context otherwise. With the unified KV cache, the grpc-server default, one request may use all of n_ctx. The capabilities endpoint and the router's prompt trimmer now use it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(openai): return an HTTP error when a stream fails before any chunk A streamed chat request set the SSE headers, then waited for the backend. When the backend failed before the first token, LocalAI sent a 200 with a `data: {"error":...}` chunk and [DONE]. Clients that do not parse error chunks saw an empty reply. cogito's LocalAI client was one of them: nib users got "streaming decision produced no content" instead of the context overflow that caused it. Nothing has been written at that point, so the handler now returns the error as a normal HTTP response. A failure after the first chunk keeps the in-stream error chunk. A prompt that exceeds the context is now a 400 on both paths, as in the OpenAI API and llama-server, and no longer a 500. The message is kept whole, because clients read the token counts from it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): check the error from closing the response body golangci-lint's errcheck flags the unchecked resp.Body.Close in the new pre-stream error helper. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>	2026-09-22 17:19:58 +02:00
Adira Denis Muhando	869000ceb7	fix(distributed): exclude pinned models from cluster eviction and idle scale-down ... pinned: true was only honoured by the per-node watchdog. Every distributed eviction path was pinned-blind: the router's LRU eviction (EvictLRU, evictLRUAndFreeNode) and the replica reconciler's idle scale-down would happily unload a pinned model — and since eviction is gated on in_flight = 0, a pinned model became eviction-eligible the instant each response completed. Under capacity pressure that surfaces as the backend being freed immediately after every request (#11101). Wire the model config loader into the router and reconciler through a new PinnedModelResolver seam (mirroring ConcurrencyConflictResolver): - EvictLRU passes the pinned set into FindLRUModel's query so the next-oldest unpinned model is selected instead of the attempt failing - evictLRUAndFreeNode filters pinned models inside its locked selection - scaleDownIdle skips pinned models entirely: trimming to the floor still means requests beyond the survivor's capacity pay a cold reload Deliberate teardown (admin unload, model delete, node drain) intentionally still applies to pinned models, as does dead-row reaping (state correction, not eviction). Regression specs verified to fail with the exclusion disabled. Addresses the cluster-side eviction gap in #11101 Assisted-by: Claude Code:claude-fable-5 [Claude Code] Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com>	2026-09-11 23:08:26 +00:00
Abdullah Mansour | عبد الله منصور	acc7284526	fix(openresponses): support Codex WebSocket warm-up (#11608) ... * fix(openresponses): support Codex WebSocket warmup Assisted-by: ChatGPT:GPT-5.6-Sol golangci-lint Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com> * docs(openresponses): document WebSocket responses Assisted-by: Codex:GPT-5.6-Sol gh Docker golangci-lint Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com> * fix(openresponses): harden WebSocket response lifecycle Ensure response ownership, continuation storage, error sequencing, and connection-local resource limits remain correct across HTTP and WebSocket transports. Assisted-by: Codex:GPT-5.6-Sol [gh] [Docker] [golangci-lint] Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com> --------- Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com>	2026-09-11 16:54:42 +02:00
localai-org-maint-botandEttore Di Giacinto	9afe10ba21	fix(distributed): survive a slow control-plane database (#11837) ... * fix(distributed): evict only when a node is known to be full scheduleNewModel asked the registry for a free replica slot and treated every error as "this node is full", so a control-plane database slow enough to time out the lookup evicted a healthy loaded model. The evicted process died, a peer frontend still holding its address dialled the dead port and retried, and the model thrashed between nodes. The comment on the branch already said it meant a full node; the code never tested for it. Evict only on ErrNoFreeSlot. Any other error now returns and names the lookup that failed, so a slow database degrades into a diagnosable load failure instead of into lost work. An audit of the rest of the router found one branch of the same shape: node selection discarded the error from its last-resort finder, so a database timeout there also produced a nil node and evicted for it. That path now returns unless the finder said gorm.ErrRecordNotFound, which is the only answer that means the cluster had no node to give. No other destructive branch in router.go fires on a generic error. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): checkpoint heartbeat writes instead of writing every beat Every heartbeat UPDATEd backend_nodes. Six nodes at a ten second beat is roughly 52,000 writes a day against a six-row table, and that churn is what turned a blocked autovacuum into a 460 MB table whose six-row scan cost 867 ms and timed out the queries that place models. A beat carrying only a fresher timestamp now waits for the checkpoint interval. Each reported field is compared against the value last persisted rather than tested for presence, because a worker sends its disk figures on every beat and presence alone would suppress nothing. A node's first beat, a changed total VRAM, total disk or GPU vendor, and a free VRAM, RAM or disk reading that has moved more than 256 MiB from the persisted value all still write at once. A node that is not active is never suppressed, because it recovers only when the health monitor sees a fresh timestamp. The persisted column is up to one interval stale by design, so the stale-node threshold moves from 60s to 5m to cover it. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): fail worker readiness when a held backend is unreachable The readiness gate tracked only the NATS link, so a worker whose backend processes had died still answered /readyz with 200 and kept receiving loads. One node did exactly that during an incident: it reported healthy while its backend port refused connections, and every load routed to it failed. Readiness is now the NATS link and, for each backend process the worker believes it is running, a short dial of its recorded address. A worker holding no backends stays ready, because idle is a healthy state. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): keep a starting backend out of the readiness dial set A backend process is inserted into the supervisor map with its gRPC address already recorded, but the address refuses connections until the gRPC server binds, which the startup poll allows up to 30 seconds for and which takes 10 to 15 seconds on a slow node. The new data-path readiness probe dialled that address straight away, so a worker answered /readyz with 503 for the whole of every cold backend start. The container HEALTHCHECK absorbs that, but a Kubernetes readinessProbe at 10s does not, and the worker would leave rotation each time it loaded a model. The skip for a stopping process had no counterpart at the other end of the lifecycle. Backend processes now carry a serving flag, set where the startup health-check gate succeeds, and the probe dials only processes that are serving and not yet stopping. backendStartStillValid becomes markBackendServing: the check and the mark must share one lock hold, so the flag can only ever land on the entry the key currently owns. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): export control-plane database health gauges Four transactions wedged on a corrupt index held the vacuum horizon open for 42 days. Nothing measured it, so the first symptom anyone saw was models failing to load six weeks later, by which time a six-row table had grown to 460 MB. Export the oldest xmin age, the longest open transaction, and the dead tuple ratio on the registry tables. The first is the number that would have caught it: it sits near zero in health and was 21,002,291. Sampling is scrape-driven behind a cache, and a failed sample reports the last good values rather than failing the scrape, because these gauges matter most when the database is already struggling. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): rate-limit failed control-plane database samples The cache advanced its clock only on a successful sample, so once the database started failing every scrape retried the query immediately. That turned the cache off in the one regime it exists for: a retry storm at scrape cadence aimed at a database already in trouble. A catalog read that consistently exceeds the 5 second timeout also paid that cost on every scrape, with all scrapes serialised behind the sampler mutex. Time every attempt rather than every success, so failures and timeouts cost the same interval as good samples. Whether a good sample exists moves to its own field, keeping the gauges absent until the first success and holding the last good values through later failures. Also note in the runbook that pg_stat_activity cannot see prepared transactions or replication slot xmins, so a healthy-looking xmin age does not by itself rule out a blocked horizon. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(distributed): pin that a failing database evicts nothing Exercises the real distributed stack against a control-plane database that refuses the router's slot lookup, and asserts the scheduler reports the lookup it could not answer instead of falling through to eviction. The failure is injected with privileges rather than a statement timeout. A timeout set with ALTER DATABASE also breaks AutoMigrate, and it leaks into every later spec in the suite unless it is reset, so the spec would end up testing the migration rather than the scheduler. Instead the spec creates a dedicated login role, points a second gorm handle at it, and revokes that role's SELECT on node_models.replica_index. This has to be a separate role: the test container's owner is a PostgreSQL superuser, and superusers bypass every privilege check, so revoking from CURRENT_USER is recorded and then ignored. The revoke is scoped to one column on purpose. Revoking the whole table would also blind node selection, which runs first and has a guard of its own, so the scheduler would never reach the slot lookup this spec is about. Leaving every other column readable lets selection succeed and lands the refusal exactly on NextFreeReplicaIndex, which plucks replica_index. The grant is restored from BeforeEach via DeferCleanup, so a failing assertion or a panic cannot hand the next spec a role that cannot read. Reverting the eviction guard fails this spec, which is the point of it: the router then reports "no replica slot on keeper and eviction failed" for an error that was never evidence the node was full. The surviving-row assertions are secondary under this injection, because the eviction path reads whole node_models rows and the same revoke blinds it too; a comment in the spec says so, so nobody mistakes them for the load-bearing ones. Also documents why the vector store and the control plane must not share a database: the removable-tuple cutoff is per database, not per table, so one transaction left open anywhere stops autovacuum reclaiming the node registry, and a six-row table bloats into hundreds of megabytes. The note names LOCALAI_AUTH_DATABASE_URL and LOCALAI_AGENT_POOL_DATABASE_URL as the two knobs that must differ, and the localai_control_plane_oldest_xmin_age gauge as the way to see it coming. grep for StaleNodeThreshold and HealthCheckInterval in core/config/runtime_settings_registry.go returns no matches: the distributed duration knobs are not exposed as runtime settings, so the new heartbeat checkpoint interval follows them and needs no registry entry. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): close the review gaps in the heartbeat and health path The stale-node threshold moved from 60 seconds to 5 minutes in this branch because checkpointing makes last_heartbeat up to one checkpoint interval behind by design. Two things were left inconsistent with that. NewHealthMonitor still fell back to a hardcoded 60 seconds when handed a zero threshold, so any future caller that stopped passing the configured value would mark every healthy, beating node offline on every cycle. And the threshold itself had a flag-name constant but no AppOption, no CLI field and no env binding, so an operator who widened --node-heartbeat-checkpoint had no way to widen the threshold to match. The fallback now tracks config.DefaultStaleNodeThreshold, and --stale-node-threshold / LOCALAI_STALE_NODE_THRESHOLD is wired the same way its sibling is. Heartbeat suppression compared the RAW reported free VRAM against the snapshot, but the column persists capAvailable(raw, ceiling). On any node with a VRAM budget set, whose actual free VRAM oscillates above that ceiling, every beat looked material while the persisted value never moved: suppression was defeated on exactly the nodes an operator had configured, and the write amplification this branch exists to remove came straight back there. The comparison and the snapshot now both hold the capped figure, so they measure the same quantity as the column. Fixing that needs the ceiling, and reading it cost a SELECT on every beat, including suppressed ones. The skip decision therefore moved ahead of the updates map and now reuses the ceiling cached on the last durable write, while the write path still re-reads it before capping anything. A ceiling that changed inside the checkpoint window can cost one extra or one late write; it cannot persist a wrong figure. A suppressed beat now costs no query at all. Also: the operations section now says to grant pg_read_all_stats to the LocalAI role, because PostgreSQL blanks backend_xmin and xact_start for sessions owned by other roles, and the transaction that wedged the horizon in the incident was a co-located vector store connecting as a different role, so without the grant the new gauge sees only our own sessions. The compose healthcheck comment now describes readiness covering the backend data path, and the control-plane gauge registration records the otel.SetMeterProvider ordering it depends on. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): resolve the gauge's table names through gorm The dead-tuple gauge queried pg_stat_user_tables against a hardcoded list of three table names. Those three do not agree on where their name comes from: BackendNode and NodeModel take gorm's default pluralisation, while GalleryOperationRecord overrides TableName, and gallery_operations already had a constant of its own that the list duplicated. A literal list keeps compiling after any of that moves, and the query then matches nothing. The failure is silent and it points the wrong way: a dead-tuple ratio that matched no rows reports the same numbers as a cluster with no bloat, so the gauge would look healthiest exactly when it had stopped working. Ask gorm what each model is stored as instead, which follows a TableName override and the default pluralisation alike. A spec pins that the override really is consulted: naive pluralisation of the type would give gallery_operation_records, so the resolution cannot quietly stop asking the model. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-09-02 12:37:12 +02:00
localai-org-maint-botandlocalai-org-maint-bot	7ff9d9942b	fix(distributed): restore node liveness tests (#11694) ... * fix(distributed): restore node liveness tests The router now probes models.running before it schedules work. The E2E workers only mocked backend.install, so every test node appeared offline. The endpoint test double also missed the new PingNode method and stopped the Linux, Apple, and lint jobs during compilation. Mock the existing worker reply in both distributed fixtures and keep the endpoint test double aligned with NodeCommandSender. Assisted-by: Codex:gpt-5 [golangci-lint] * fix(tests): check node liveness replies The liveness test subscriptions ignored setup and reply errors. Errcheck rejected each branch that carried them. Assisted-by: Codex:gpt-5 [golangci-lint] --------- Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>	2026-08-24 09:32:48 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto	82c191afad	fix(distributed): keep model replicas config-consistent (#11664) ... * docs: design configurable copy buffering Document the context-aware copy buffer option and its validation plan. Assisted-by: Codex:gpt-5 * docs: design durable distributed staging operations Assisted-by: Codex:gpt-5 * docs: design distributed model config revisions Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] * feat(config): add stable model revisions Hash typed model configuration and effective protobuf options deterministically for distributed revision comparisons. Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] * feat(worker): acknowledge exact model stops Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] * feat(nodes): track model config revisions Assisted-by: Codex:GPT-5 [apply_patch] * fix(distributed): retry quarantined model cleanup Stop quarantined replicas by exact process identity, retain failed cleanup as durable capped retries, and compare-and-delete only the claimed registry row. Process one sufficiently leased row at a time so multiple frontends cannot duplicate slow cleanup work. Assisted-by: Codex:gpt-5 * fix(distributed): bind loads to config revisions Assisted-by: Codex: GPT-5 [OpenAI Codex] * fix(modeladmin): apply config revisions consistently Route model edits, patches, state changes, deletion, and peer refreshes through the same revision lifecycle. Quarantine stale replicas before exact cleanup and report durable pending cleanup without failing successful config writes. Assisted-by: Codex: GPT-5 [OpenAI Codex] * feat(distributed): expose model config revision state Document replica revision observability and durable cleanup behavior. Keep pending cleanup explicit in model mutation responses and verify endpoint contracts expose revision state without serialized load options. Assisted-by: Codex:GPT-5 [OpenAI Codex] * test(distributed): cover model revision convergence Exercise cross-frontend quarantine, stale replay rejection, exact cleanup retry, worker re-registration, and current-generation replica convergence against the distributed PostgreSQL harness. Assisted-by: Codex:gpt-5 * fix(distributed): pass config revision CI checks Keep configured gallery sources out of authoritative runtime snapshots only after validating their real schema, and harden rollback snapshots against symlink races and non-regular files. Assisted-by: Codex: GPT-5 [OpenAI Codex] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-08-22 22:44:03 +02:00
Ettore Di Giacinto	2383726d6d	Revert "chore(tests): Avoid network, sleep and more during tests" (#11601) ... Revert "chore(tests): Avoid network, sleep and more during tests (#11050)" This reverts commit cb3bf7af3f.	2026-08-19 16:39:39 +02:00
Richard Palethorpeandlocalai-org-maint-bot	cb3bf7af3f	chore(tests): Avoid network, sleep and more during tests (#11050) ... * test: make coverage failures observable Keep per-root logs, reject concurrent coverage runs, and avoid relying on /bin/sleep in the worker timeout test. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: parallelize coverage without remote fixtures Assisted-by: Codex:gpt-5 [apply_patch] [exec_command] Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: add offline resource infrastructure Introduce versioned resource manifests, a checksum-verified CAS preparer, offline test wrappers, and a guarded network transport. Replace live Hugging Face, GitHub, and OCI cases with deterministic fixtures and inject fixture metadata into importer discovery. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: enforce offline resource replay Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: harden offline resource refresh Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: expose slow coverage waits Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: eliminate avoidable wall-clock waits Inject a clock into Hugging Face retry handling, reuse a process-scoped PostgreSQL container with per-spec schemas in the nodes suite, and poll local import jobs promptly. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: remove repeated fixture startup waits Share PostgreSQL fixtures across parallel endpoint and agent suite workers, and make the worker Free deadline injectable so the wedged-backend test does not spend five seconds on wall-clock time. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: fix offline resource CI portability Normalize Docker archive metadata before content addressing, derive archive checksums during explicit refreshes, make network lint portable to macOS, and prepare distributed images before running their offline suite. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: cache Go modules before offline tests Warm the complete module graph before the Linux and macOS test jobs enter offline replay mode, so tool dependencies such as Ginkgo are not fetched through the guarded proxy. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: drop the static network lint in favour of real isolation The offline test suite already prevents tests from reaching the network twice over: run-test-linux-offline.sh puts the test process in a cgroup and REJECTs egress outside the private ranges, and HardenedTransport installs testnetwork.LocalGuard to refuse dials that resolve to a public address. Both fail the test with a precise error at the moment of the dial. test-network-lint.sh added neither. Its diff stage defaulted to a HEAD base, so on a clean checkout it compared the tree against itself and inspected nothing; the branch's own commits were never examined. It only produced output when an earlier job step dirtied the tree, and then it matched a bare https?:// against whatever changed. make react-ui runs npm install rather than npm ci, so CI rewrote core/http/react-ui/package-lock.json and the lint reported an npm registry URL as forbidden test network access: + "resolved": "https://registry.npmjs.org/hono/-/hono-4.12.25.tgz", Its fingerprint stage was self-defeating in a quieter way: hashing the whole tree's network-mechanism inventory meant every rebase onto a master that touched any _test.go needed a manual baseline bump, so the check mostly caught its own staleness. Remove the script, its make target and the two prerequisite edges, along with the test-network: fixture markers that existed only to suppress it. The isolation itself is untouched. Assisted-by: Claude:claude-opus-5 [go vet] Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: keep hidden files in the offline test bundle artifact Cherry-picked from 15a37b0ac on the remote branch. The offline bundle lives under .cache/, which actions/upload-artifact skips by default, so the Linux job packed an artifact missing the very file the next step restores. The other half of 15a37b0ac moved test-network-lint out of the `test` and `test-coverage` prerequisite lists into a recipe line, so parallel make could not fingerprint the tree while generated fixtures were still changing. That is dropped: the preceding commit removes the lint entirely, and the race it worked around is one more reason a whole-tree fingerprint was the wrong mechanism. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * refactor: share bounded exponential backoff Use overflow-safe saturating arithmetic for retry delays across model import polling, downloads, registration, node operations, and model loading. Keep model import status checks responsive initially while capping their interval at 500ms. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: mirror Jetson Python wheels Keep the CUDA aarch64 wheel subset in GHCR and serve it as a local PEP 503 index during L4T backend builds, preserving last-known-good packages through upstream outages. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * docs(agents): index the Jetson wheels mirror Mention the GHCR-hosted L4T wheel mirror in the CI caching guide summary so maintainers can find its outage and cache documentation. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: add defensive build network proxy Record build destinations and byte counts, retry observable idempotent HTTP downloads, and isolate explorer database tests that race under coverage. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(kokoros): implement updated backend trait Return unimplemented for image upscaling, matching the backend's other unsupported modalities after the protobuf API update. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): clear recovered proxy errors Do not mark a request failed when a later safe retry succeeds. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: require HTTPS build interception Inject a short-lived proxy CA into BuildKit and Dockerfile RUN steps, reject plain HTTP and opaque tunnels, and retain method/status/byte telemetry for verified HTTPS traffic. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): preserve system trust in unproxied builds Mount the generated interception CA at a dedicated secret path and add it to the trust bundle only in proxy-aware dependency stages. This prevents optional secret mounts from masking the system CA bundle in ordinary backend test builds. Install the requested Go toolchain before starting the proxy and satisfy cleanup error checks found by CI lint. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): persist build proxy trust Install the generated proxy CA through the system-managed local certificate directory so ca-certificates upgrades retain it. Avoid turning canceled matrix jobs into proxy cleanup failures. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): trust proxy in nested build scripts Install the build proxy CA before nested source fetches, route the DS4 package setup through the HTTPS mirror helper, and avoid repeated OCI setup in gallery behavior tests. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): use HTTPS apt sources for Bonsai Rewrite ARM64 package sources before installing GCC and check gallery fixture cleanup errors so the optimized tests satisfy errcheck. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(privacy-filter): trust build proxy CA Install the mounted build proxy certificate before privacy-filter's make target fetches its HTTPS sources, for both source and prebuilt builder paths.\n\nAssisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * test: fail on hidden offline egress Count cgroup-scoped firewall rejects and fail the offline test harness with bounded aggregate diagnostics. Inject the gen-audio GGUF probe so fixture-backed importer tests do not attempt real network access. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): preserve system CA trust Build a combined runner certificate bundle instead of replacing public roots with the generated proxy CA. Centralize additive container installation in the shared proxy CA helper. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com> Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com>	2026-08-19 10:59:31 +02:00
Richard Palethorpe	d10374f849	feat(router): make KNN a first-class classifier with a persisted, curated corpus (#10652) ... * feat(router): make KNN a first-class classifier with a persisted, curated corpus Add `classifier: knn` — similarity-weighted voting over labelled example prompts. Unlike score/colbert it needs no classifier model: label knowledge lives in a corpus seeded and curated through the admin API, so routing decisions are deterministic, auditable, and grounded in graded experience rather than a model's opinion. Epistemic gate: corpus entries below knn.similarity_threshold cannot vote; when none clears it the classifier activates no labels and the router uses the fallback — a prompt unlike all labelled experience is treated as undecidable, not guessed. Decisions record nearest_similarity (also on fallback rows) so admins can see how far the nearest labelled experience was; the Routing tab explains out-of-corpus fallbacks and shows per-label corpus counts. Persistence: one JSONL file per router under <data path>/router-corpus (text, labels, vector, embedder fingerprint). The file is the source of truth; the local-store index is rebuilt from it at classifier build time and stays a pure in-memory index. Entries recorded under a different embedding model re-embed on load. Also corrects the docs' false claim that local-store collections persist — the embedding cache never survived restarts (and still doesn't); the corpus does. Corpus input is API-only by design (entries may contain example user content): POST /api/router/{name}/corpus seeds (labels validated against declared policies, embedded server-side, indexed immediately), GET .../corpus/stats inspects — label counts only, entry texts are never returned by any surface — DELETE .../corpus wipes. Admin-gated like the sibling router endpoints, and exposed as MCP tools (seed_router_corpus / get_router_corpus_stats / clear_router_corpus) in both the httpapi and inproc clients with coverage-test route mappings. Plumbing: VectorStore gains SearchK (top-K was hardcoded to 1); local-store gets InsertBatch/Delete as optional fast paths; RouterConfig gains a knn block (embedding_model, k, similarity_threshold, vote_threshold, store_name) with meta-registry fields; the classifier dropdown now offers knn and the previously-missing colbert; embedding_cache is ignored (with a warning) for knn — it IS an embedding-KNN lookup; the stale /api/instructions intelligent-routing entry is rewritten (it described a classifier that no longer exists); swagger regenerated. Tests: KNN vote/gate specs with hand-computed vote shares, corpus manager suite (restart reload without re-embedding, fingerprint re-embed, dedupe, hostile store names), middleware specs (corpus routing, gate fallback, config validation, cache-wrap refusal), corpus endpoint specs pinning the texts-never-returned contract, MCP catalog + route-mapping gates, and a Playwright spec for corpus stats and the out-of-corpus decision detail. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(router): name consulted corpus neighbours in knn decisions Every knn decision (decision log rows and the /api/router/decide response) now carries neighbors: the K retrieved corpus entries by descending similarity - including ones below the epistemic gate, which is what makes fallback decisions diagnosable - each as {id, similarity, labels}. The id is the entry's content hash (first 8 bytes of the SHA-256 of its text, hex): stable across reseeds and re-embeds, and text-free, so an external platform that seeded the corpus can recompute text->id on its own copy and bucket decisions by corpus region (per- region reliability accounting) without corpus text ever leaving the server. A corrupt index payload surfaces as an id-less neighbour at a real similarity instead of disappearing. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * refactor(router): deduplicate knn plumbing and cut corpus hot-path waste Post-review cleanup of the knn-first-class-router branch; no behaviour changes on the API surface. Reuse/altitude: - RouterKNNConfig.ResolvedStoreName is now the single source of the router-corpus-<name> default (was hand-derived in four files). - corpus.ResolveKNNRouter + corpus.Seed carry the shared model resolution and seed validation; the REST endpoints and the assistant MCP client are thin transport adapters over them, with sentinel errors mapped to HTTP statuses at the echo boundary. - middleware.NewClassifierDeps assembles the classifier dependency set once for all five entry points (OpenAI, Anthropic, realtime, decide, corpus) instead of five hand-copied literals. - router.AllClassifiers feeds both the status endpoint and the unknown-classifier error, ending the classifier-list drift. - Per-classifier requirements moved out of validateRouterPolicies into their buildClassifier arms; the knn arm owns its embedding_cache opt-out instead of a name-check in the shared wrap tail. - adminOnly replaces four inline copies of the admin gate in the middleware routes. - localVectorStore.Search delegates to SearchK (identical traces). Efficiency: - Manager.Add embeds outside the manager mutex and appends to the JSONL file (O(new) instead of O(corpus) rewrite); a torn tail from a crash mid-append is tolerated on read and repaired on next write. - Stats memoises per store keyed on the file's stat fingerprint and no longer takes the manager mutex, so the 5s status poll stops parsing vector-laden JSONL and stops blocking behind seeds. - KNN Classify decodes each neighbour payload once (was twice) and builds refs and votes in a single pass with one fallback return. - Corpus file writes fsync before rename/close. - The corpus manager is built eagerly in newApplication (sync.Once dropped); test helper dead branch removed. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(router): bind knn corpus vectors to an embedder fingerprint and fail closed on mismatch Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * chore(mcp): align corpus tool prompts and the mutating-tool safety list Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(proto,backend): report embedding shape from the llama-cpp backend Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(embeddings): Go-side pooling — mean/last/decayed_mean with half-life Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(embeddings): accept chat messages[] and per-request pooling on /v1/embeddings Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * chore(middleware): name the failing fields when post-merge validation 400s An intermittent post-merge validation failure surfaced as an opaque 400 during integration (pooling scheme mismatch that no client had sent). Log the model, the request's pooling override, and the merged config's pooling fields at the failure point so the next occurrence identifies whether the request or the stored config carried the bad value. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(embeddings): scheme override must not inherit the config's half-life A model config defaulting to decayed_mean pooling carries pooling_half_life_tokens; a request overriding the scheme to mean/last without its own half-life inherited that value, and post-merge validation rejected the pair the server itself had assembled. Zero the inherited half-life when the overridden scheme is not decayed_mean; a request that explicitly pairs a half-life with a non-decayed scheme still 400s. Assisted-by: Claude:claude-fable-5 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix embedding pooling validation and router bounds Declare backend embedding layouts and reject incompatible pooling modes. Reset local-store dimensions after a full clear, validate KNN thresholds, and add real backend and store integration coverage. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * ci: run local-store integration tests Build and install the local-store backend in the Linux test job, then run the existing store integration suite so new specs are discovered automatically. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-08-18 09:37:43 +02:00
Adiraandlocalai-org-maint-bot	ef724a3c9d	feat(api): add /v1/detokenize endpoint (#9620) ... * feat(api): add /v1/detokenize endpoint Closes #1649. Mirror of the existing /v1/tokenize path, requested by @benniekiss in the issue thread for "complete API workflow" use cases that need to turn token IDs back into text without local processing. - Add Detokenize gRPC RPC with DetokenizeRequest{tokens} / DetokenizeResponse{content} messages. - Implement in the llama.cpp backend using common_token_to_piece, the same primitive TokenizeString already uses internally. - Other backends inherit the default Unimplemented from base.Base, in line with how Detect, Rerank, etc. are gated per-backend. - Wire up the Go gRPC interface, server, client, and in-process embed wrapper alongside their TokenizeString counterparts. - Add the schema types, ModelDetokenize wrapper, HTTP handler, route registration, RouteFeatureRegistry entry (gated by FeatureTokenize so no new feature flag is needed), and the discovery map entry under ai_functions. - Regenerated swagger reflects the new endpoint and types. - Update authentication.md to list /v1/detokenize alongside /v1/tokenize. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> * test(e2e): add mock backend tests for /v1/detokenize Add Detokenize to the mock gRPC backend and wire up two e2e tests in the MockBackend suite: one that posts known token IDs and asserts a non-empty content response, and a round-trip that tokenizes first then detokenizes the returned IDs. Addresses reviewer feedback on #9620. Assisted-by: Claude:claude-sonnet-4-6 Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> * fix(kokoros): implement detokenize in the Rust backend service The Detokenize RPC added in this PR grows the tonic-generated Backend trait. Unlike the other languages there is nothing to inherit a default from — Rust trait impls must list every method — so backend/rust/kokoros failed to compile: error[E0046]: not all trait items implemented, missing: `detokenize` --> src/service.rs:72:1 72 | impl Backend for KokorosService { Go backends pick up the Unimplemented default from base.Base, and the generated C++/Python servicer bases default to UNIMPLEMENTED, which is why the Rust backend was the only one that broke. kokoros is the sole Rust crate in the tree, so this is the full extent of the fallout. Return Status::unimplemented("Not supported"), matching how this same file already gates tokenize_string and ~20 other unsupported RPCs. Fixes the tests-kokoros and backend-jobs-singlearch-4 (-cpu-kokoros) failures on the previous head. Assisted-by: Claude:claude-opus-5 cargo Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> --------- Signed-off-by: Adira Denis Muhando <dennisadira@gmail.com> Co-authored-by: localai-org-maint-bot <bot-opensource@localaisrl.com>	2026-07-30 16:01:47 +02:00
Richard Palethorpe	49ef40a187	feat(classifier/VAD): support voice control on low power devices (#10804) ... * feat(llama-cpp): route Score through the slot loop Score previously bypassed the slot loop with a direct llama_decode: a conflict guard aborted the whole process if scoring raced generation, the config validator had to reject score alongside chat/completion/embeddings, and every candidate re-decoded the full shared prompt. Add SERVER_TASK_TYPE_SCORE to the (patched) upstream server so score tasks are scheduled like any other slot work: generation and scoring serialize naturally, the shared prompt is decoded once per call, and the slot's prompt cache carries the conversation prefix across calls. Context checkpoints at the score boundary and at the cache-divergence point keep SWA/hybrid/recurrent models (e.g. LFM2.5) from re-prefilling the whole prompt per candidate: warm-turn scoring on a 6-option set drops from ~8s to ~0.5s on a desktop CPU. The conflict guard and the validation split are removed; declaring score with generation usecases on one config is now supported and shares the slot cache. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): classifier wire types and pipeline config Wire types and YAML config for realtime classifier mode: sessions carry a localai_classifier extension (options with canned replies/tool calls, softmax threshold, normalization, history trimming, fallback modes, and a deterministic wake-word address gate), mirrored by pipeline.classifier in the model YAML and surfaced in the config-meta registry. The localai.classifier.result server event reports the full score distribution per turn. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): classifier response flow Classifier-mode responses: instead of autoregressive generation, each user turn is prefill-scored against the option list (router.ScoreClassifier prompt/candidate shapes over the Score primitive) and the winning option's canned reply and tool call are emitted through the existing response machinery. Below-threshold turns take the configured fallback (none / canned reply / generate); empty transcripts and unaddressed turns (wake word not mentioned) skip scoring entirely. The scoring probe defaults to the latest user message only — small scorers echo canned replies from prior turns back as the top option otherwise. Built for hardware that can afford prompt processing but not decode: with slot-based Score the option list stays KV-cached across turns, so a turn costs roughly one forward pass over the new words. session_update_error events now carry the validation cause instead of a generic message. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): bound the VAD tick's scan window and buffer retention The VAD tick loop re-scanned the entire input buffer every 300ms and only trimmed it on zero-segment ticks or commits. Audio that keeps producing segments without a committing pause (steady noise a mic pipeline lets through, music, continuous speech) grew the buffer toward the 100MB cap with each tick rescanning all of it — O(n^2), measured at ~3.3ms of silero per buffered second: past ~90s retained, ticks run back to back and pin ~4 cores until the stream stops. Silero's recurrent state only carries a few hundred ms of context, so rescanning old audio buys nothing. Clip the slice handed to the VAD to the largest silence the commit test can need to measure (server_vad silence window or the semantic eagerness fallback) plus a warm-up margin, and rebase the returned segment times so every downstream consumer keeps whole-buffer coordinates. An open turn whose clipped window is all silence now commits (the silence outran the window) instead of being discarded as no-speech. Independently, retain at most 90s of raw buffer, rebasing the live-feed and EOU cursors on trim — this also bounds the previously unbounded VAD-error path. Turn boundaries are otherwise unchanged: no forced commits, no new coordinator states. pipeline.turn_detection.vad_window_sec can widen the scan window; values below the automatic floor are ignored. The tick body is extracted into vadTick so specs can drive turn detection synchronously (same shape as classifySoundWindow); the babble reproduction that pinned 4 cores now plateaus under 10% of one core. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(backend): let per-model threads override the global default ModelOptions overrode a set per-model threads value with the app-level --threads whenever the latter was non-zero — and WithThreads defaults it to the physical core count, so it always was. The YAML threads: knob has been dead config: a tiny VAD model could never opt down from the global pool size. SetDefaults already fills an unset per-model value from the app config, which is the intended precedence; resolve threads through a helper that honors it (explicit threads: 0 still means unset). Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * chore(gallery): single-thread the silero VAD Silero is a ~2MB recurrent model with no exploitable graph parallelism: measured per-call latency is identical at 1 and 10 ORT threads, while every extra pool thread just spin-waits between the realtime loop's frequent tiny inferences. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * docs(realtime): classifier mode, VAD scan window, threads precedence Document the realtime classifier mode (options, threshold guidance, wake-word address gate, empty-transcript handling), the VAD scan window and 90s buffer retention (pipeline.turn_detection.vad_window_sec), the per-model threads precedence, and the M3 classifier note in the realtime state-machine design doc. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * perf(llama-cpp): score all candidates in one batched decode One scoring call is now a single SERVER_TASK_TYPE_SCORE task: the slot decodes the shared prefix (prompt + longest common candidate token prefix) once, then forks one sequence per candidate off it (metadata-only for the unified KV cache, copy-on-write for recurrent state) and decodes every candidate's unique tail in one llama_decode. Previously each candidate was its own task that restored the boundary checkpoint and re-decoded its full tail sequentially, paying per-candidate task and decode overhead. The context reserves SERVER_SCORE_FORK_SEQS extra sequence ids (and recurrent-state cells) beyond the parallel slots via the new common_params::n_seq_score_forks. Forking requires the unified KV cache (already this backend's default) since per-sequence streams would shrink n_ctx_seq; an explicit kv_unified:false disables forking and Score calls that need it fail cleanly. Candidates beyond the fork/output budget decode in successive chunks. Wire contract and scores are unchanged: per-token logprobs are stitched from the shared region and the forked tails. Verified bitwise deterministic call-to-call and independent of candidate order (no cross-fork leakage via equal-length candidate swap); ranking matches the per-candidate implementation on the drone battery (winner softmax 0.99996 vs 0.99997), and >16-candidate chunking, prefix-of-another and empty candidates all pass. Measured on a desktop CPU: warm /api/score calls 0.52s -> 0.23s; warm realtime classifier turns 196-303ms. The 9-candidate drone turn decodes ~17 unique tail tokens in one batch instead of nine sequential ~220ms checkpoint-restore tasks. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): gate scoring capacity by model usecase Reserve llama.cpp scoring slots only for models that explicitly declare the score usecase, while allowing score to coexist with chat and completion. Reject incompatible unified-KV settings and classifier activation on models without scoring capacity. Propagate application defaults when resolving realtime and preload pipeline stages so unset thread counts are resolved consistently without overriding explicit model settings. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(ci): honor APT mirrors in the prebuilt llama-cpp compile step The builder-prebuilt path installs gcc-14 with apt directly and ignored the APT_MIRROR/APT_PORTS_MIRROR build args the from-source path already honors, so an ubuntu mirror outage broke every arm64 backend build. Pass the args into the stage and run apt-mirror.sh (already in the build context via COPY . /LocalAI) before the apt step. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): classifier argument slots via constrained completion Hybrid classify-then-complete: a classifier option's canned tool call can declare typed argument slots (number | enum | string, with defaults and prompt hints) referenced as "{{name}}" in the arguments template. When the option wins, the slots are filled by a short grammar-constrained completion that continues the exact scoring prompt — rendered by the same cached ScoreClassifier, so the llama.cpp prompt cache is already warm — with the chosen route JSON re-opened at the first slot field. A GBNF grammar pins the field skeleton and frees only the values; temperature 0, a couple dozen tokens at most (~300ms on a desktop CPU for two slots). Slot declarations and hints ride the option descriptions in the shared system prompt, informing scoring and the fill alike at no per-turn token cost. The localai.classifier.result event carries the final arguments and a fill_latency_ms. On inference failure the slots' defaults apply; a slot without a default fails the response (or falls through with fallback.mode: generate). Slot filling requires completion alongside score in the scoring model's known_usecases. Verified end-to-end on the Pi drone demo: "fly forward three meters" in distance mode classifies forward and infers {"distance": 3, "units": "meters"} in ~310ms, and the drone flies exactly 3 units. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): splice filled slot values into classifier replies A classifier option's spoken reply can now reference its tool's argument slots ("Going forward {{distance}} {{units}}."): the values inferred by the slot-fill completion — or the recovery defaults — are spliced into the reply as plain text before it is emitted, so what the assistant says confirms what it actually inferred. Placeholders without a value stay literal, and options without slots are untouched. FillToolArguments now returns the raw slot values alongside the spliced arguments JSON to make the reply templating possible. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): harden classifier slot completion Reserve context for constrained slot filling, size completions from their encoded output, and encode enum grammar literals as valid JSON. Reject empty enum values and cover the failure modes with regression tests. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): prewarm the classifier scoring prompt on registration Swapping a session's classifier option list (a voice-switched command mode, for instance) made the next turns pay a full re-prefill of the new option-list prompt — measured 2.4s vs 0.3s warm on a desktop CPU, and worse: on hybrid-memory models like LFM2.5, whose state cannot be partially rewound (llama.cpp can only restore checkpoints), *every* probe change re-prefilled from scratch whenever the last checkpoint missed the probe boundary, so even same-list turns intermittently cost full prefills. Registering an option list (pipeline seed or session.update) now fires a best-effort background prewarm: two throwaway scores with distinct probes. The first prefills the new option-list prompt; the second, diverging exactly where per-turn probe text starts, plants the backend's rewind point (KV checkpoint) at the stable-prefix boundary that every real turn reuses. The prewarm hides behind the canned mode-switch reply — by the time it finishes speaking, the cache is warm. Idempotent per option set, detached from the registering request's lifetime. Measured on the drone demo (LFM2.5-1.2B, desktop CPU): first turn after a mode switch 2374ms -> 340ms; intermittent same-list full prefills (1.3-2.1s) all -> under 0.5s. For clients that swap lists frequently, options: [parallel:2] on the scoring model additionally keeps one slot per list via prefix-similarity routing (+26MB RSS, unified KV). Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * perf(llama-cpp): checkpoint scoring at the caller-declared stable prefix Hybrid-memory models (LFM2.5 shortconv, Qwen3.5 deltanet — where new small models are headed) cannot rewind their state, so any prompt-cache reuse that needs a rewind falls back to a full re-prefill. For classifier scoring that meant every probe change re-processed the whole option-list prompt: the server's checkpoints were placed reactively (at wherever the previous task happened to diverge), so a checkpoint past the next divergence was erased rather than restored — measured as intermittent 2-10s turns on prompts with a 95%+ common prefix. The classifier now computes the probe-invariant prompt prefix once (the byte-wise common prefix of two synthetic probe renders) and declares its length with every Score request; the server maps it to a token boundary and forces a KV checkpoint exactly there on each score prefill. That checkpoint sits at or before every future divergence under the same option list, so it always survives and always restores — repeat scoring costs probe+candidates regardless of how the probe changes. Also: - prewarm reruns on every option-list registration instead of memoizing per list: with boundary checkpoints a redundant rewarm costs two probe-sized decodes, while skipping one after a slot eviction (three lists sharing fewer slots evict in LRU cascades) silently moves a full re-prefill onto the user's next turn - new llama.cpp backend option rs_seq:N exposes bounded recurrent-state rollback outside speculative decoding; measured impractical for deltanet-scale states (65GB for 64 snapshots on Qwen3.5-4B) but cheap insurance for small-state models - docs: the multi-list recipe (parallel:N + sps:0.5 — the default slot similarity threshold funnels distinct lists onto one slot) Measured on the drone demo (LFM2.5-1.2B scorer, desktop CPU), steady state: every turn 285-421ms including mode switches, vs 2.4s post-switch and intermittent 1.3-2.9s re-prefills before. Assisted-by: Claude:claude-fable-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): align classifier cache guidance Document the single-score prewarm behavior and clean the vendored score patch formatting. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(llama-cpp): guard score task for fork backends TurboQuant and Bonsai reuse the primary gRPC server against llama.cpp forks that do not carry LocalAI's slot-based Score patches. Compile the Score integration only for the patched primary backend and return UNIMPLEMENTED from fork builds instead of referencing absent task types and common_params fields. Assisted-by: Codex:gpt-5 [gh] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(dev): generate gRPC code before commit lint The coverage phase regenerates ignored protobuf bindings, but lint runs first and can fail against missing or stale output. Generate the pinned bindings before lint so the gate always type-checks the current schema. Assisted-by: Codex:gpt-5 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-07-29 12:50:22 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto	878a0d00a1	fix(distributed): reaper reaps live backends, ghost model stubs, in_flight leak, sidecar staging runaway (#11142) ... * fix(distributed): stop the probe reaper from orphaning busy backends The reconciler's liveness probe is a 1s gRPC HealthCheck, and a single failed probe deleted the model's node_models row. A backend that is merely busy cannot answer it: single-threaded Python backends (video and avatar generation) block for minutes inside one request, so the reaper was deleting registry rows for backends that were alive and mid-request. The model then vanished from the nodes page while it was still generating, and because the row was gone the in-flight decrement had nothing to decrement ("DecrementInFlight: no matching row or already zero"). Every subsequent request re-routed and re-staged the full model from scratch. Two guards: - Replicas with in-flight requests are excluded in SQL. A row that is actively serving is proof of life, and the running request is exactly what stops the backend from answering the probe. - Idle replicas must miss three CONSECUTIVE probes before removal, so a transient blip cannot orphan a live replica. A successful probe resets the streak. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(distributed): drop the local model stub when its last replica goes In distributed mode every routed model leaves an in-process stub in the frontend's ModelLoader, and DistributedModelStore.Range reports local stubs UNION the registry rows. Every registry removal path deletes only the DB row, so the stub outlived the replica and the model was reported as loaded forever. That is the "loaded on the home page, absent from every node" ghost: /system reads the union and still sees the stub, while /api/nodes/models reads the registry and correctly sees nothing. It never self-healed, and both frontend replicas showed it independently. The replica-removed chokepoint could not fix this as it stood, because it held a SINGLE hook that the prefix cache already owned, and it was registered only when the prefix cache was enabled. Registering a second listener would have silently displaced the first. - Turn replicaRemovedHook into a list (AddReplicaRemovedHook), so independent subsystems can each register without displacing others. - Add NewLocalStubInvalidator, which drops the local stub once no healthy replica of the model remains anywhere in the cluster, and wire it unconditionally in startup. The stub is kept while another node still serves the model: the frontend is right to consider it loaded, and each request re-routes through SmartRouter to pick a live replica anyway. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(distributed): stop staging checksum sidecars back to workers The file transfer server writes a "<file>.sha256" sidecar next to every file it accepts. The sender walked the model directory with no filter, so it staged those sidecars too, and the receiver duly wrote a sidecar for each sidecar. Every staging pass multiplied the tree: config.json -> config.json.sha256 -> config.json.sha256.sha256 -> ... One LongCat snapshot had grown to 498 files, 466 of them chained, up to 29 levels deep, and the staged file count climbed on every pass. This inflates each transfer and grows disk without bound on both ends. Skip hash sidecars in stageDirectory, and mirror the skip in countStageableFiles so the progress bar still reaches 100%. The check is "a sidecar sitting next to a real file" rather than a blanket suffix ban, so a model that genuinely ships a .sha256 payload with no corresponding base file is still transferred. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(distributed): classify the liveness probe instead of gating on in_flight The previous commit excluded replicas with in-flight requests from the probe reaper. That was the wrong guard, and could invert the bug it fixed. in_flight has no decrement guarantee: track() balances its increment with a defer, but a frontend killed mid-request never runs it, and the load-time reservation is released only when the first inference completes. Nothing resets a leaked counter. Gating the reaper on it therefore meant a leaked counter would shield a genuinely dead replica from ever being reaped. Nor was patience alone a fix: three misses at the default interval is ~90s of silence, while the generation that triggered this blocks for 15+ minutes. The real conflation was in the probe itself. A gRPC HealthCheck against the backend's serving port measures "is it idle enough to answer", not "does the process exist", and probeLoadedModels discarded the error that tells them apart. Because the gRPC client is lazy, the status code is decisive: - DeadlineExceeded: transport fine, nothing serviced the RPC. Busy. - Unavailable: nothing is listening. Gone. ModelProber now returns a ProbeOutcome, and only ProbeUnreachable counts toward the reap threshold. ProbeBusy clears the streak: it is evidence of life. A blackholed network reads as busy too, deliberately, since whole-node failure is the health monitor's job. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * feat(distributed): reconcile replicas against worker-reported processes Probing a backend's own serving port cannot distinguish "busy" from "gone" without inferring it from an error code. The worker can answer directly: it spawned the process, holds the handle, and its reply is not blocked by whatever that backend is doing. Adds a models.running request-reply subject. The worker answers out of its in-memory process table, reporting each live process as (modelID, replicaIndex, address) — the supervisor's process keys are `modelID#replica`, which is isomorphic to a NodeModel row, so the reconciler can diff the two directly. reconcileNodeProcesses runs before the port probe and reaps rows for models the worker is not running. Models the worker vouches for get updated_at bumped, which takes them out of the port prober's stale set entirely: that is what keeps a backend deep in a long generation away from the probe in the first place, rather than relying on classifying its silence after the fact. A worker that does not answer is skipped, not assumed empty. A messaging failure says nothing about the processes, and assuming the worst would delete a node's rows on a transient NATS blip; the port probe stays as the fallback for those nodes. Rows younger than probeStaleAfter are ignored so a freshly created row is never judged against a process table that has not caught up. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] * fix(distributed): stop in_flight leaking and pin replicas against eviction A leaked in_flight counter is not cosmetic. FindLRUModel, FindGlobalLRUModelWithZeroInFlight and the router's eviction query all require in_flight = 0, so a replica whose counter never came back is pinned and its VRAM is unreclaimable for the lifetime of the process. Two halves. The source: routing reserves in_flight = 1 at load time so a freshly loaded replica is not evicted out from under the request that caused the load. That reservation was released ONLY by the first inference completing, so a route torn down before any inference ran (client disconnect, handler error, failure between load and the backend call) stranded it. newRouteResult now wires the reservation to a sync.Once fired by whichever comes first, the first inference or route teardown, and replaces three copies of the old wiring. The backstop: a sweeper for counters leaked by paths that cannot run a defer at all, such as a frontend killed mid-request. Identifying a leak by elapsed time alone is unsafe. IncrementInFlight stamps last_used at request START and nothing moves it while the request runs, so a long generation is indistinguishable from a leak by age, and resetting there would expose a serving model to eviction. The probe supplies the missing bit: a backend that answers a health check promptly is not inside a request, because that is precisely what a busy one cannot do. Requiring the row to also be idle for 30 minutes covers backends that serve in parallel and can answer while working, since those keep last_used fresh through each new increment. Two existing tests asserted the old behaviour ("No decrement on Release"). That assertion was the leak, so both now pin the release instead. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-5 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-27 18:36:31 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto	1cd7d63c7b	fix(distributed): reject wrong-model requests on the remaining modalities (#10990) ... #10970 gave the four PredictOptions RPCs a model-identity check so a backend reached through a stale distributed route rejects the request instead of answering from whatever model it holds (#10952). Every other modality shares that exposure: the route is cached by host:port, a worker can recycle a stopped backend's port for another model's backend, and a liveness-only probe cannot tell a stale row from a valid one. Extends the same mechanism to the 21 remaining request messages that reach a backend through the router, using the pattern #10970 established rather than a parallel one: - proto: ModelIdentity on each modality request message. - controller: populated from ModelConfig.Model at the call site that also builds ModelOptions, so load-time and request-time values are equal by construction. - backends: one generic guard in pkg/grpc/server.go (27 Go backends), the method set in backend/python/common (36 Python backends), llama-cpp (AudioTranscription/Stream, Rerank, Score) and privacy-filter (TokenClassify). - reconcile already drops the stale row on IsModelMismatch; no change. TTSRequest and SoundGenerationRequest get a SEPARATE ModelIdentity field rather than reusing their existing `model`: FileStagingClient rewrites `model` to a worker-local path, so comparing it would reject valid requests in exactly the configuration this guards. AudioEncode/AudioDecode are deliberately left unguarded: the opus codec backend is loaded from a literal rather than a ModelConfig, so no value carries the equality guarantee the comparison depends on. The four bidirectional stream RPCs are out of scope; they bypass reconcile. Empty means skip on both sides, so an old controller, an old backend, and the bare request structs in tests/e2e-backends all keep working. Assisted-by: Claude Code:claude-opus-4-8 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-20 21:58:19 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto	465d488c90	fix(distributed): reject wrong-model requests at the backend (#10970) ... fix(distributed): reject wrong-model requests at the backend (#10952) In distributed mode the controller caches a NodeModel row naming a backend's host:port. A worker can recycle a stopped backend's gRPC port for a different model's backend, and probeHealth verifies liveness rather than identity, so the probe succeeds against whatever now occupies the port and the request is dispatched to the wrong backend. The caller gets a silent wrong-model answer. Nothing in the request could catch this: PredictOptions had no model field, so model identity crossed the wire only in ModelOptions.Model at LoadModel time, and the cached-hit path issues no LoadModel. Every backend's "model not loaded" guard checks a nil handle, which a process holding a different model passes, so the stale row was never dropped either. Add PredictOptions.ModelIdentity and enforce it at the point of use: - The controller populates it in gRPCPredictOpts from ModelConfig.Model, the same expression ModelOptions feeds to model.WithModel and therefore the same value the backend received as ModelOptions.Model. Both are read from one config value in one function, so they are equal by construction and the comparison cannot false-reject. - Backends compare it against what they loaded and return NOT_FOUND with a fixed sentinel. Enforced in pkg/grpc/server.go (27 Go backends), an interceptor in backend/python/common (all 36 Python backends, no per-backend change), and the llama-cpp / ik-llama-cpp / ds4 C++ servers. That is every backend with real exposure: kokoros answers all four RPCs with unimplemented and privacy-filter implements none of them. - The router's reconcile drops the stale replica row on a mismatch, so the next request reloads somewhere correct. Empty means "skip the check" on both sides: a controller that predates the field sends nothing, a backend loaded by such a controller has nothing to compare, and the C++ server synthesizes PredictOptions internally for ASR. That keeps upgrades working in both directions. Scoped to the four PredictOptions RPCs. TTSRequest.model and SoundGenerationRequest.model are deliberately NOT validated: FileStagingClient already rewrites them to worker-local absolute paths, so in distributed mode they already differ from the load-time value and comparing them would reject valid requests. IsModelMismatch requires both the NOT_FOUND code and the sentinel, unlike the neighbouring helpers which accept either. insightface's Embedding returns NOT_FOUND "no face detected" on a PredictOptions RPC, and a code-only check would drop a healthy replica row on every faceless image. Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-20 13:05:47 +02:00
mudler's LocalAI [bot]andEttore Di Giacinto	f735cb24c0	fix(worker): reap deleted backends and stop models that live on a worker (#10956) ... * fix(worker): reap deleted backends and stop models that live on a worker Three related backend-lifecycle defects, all reachable from the same production incident on a Jetson/Thor worker: a deleted backend's gRPC process survived ~40 minutes with its directory removed from disk, a later model load was routed to that orphan and failed with a certifi path pointing into the deleted directory, and the admin could not stop the model because the frontend reported it as not loaded. 1. backend.delete orphaned the process it claimed to delete ------------------------------------------------------------ s.processes is keyed by `modelID#replicaIndex` (buildProcessKey), so the backend name never appeared in a key and was recorded nowhere on the process. backend.delete resolved its target via isRunning/stopBackend, whose prefix path only matches a bare *modelID* - a delete keyed on a backend name resolved to zero keys, the stop silently no-op'd, and the files were removed out from under a live process. The install fast path then handed that orphan back out: it returns any live process for the (model, replica) slot without checking which backend started it, so a reinstalled variant inherited the deleted backend's port. - Record backendName on backendProcess, threaded installBackend -> startBackend. - Add resolveProcessKeysForBackend, matching the recorded name and resolving alias <-> concrete via ListSystemBackends *before* DeleteBackendFromSystem erases the metadata that carries the alias. Alias resolution failure degrades to name-only matching so a delete never fails on it. - backend.stop goes through resolveStopTargets, which accepts a backend name, a model name, or an exact modelID#replica key. Its payload field is named "backend" but is published with all three meanings: the admin UI sends a backend name, UnloadRemoteModel sends a model name, and the router's abandoned-load reap (#10948) sends an exact replica key. Narrowing it to backend names alone would strand the latter two. backend.delete stays strict - its identifier is unambiguously a backend. - Gate the install fast path on processMatchesBackend so a slot held by a different backend is restarted rather than reused. Processes with no recorded name (pre-upgrade) are accepted, so rollout does not restart every running backend. - stopBackendExact reports a real stop failure - the process still being alive afterwards, which is precisely what finishBackendStop already detects to keep the entry and its port reserved - and backend.delete no longer replies success when it knew about a process and could not kill it. "No process was running" stays a success but is logged, so the orphan case is visible rather than silent. 2. /backend/shutdown reported a running model as missing --------------------------------------------------------- ModelLoader.deleteProcess short-circuits on a miss in this replica's in-memory store. In distributed mode the authoritative record of "is this model loaded" is the shared node registry: a frontend replica that never served the model itself (load balancer picked a peer, or the replica restarted) has no local entry. The remote unload path that pkg/model documents ("when ShutdownModel is called for a model with no local process, UnloadRemoteModel is called") sat behind that short-circuit, unreachable in exactly the case it exists for. #10865 reworked this function but kept the short-circuit at the top, so the gap survived that refactor. - deleteProcess consults the remote unloader on a local-store miss, via a shared unloadRemote helper so this branch and the existing no-local-process branch both prefer #10865's RemoteModelContextUnloader, preserving force propagation across the distributed boundary. - UnloadRemoteModelContext reports ErrRemoteModelNotLoaded when no node has the model; it previously returned nil, making a no-op stop indistinguishable from a real one. The converse case (nodes have it, none could be stopped) already errors since #10865 joined the per-node failures, so that half of the original fix was dropped as redundant. - Only when the model is absent locally AND cluster-wide does the endpoint report not-found, now 404 naming both scopes rather than a bare 500. - modelNotFoundErr becomes the exported ErrModelNotFound so the HTTP layer can map it without string matching; watchdog's identity comparison becomes errors.Is. 3. Coverage for the bounded Free() that #10865 shipped untested ---------------------------------------------------------------- The original branch also bounded the pre-stop Free(), but #10865 landed that fix first (workerBackendFreeTimeout, applied in both stopBackendExact and handleModelUnload). That production change is therefore DROPPED here as superseded - master's version is strictly better, since it also releases the supervisor mutex across the call and keeps the port reserved until termination completes. What #10865 did not ship is a test, and the bound is load-bearing: the router-side reap in #10948 sends backend.stop for an abandoned load, and against a wedged backend an unbounded Free would swallow that stop before it reached the process. Nothing failed if the bound regressed. The spec stands up a real gRPC backend server whose Free handler never returns - what a Python backend looks like when its single worker thread (PYTHON_GRPC_MAX_WORKERS=1 on 37 backends) is occupied by a stuck LoadModel. A stub socket is not sufficient and was tried first: without a completed HTTP/2 handshake, gRPC's own ~20s connect timeout ends the call, so that version passed against the very bug it targets. With the connection READY, only the caller's deadline can end it, so the spec hangs to its 60s limit if the timeout is removed and passes with it. Its fixture process is deliberately never started. go-processmanager v0.1.1 writes Process.pid from readPID() without synchronization, so a live process races its own monitor goroutine under -race - reproducible with a bare Run()+Stop() and unrelated to this spec. Since scripts/model-lifecycle-conformance.sh runs this package with -race and is fail-closed, starting one would turn that gate red on an upstream defect. An unstarted process still proves the point: the stop is reached and the slot released, which is exactly what an unbounded Free prevents. Verified: make lint (new-from-merge-base origin/master) reports 0 issues; scripts/model-lifecycle-conformance.sh passes all three stages including the FizzBee liveness check (1458 states, IsLive: true). Assisted-by: Claude:claude-opus-4-8 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): keep remote unload idempotent, ask presence separately 2035a4d25 made UnloadRemoteModel return ErrRemoteModelNotLoaded when no node holds the model, so ShutdownModel could answer 404 instead of a misleading 500. That narrowed a shared adapter contract to serve one caller and broke the documented idempotent-unload guarantee, which CI caught on PR #10956: [FAIL] Node Backend Lifecycle (NATS-driven) > NATS backend.stop events should be no-op for models not on any node [Distributed] Expected success, but got: model not loaded on any node The spec name states the contract outright. The matching unit assertion was updated in that commit; this e2e one was missed because it lives under tests/e2e/ with no build tags and does not run in package-scoped test runs. Caller audit - who breaks when an idempotent unload becomes an error: - pkg/model/watchdog.go:902 (LRU memory reclaimer) is the serious one. It untracks a model ONLY when shutdown returns nil or ErrModelNotFound. A new error type means the model is never untracked, so the reclaimer keeps re-selecting the same entry and never reclaims - a live wedge whenever a local store entry outlives the remote model. - core/services/galleryop/managers_local.go:43 (DeleteModel) would warn on every deletion of an already-unloaded model. - core/services/modeladmin/{state,config,remote_sync}.go stop instances best-effort against models that are frequently not loaded. - deleteProcess itself: the no-local-process branch returns the unload result directly, so a stale local entry for a model no longer on any node turned a previously-successful cleanup into a failure. Only ShutdownModel wants the distinction, and only on the local-store-miss path. So the distinction moves to the caller instead of the contract: - UnloadRemoteModel/UnloadRemoteModelContext return nil again when no node has the model, and ErrRemoteModelNotLoaded is removed. - New optional RemoteModelPresenceChecker (HasRemoteModel) answers the question directly. deleteProcess consults it BEFORE unloading, because an idempotent unload cannot report afterwards whether anything was stopped. Absent locally AND cluster-wide is the only case that reports 404. - A failed registry lookup is surfaced rather than reported as absence: an unreachable registry is not evidence a model is gone, and answering a confident 404 off a failed lookup is how an operator gets told a running model does not exist. - Unloaders that predate the extension keep working - deleteProcess attempts the unload rather than refusing it - and compile-time assertions in the nodes package now pin all three optional interfaces, since both are consumed by runtime type assertion where drift degrades behavior silently instead of failing the build. The contract is now pinned at both levels that disagreed, each spec pointing at the other: "with no nodes returns nil" in unloader_test.go and "should be no-op for models not on any node" in node_lifecycle_test.go. Verified: full distributed e2e suite 233 passed / 0 failed (the suite that failed 232/1 in CI); pkg/model and core/services/nodes green; make lint new-from-merge-base reports 0 issues. Assisted-by: Claude:claude-opus-4-8 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): drop replica rows when a worker stops a backend A worker returns a stopped backend's gRPC port to its allocator as soon as the process is confirmed dead, and hands it to the next backend that starts. The controller's NodeModel row for the old address survives, and both SmartRouter.probeHealth and the HealthMonitor per-model probe verify liveness, not identity, so once an unrelated backend binds the recycled port the stale row passes every check and the request is served by the wrong backend instead of failing. backend.delete is newly able to trigger this: before #10956 a delete never actually stopped a process, so it never recycled a port. backend.upgrade has the identical gap and always did — upgradeBackend force-stops every process using the binary and starts none back up, while DistributedBackendManager.UpgradeBackend never removes rows. model.unload is the one path that gets this right today: it calls RemoveAllNodeModelReplicas straight after StopBackend. Report the process keys the worker terminated on the delete and upgrade replies, and drop the matching rows in RemoteUnloaderAdapter, which already holds a ModelLocator with RemoveNodeModel. All three call sites funnel through that adapter, so no new interface, DB migration, or proto change is needed. A key is reported only once its process is confirmed gone, so the list stays trustworthy on the partial-failure replies too. Old workers never populate the new fields. ReportsStoppedProcesses tells "stopped nothing" apart from "does not report", so an old worker's silence falls back to the pre-existing probe-based staleness recovery instead of being mistaken for a completed cleanup. Quarantine released ports for a short window as an interlock covering the NATS round-trip between the worker freeing the port and the controller dropping the row. It is deliberately not derived from HealthCheckInterval: that cadence is operator-tunable and the per-model reaper can be disabled outright, so coupling a worker-local constant to it would be silently wrong on some clusters. Eager row removal is the fix; the delay only closes the handoff gap. Identity verification in probeHealth was considered and rejected: Health and Status carry no backend identity, so it needs a proto change plus an implementation in 36 Python and 4 C++ Health servicers, it is fail-open for any backend not yet rebuilt, and the probeCache short-circuit means it would not even execute during the 30s window where the misroute happens. Fixes #10952 Refs #10954, #10956 Assisted-by: Claude:claude-opus-4-8 golangci-lint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(deps): bump go-processmanager, assert real backend termination go-processmanager wrote Process.PID from readPID() with no synchronization while its own monitor goroutine cleared the same field on exit, so a bare Run()+Stop() tripped the race detector without any concurrent access from the caller. LocalAI hit this on every backend stop. Upstream fixed it in a94e2b7 by guarding PID with a mutex and adding CurrentPID() as a race-safe accessor. The exported field was kept to avoid a breaking change but is now deprecated: a direct read still races the monitor. No tag carries the fix yet, so pin the pseudo-version. GetGRPCPID reads through CurrentPID() instead of the field. The accessor returns the same string under an RLock, so the empty-PID and strconv error paths are unchanged; it is the only direct field read in the tree. With the race gone, the Free-timeout spec no longer has to leave its fixture process unstarted. It now runs a real child and asserts the child genuinely exits, which is exactly what the earlier workaround gave up: the spec could show the stop was reached and the slot released, but not that SIGTERM ever landed. Termination is observed through Done(), which closes only once the library has waited on the child. The pidfile-based liveness helpers cannot serve here, because Stop() deletes the pidfile while releasing the handle and so reports "not alive" even if no signal was sent. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-4-8[1m] [Read] [Edit] [Bash] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-20 01:09:22 +02:00
LocalAI [bot]andEttore Di Giacinto	b00422e45f	feat(backends): add LongCat video and avatar generation (#10792) ... * feat(backends): add LongCat video and avatar generation Assisted-by: Codex:GPT-5 [apply_patch] [exec_command] [web] * refactor(config): declare model I/O modalities Make model configs declare input and output modalities so capability discovery no longer branches on backend or checkpoint names. Complete the LongCat gallery and user documentation, make the SDPA patch apply to the pinned upstream revision, and stabilize the Agent Jobs race exposed by the required hook. Assisted-by: Codex:GPT-5 [web] --------- Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-12 23:58:46 +02:00
LocalAI [bot]andEttore Di Giacinto	d7d7721eae	feat(distributed): SyncedMap component + migrate finetune/quant/agent-tasks to cross-replica state (#10542) ... * feat(distributed): add SyncedMap cross-replica in-memory state component Introduce core/services/syncstate.SyncedMap[K,V]: a thread-safe in-memory map that keeps itself consistent across frontend replicas via NATS, with an optional pluggable durable Store and hydrate-from-source convergence. Several features keep process-local state surfaced to the API (finetune/quant jobs, agent tasks, model configs) and each hand-wired the same in-memory + NATS broadcast + read-through-store legs - or forgot to, reintroducing cross-replica staleness. SyncedMap makes that consistency a configuration choice: - local writes mutate the map, write through the Store, then broadcast a delta; - the apply path is memory-only and never re-publishes or re-writes the Store (structural echo-loop guard, mirroring galleryop.mergeStatus); - on Start and on NATS reconnect the map re-hydrates from the source (Store, else Loader); an optional periodic Reconcile repairs silent drift; - standalone mode (nil NATS client) is a strict in-memory no-op. Reconnect re-hydrate is wired via a new *messaging.Client.OnReconnect callback, consumed through an optional type-assertion so MessagingClient stays minimal. Adds messaging.SubjectSyncStateDelta and a reusable testutil.FakeBus (synchronous in-process MessagingClient with wildcard matching) for adopter tests. Component only; service migrations follow in subsequent commits. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * refactor(finetune): back jobs with SyncedMap for cross-replica consistency FineTuneService kept jobs in a process-local map and, although it wrote them to Postgres, ListJobs/GetJob never read the store back and the wired natsClient was never used - so in distributed mode a job created on one replica was invisible to the others. Replace the map and the dead client with a syncstate.SyncedMap keyed by job ID, value *schema.FineTuneJob (the exact REST shape, so responses are unchanged). - Add a Store adapter (core/services/finetune/syncstore.go) over FineTuneStore, plus FineTuneStore.ListAll (global hydrate; per-user List kept) and an idempotent Upsert (create-or-update; Create alone fails on dup key). - Writes go through SyncedMap.Set/Delete (write-through + broadcast); reads use List/Get. The on-disk state.json path becomes the standalone Loader, keeping single-node restart recovery (stale->stopped / exporting->failed fixups). - Fold SetNATSClient/SetFineTuneStore into NewFineTuneService; app.go passes the distributed NATS client + store when distributed, nil otherwise. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * refactor(agentpool): back agent tasks with SyncedMap for cross-replica consistency AgentJobService.ListTasks read the process-local tasks map only, while ListJobs already read through the DB persister + dispatcher NATS - so in distributed mode a task created on one replica was invisible to the others. Back tasks with a syncstate.SyncedMap keyed by task ID (value schema.Task, the exact REST shape); jobs are left untouched. - Store adapter (task_syncstore.go) over the existing JobPersister (LoadTasks/SaveTask/DeleteTask); reads svc.persister/userID live so a persister swap needs no rebuild. No new persister methods required. - Task reads -> SyncedMap.List/Get; create/update -> Set (write-through + broadcast); delete -> Delete. The file persister now owns its own task set so the write-through path does not re-enter the SyncedMap lock (deadlock guard). - The distributed NATS client is not available at construction (start() precedes initDistributed), so it is injected via SetTaskSyncNATS, which rebuilds the still-empty map before Start/hydrate. Wired at the main, restart, and per-user (UserServicesManager) distributed sites. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * refactor(quantization): back jobs with SyncedMap + durable QuantStore QuantizationService kept jobs in a process-local map persisted only to a local state.json, so in distributed mode jobs were neither visible across replicas nor durable cluster-wide. Back jobs with a syncstate.SyncedMap keyed by job ID (value *schema.QuantizationJob, the exact REST shape). - New distributed.QuantStore (GORM, table quantization_jobs) mirroring FineTuneStore: Create/Get/ListAll/Upsert(idempotent)/Delete, registered for AutoMigrate via distributed.InitStores (Stores.Quant). - New adapter (quantization/syncstore.go) over QuantStore implementing syncstate.Store, with record<->schema conversion. - Reads go through List/Get, writes through Set/Delete (write-through + broadcast); state.json is kept as the standalone Loader for single-node restart recovery (stale-job fixups preserved). - app.go passes the distributed NATS client + QuantStore when distributed, nil otherwise; Start/Close lifecycle mirrors finetune. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * fix(syncstate): annotate gosec G118 false positive on lifeCtx gosec flagged the WithCancel in Start as "cancellation function not called" because the returned cancel is stored on the struct rather than called/deferred in scope. It is invoked in Close (covered by tests), and lifeCtx must outlive Start to drive the reconnect/reconcile goroutines. Suppress the verified false positive with a justified #nosec G118. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] * test(distributed): e2e two-replica SyncedMap sync over real NATS + Postgres Adds the real-infrastructure counterpart to the fake-bus unit tests, in the existing distributed e2e suite (testcontainers NATS + PostgreSQL). Two SyncedMap instances stand in for two frontend replicas - each with its OWN NATS connection to a shared server and a SHARED Postgres store (the distributed-mode invariant) - and assert, over the wire: - a create on replica A is observed by replica B; - an update and a delete propagate A -> B (delete prunes, which a reload cannot); - a late-joining replica recovers a job it never received a delta for, via store hydrate on Start (the at-most-once gap a fake bus cannot exercise); - a local Set is written through to the shared Postgres store. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-8 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-27 23:23:51 +02:00
LocalAI [bot]andEttore Di Giacinto	69c16481c8	fix(test): update e2e UpdateProgress calls for new cancellable arg (#10460) ... PR #10454 added a `cancellable bool` parameter to GalleryStore.UpdateProgress but missed two callers under tests/e2e/distributed, breaking the build on master (golangci-lint and tests-e2e-backend both failed to compile with "not enough arguments in call to ... UpdateProgress"). Pass cancellable=true (both ops are downloading installs, which are cancellable) and assert the flag is persisted, exercising the new behavior. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-22 23:45:22 +02:00
Richard Palethorpe	63bcbf6c12	fix(pii): post-merge review fixes + live NER e2e for the privacy-filter tier (#10401) ... * fix(pii): post-merge review fixes + live NER e2e for the privacy-filter tier Follow-up to the NER tier engine (#10360), already on master. This carries only the incremental review fixes and tests that postdate that merge — the feature itself is not re-introduced. Review fixes: - openai_completion.go: remove the dead `elem >= 0` conjunct in applyAnyText (the `elem < 0` guard above already returns). - application.go: collapse ResolvePIIPolicy's inline re-implementation of PIIIsEnabled to a single cfg.PIIIsEnabled() call (sole source of the "explicit pii.enabled wins, else cloud-proxy default" rule) and return true past the !enabled guard where it is provable. - pattern.go: hoist the triple `appConfig != nil && EnableTracing` check in patternDetector.Detect into one local. - grammar.go: MaxQuantifier was 4096, but Go's regexp/syntax rejects repeat bounds above 1000 at Parse time, so walk()'s {n,m} guard could never fire — dead code shadowed by the parser. Lower it to 512 so a bound in (512,1000] is rejected here with an actionable error; >1000 still fails closed via Parse. Specs pin the relationship so the guard can't silently revert. - PatternListEditor.jsx: clamp a directly-typed negative min_len to >=0 and force the DOM value back when clamping (min={0} only constrained the spinner, so a negative reached saved config and silently disabled the length filter). Tests: - piipattern_test.go: MaxQuantifier guard specs (must stay live, not dead). - model-config.spec.js: assert the min_len clamp, and that entity_actions collapses a duplicate group to a single row (map semantics; regression guard against emitting an array that drops a row on save). - tests/e2e-backends: token_classify capability driving the TokenClassify gRPC RPC against the backend image, asserting byte-correct, UTF-8 rune-aligned spans (entity.Text == text[start:end]) at threshold 0. Verified on CPU via `make test-extra-backend-privacy-filter` (3/3 specs). - Makefile: test-extra-backend-privacy-filter wrapper. - tests/e2e: e2e_pii_ner_test.go drives /api/pii/analyze + /api/pii/redact (mask + block) through the full HTTP -> detector -> redactor path; gated on PII_NER_MODEL_GGUF so the default suite is unaffected. - .github/workflows/tests-pii-ner-e2e.yml: path-filtered / nightly CI job running the container harness on CPU. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(gallery): add privacy-filter-nemotron (f16 + q8) GGUF conversions of OpenMed/privacy-filter-nemotron — a fine-grained English PII token-classifier (55 categories / 221 BIOES classes), fine-tuned from openai/privacy-filter on NVIDIA's Nemotron-PII dataset. Sibling to the existing privacy-filter-multilingual entry, trading language breadth for category depth. - privacy-filter-nemotron: F16 reference artifact (~2.8 GB). - privacy-filter-nemotron-q8: Q8_0 quant (~1.64 GB) for RAM-constrained / edge use; description notes the size/speed tradeoff and to validate on your own data (a single dropped span is a PII leak). Both run on the privacy-filter backend with known_usecases [token_classify] and a default mask policy (min_score 0.5); operators add per-category entity_actions as needed. sha256s taken from the HF repo's LFS object ids. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-06-22 18:26:19 +02:00
3 people	32c47706ae	feat(realtime): speaker-aware conversations - surface identity to client and LLM (#10424) ... * feat(realtime): add voice_recognition enforce + identity config Add Enforce *bool and Identity *VoiceIdentityConfig to PipelineVoiceRecognition, plus EnforceGate/IdentityEnabled/ AnnounceEnabled/PersonalizeEnabled helpers. Enforce nil defaults to gating (backward compatible); identity surfacing is independent of the gate. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): add Speaker type and conversation.item.speaker event Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(realtime): split voiceGate into Resolve + authorize Split the speaker authorization into a Resolve step (embed once, produce a types.Speaker identity) and a pure authorize policy step, with a 0..100 confidence score mirroring /v1/voice/identify. The legacy Authorize wrapper is kept so existing specs stay green. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): resolve speaker per turn and emit conversation.item.speaker Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): personalize LLM turns with recognized speaker Set the per-message name field on each recognized user turn and append a current-speaker note to the system message, both gated by the voice recognition identity config. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(realtime): document speaker identity surfacing and personalization Document the new voice_recognition keys (enforce, identity.*) and the LocalAI-extension conversation.item.speaker server event in the realtime feature docs. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(realtime): cover when:first+identity re-resolution and multi-speaker history Add two integration specs to harden the speaker-aware realtime path: - when:first with an Identity block re-resolves the speaker every turn even though re-authorization is skipped after the first match: a later resolve error now fails closed, while a clean later resolve still surfaces and names the speaker. - multi-speaker history attribution: each user turn carries its own per-message name and the injected system note reflects the latest speaker. Test-only change; no production behavior was modified. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(realtime): surface speaker labels in conversation.item.speaker Carry the registered speaker's labels (identify mode) on types.Speaker so they flow into the conversation.item.speaker event and the stored item. Verify mode has no labels, so the field is omitted there. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e): cover conversation.item.speaker over a real websocket Add a realtime-pipeline-identity config (verify mode, enforce:false, identity announce+announce_unknown+personalize) and two e2e specs driving the real server over a real WebSocket with the mock VoiceEmbed backend: an authorized speaker yields a conversation.item.speaker event naming e2e-speaker (matched true) and reaches response.done; an unauthorized speaker yields an unknown (matched false, no name) event and still responds, proving enforce:false never drops a turn. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(config): register voice_recognition enforce + identity fields The meta registry coverage test (TestAllFieldsHaveRegistryEntries) requires every config field to have an entry in core/config/meta/registry.go. The new voice_recognition.enforce and voice_recognition.identity.* fields were missing, failing tests-linux and tests-apple. Add registry entries (toggles) so the fields are surfaced in the model-config editor and the coverage test passes. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>	2026-06-21 21:07:10 +02:00
Richard Palethorpe	3fa7b2955c	feat(pii): NER tier engine — privacy-filter.cpp backend + NER-centric PII filter (#10360) ... Squashed feat/pii-ner-tier-engine rebased onto master (was 45 commits; see backup/pii-ner-tier-engine-prerebase). Net change: - privacy-filter.cpp: standalone GGML engine for the openai-privacy-filter PII/NER token classifier, wired as a LocalAI gRPC backend (CPU/CUDA/Vulkan). TokenClassify moves off the patched llama.cpp path onto this backend. - PII filter reworked to be NER-centric (encoder/NER detection tier scanning whole conversations as one document), with a recreated bounded restricted- regex secret-matching pattern detector tier alongside it (per-model pii_detection.builtins / .patterns + core/services/routing/piipattern). - Detection labelled by source (ner vs pattern); backend trace / confidence / debug observability; analyze/redact exposed as a synchronous API. - Instance-wide default detector policy + per-usecase default-on; request filtering extended to completions, embeddings, edits & Ollama. - React UI: NER-centric PII editor, detector-models table, pattern/builtins editor, middleware default-policy UI. - Gallery: privacy-filter-multilingual token-classify model + NER install filter; token_classify known_usecase; batch sized to context for NER models. privacy-filter backend registered in the backend gallery (cpu/vulkan/cuda-13 meta + image entries with a capabilities map) matching its CI matrix jobs, and an /import-model auto-detect importer (PrivacyFilterImporter, narrow privacy-filter GGUF detection) replacing the prior pref-only registration. Reconciled against master's independent evolution: - Dropped master's PIIPatternOverrides feature (global-pattern runtime overrides + /api/pii/patterns API + runtime_settings.json persistence). The per-model NER + pattern-detector design supersedes it; it was built on the global redactor pattern set this branch replaced. - Reverted the llama.cpp Score carry-patch (0006-server-task-type-score): removed the patch and restored master's grpc-server.cpp Score RPC (direct llama_decode, slot-loop bypass) and LLAMA_VERSION pin, plus master's model_config validation forbidding score + chat/completion/embeddings on llama-cpp. token_classify is unaffected (it runs on the privacy-filter backend, not llama-cpp). Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-06-18 11:45:22 +01:00
LocalAI [bot]andEttore Di Giacinto	1ab61a0875	feat: generic chat_template_kwargs (model config + per-request metadata) (#10359) ... * feat(config): add chat_template_kwargs model field + resolver Adds the ChatTemplateKwargs model-config map and RequestMetadata carrier, plus ResolveChatTemplateKwargs which layers the config map under coerced request metadata. Foundation for generic jinja chat-template kwargs (issue #10329). Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend): forward resolved chat_template_kwargs blob to backends gRPCPredictOpts now merges per-request client metadata over the server-derived enable_thinking/reasoning_effort (reaching all backends via the standalone keys) and serialises the resolved chat_template_kwargs map into a JSON blob for llama.cpp, written last so a client cannot clobber it. Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(http): wire request metadata to config.RequestMetadata The OpenAI request metadata field was parsed but unused; stamp it onto the per-request ModelConfig so gRPCPredictOpts forwards it as chat_template_kwargs overrides. Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(llama-cpp): generic chat_template_kwargs merge (drop per-key blocks) Replace the per-key enable_thinking/reasoning_effort handling in both the streaming and non-streaming chat paths with a single block that parses the chat_template_kwargs JSON blob resolved by the Go layer and merges every key into body_json. New jinja template levers (e.g. preserve_thinking) now need no C++ change. Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: document custom chat_template_kwargs (model + per-request) Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(backend): pin reasoning_effort as a string in the chat_template_kwargs blob Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(http): e2e guard pinning chat_template_kwargs forwarded to gRPC Adds an ECHO_PREDICT_METADATA marker to the mock-backend that echoes the received PredictOptions.Metadata, and an app_test.go spec that drives a real /v1/chat/completions request (model chat_template_kwargs + per-request metadata override) and asserts the exact metadata + chat_template_kwargs blob the REST layer forwards to gRPC. Locks the REST->gRPC contract against regressions. Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(config): grandfather chat_template_kwargs in registry coverage chat_template_kwargs is a free-form map[string]any (like engine_args, already on the list), not a scalar the config UI registry can surface, so it is exempt from the registry-entry requirement. Fixes the TestAllFieldsHaveRegistryEntries failure introduced by the new field. Issue #10329. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-16 12:16:34 +02:00
LocalAI [bot]andEttore Di Giacinto	4d3d54d61b	test(e2e): live-server voice-recognition gate test (#10324) ... Add mock-backend VoiceEmbed/VoiceVerify (deterministic DC-offset speaker discrimination) and a verify-mode gated realtime pipeline, then drive the real HTTP/WS stack: an authorized speaker reaches response.done while an unauthorized one is dropped before the LLM with a speaker_not_authorized event. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-13 23:54:27 +02:00
Richard Palethorpe	085fc53bbc	fix(router): production-ready request router + auto-size batch for embedding/rerank (#10104) ... * fix(router): score classifier production-readiness Conversation trimming runs through the classifier model's chat template and trims by exact token count, sized to the model's n_batch which is now scaled to context so long probes can't crash the backend. Missing chat_message templates are a hard error at router build time. Router- facing factories (Embedder/Scorer/Reranker/TokenCounter) re-resolve ModelConfig per call so a model installed post-startup doesn't bind a stub Backend="" config and silently fall into the loader's auto- iterate path. New 'vector_store' backend trace recorded inside localVectorStore on every Search/Insert — including the backend-load-failure path that previously vanished into an xlog.Warn — with outcome tagging (hit/miss/empty_store/backend_load_error/find_error/insert_error/ok). Companion cleanup drops misleading similarity:0 and input_tokens_count:0 from non-hit and text-mode traces. Gallery local-store-development aliases to 'local-store' so the master image satisfies pkg/model.LocalStoreBackend lookups from the embedding cache. Misc: llama-cpp TokenizeString reads the correct 'prompt' JSON key (the original bug); ModelTokenize nil-guard; non-fatal mitm proxy startup; PII 'route_local' renamed to 'allow' with docs/UI in sync; model-editor footer no longer eats the edit area on small screens; several config-editor template/dropdown/section fixes. Tests: e2e router specs (casual/code-hint + long-conversation trim), vector_store trace specs, lazy-factory specs, gallery dev-alias resolution, Playwright trace badge + scroll regression. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(backend): auto-size batch to context for embedding and rerank models Embedding and rerank models pool over the whole input in a single physical batch (n_ubatch). With batch left at the 512 default, the backend rejects longer inputs with "input is too large to process", silently capping a large-context embedder (e.g. 8k/32k) at 512 tokens. Size n_batch to the context for these single-pass usecases, mirroring the existing FLAG_SCORE behaviour; an explicit batch: still wins. Extracts EffectiveContextSize/EffectiveBatchSize from grpcModelOpts so the effective decode window has one home for other callers to reuse. Adds an e2e-aio regression test that embeds a >512-token input. The AIO embedding model is switched to nomic-embed-text-v1.5 (2048 context) because the previous granite model was capped at 512 tokens and could not exercise the larger batch. Assisted-by: claude-code:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(gallery): raise arch-router scoring output cap via parallel:64 Scoring decodes the whole prompt+candidate in a single llama_decode and reads one logit row per candidate token. The vendored llama.cpp server caps causal output rows at n_parallel, so the default of 1 aborts with GGML_ASSERT(n_outputs_max <= cparams.n_outputs_max) on multi-token route labels. Set options: [parallel:64] on both arch-router quant entries to lift the cap; kv_unified (the grpc-server default) keeps the full context per sequence, so this does not split the KV cache. Assisted-by: claude-code:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-06-12 16:21:15 +02:00
Richard Palethorpe	3a932a9803	feat(distributed): Add NATS JWT authentication and TLS/mTLS options (#10159) ... * feat(distributed): NATS JWT auth, TLS/mTLS options, and e2e coverage Mint per-node NATS user JWTs at registration when LOCALAI_NATS_ACCOUNT_SEED is set, and connect workers with scoped credentials from the register response. Add optional LOCALAI_NATS_TLS_CA/CERT/KEY for private CA and mTLS alongside tls:// URLs, plus test-e2e-distributed and NatsJWT container e2e specs. Document JWT setup (nats-auth-setup.sh) and TLS env vars in distributed-mode. Assisted-by: Grok:grok grok-build Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(distributed): correct NATS JWT scoping and harden client auth The JWT-auth path added in 46467cc7 had several gaps that fail silently under LOCALAI_NATS_REQUIRE_AUTH: - Agent-worker minted JWTs did not allow the subjects the agent worker actually subscribes to (jobs.mcp-ci.new and nodes.<id>.backend.stop), so MCP-CI jobs and backend-stop session cleanup were silently dropped. Scope the agent permission set to those subjects. - NATS subscription permission violations were swallowed (Subscribe returned a live-but-dead subscription). Confirm subscriptions with a server round-trip so a denial surfaces synchronously, and log async permission errors. - The backend worker connected anonymously when given a JWT without its paired seed; reject the unpaired credential instead. - The documented service-user permissions in nats-auth-setup.sh omitted prefixcache.>, which the frontend publishes and subscribes; add it. Also: add a credential-provider hook to the messaging client (consumed by the follow-up credential-lifecycle change), drop the always-nil error from NatsMessagingOptions, run go mod tidy (jwt/v2 and nkeys are now direct), and gofmt the feature's files. Tests: an agent-JWT e2e spec that connects to the enforcing NATS server and exercises every subscription the agent worker makes, plus permission allow-list coverage unit tests. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(distributed): acquire and auto-refresh worker NATS credentials Workers fetched NATS credentials once at startup, which broke two cases under JWT auth: a worker that registered while still pending admin approval never received a minted JWT (it connected unauthenticated and gave up), and a long-running worker's 24h JWT expired with no way to renew it. Introduce workerregistry.NATSCredentialManager, built on idempotent re-registration (the frontend preserves the node row and mints a fresh JWT each call): - Acquire re-registers through admin approval until the node is approved and credentials are minted (or returns the first success when auth is not required, preserving anonymous-NATS behavior). - RefreshLoop re-registers before the JWT expires (~75% of its lifetime), updating the credentials served to the connection. - Both are bounded (default 100 attempts / consecutive failures) and return an error on exhaustion, so an unapprovable or unrenewable worker exits non-zero and surfaces the problem instead of hanging or drifting toward an expired credential. The messaging client gains WithUserJWTProvider, fetching credentials on each (re)connect so the connection transparently adopts a refreshed JWT when the server expires the old one. RegisterFull exposes the approval status and full response; Register delegates to it. Both the backend worker and the agent worker are wired to this: explicit env credentials are used as-is, minted credentials are acquired-with-wait and refreshed, and a permanent refresh failure shuts the worker down so it restarts and re-acquires. Tests cover Acquire (wait-through-pending, bounded give-up, context cancel), RefreshLoop (refresh-before-expiry, bounded failure, no-expiry exit) and jwtExpiry decoding. Docs updated in distributed-mode.md. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-06-03 19:43:56 +02:00
LocalAI [bot]andEttore Di Giacinto	a44bdb29d4	feat: prefix-cache-aware routing for distributed mode (#10071) ... * feat(radixtree): generic prefix tree skeleton with longest-match Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(radixtree): Insert with path recency refresh and entry cap Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(radixtree): TTL idle-expiry and Evict sweep with branch pruning Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(radixtree): recency-weighted per-value Weight Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(radixtree): Remove all entries for a value Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(radixtree): race-free concurrency smoke test Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(radixtree): reclaim empty branches, RWMutex reads, TTL boundary, empty-key guard Address review findings on the generic prefix tree: - Extract a shared pruneWalk helper parameterized by a shouldClear predicate and use it from Evict, Remove, and the MaxEntries path. Previously evictOldestLocked cleared a victim's value but never removed the now value-less node or its childless ancestors, so internal nodes accumulated under sustained churn at the cap. The MaxEntries path now prunes the victim and its empty ancestors. - DRY: pruneWalk replaces the duplicated logic in the former pruneLocked and Remove's inner closure. - Switch Tree.mu to sync.RWMutex; LongestMatch, Weight and Len take the read lock (RLock) while Insert, Evict and Remove keep the write lock. Confirmed race-clean under go test -race. - Document the strict greater-than TTL boundary on Options.TTL and expired: age exactly equal to TTL is still live. - Guard Insert against an empty key (no-op): the root never holds a value. Adds Ginkgo specs covering MaxEntries eviction, ancestor reclamation, the no-growth-past-cap invariant, the TTL boundary, and empty-key behavior for both Insert and LongestMatch. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(prefixcache): RoutePolicy enum with parse/resolve Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(prefixcache): Config with defaults and validation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(prefixcache): deterministic xxhash prefix-chain extractor Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(prefixcache): pure filter-then-score replica selection Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(prefixcache): Provider interface and radix-tree-backed Index Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * style(prefixcache): gofmt policy enum comment alignment Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(prefixcache): head-first prefix chunking and hoist Weight out of sort Address code-quality review findings in the prefixcache package. Correctness: ExtractChain now chunks from absolute offset 0 with fixed [0,W),[W,2W),... boundaries and caps the chain to the FIRST MaxDepth head blocks. The previous tail-keeping logic shifted the byte offset by a non-window amount once a conversation grew past MaxDepth*WindowBytes, changing every hash each turn and silently breaking cross-turn longest-prefix matching. The reusable KV/prefix cache lives at the head of the prompt, so anchoring at offset 0 makes the chain a true prefix-chain: P and P+suffix share their full leading overlap. Add a regression spec proving cross-turn stability past the cap. Performance: Index.Decide precomputes each candidate's Weight once (decorate-sort-undecorate) instead of calling the O(tree size) Weight inside the O(n log n) sort comparator. Behavior is unchanged. Lint: encode prev with binary.LittleEndian.PutUint64 instead of a manual byte loop, clearing the modernize rangeint finding. Also add a concurrent Decide/Observe/Invalidate spec to exercise Index's documented concurrency safety under go test -race. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(messaging): prefixcache observe/invalidate subjects and payloads Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(prefixcache): NATS sync publish/apply for observe and invalidate Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributedhdr): ctx carrier for prefix-hash chain Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributedhdr): PrefixChainHook indirection for backend-side chain build Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend): stash prompt prefix chain on ctx before distributed routing Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(backend): mirror modelID fallback for prefix-chain salt parity Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nodes): scheduling config columns for prefix-cache routing Add RoutePolicy and per-model balance/prefix-match override columns to ModelSchedulingConfig and include them in the SetModelScheduling upsert DoUpdates list so updates are not dropped on conflict. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nodes): optional route preference in FindAndLockNodeWithModel Add a RoutePreference type and a new pref parameter so the atomic pick+lock+increment can be biased toward a preferred node without weakening atomicity. A nil preference reproduces the previous ORDER BY behavior exactly. Update the ModelRouter interface, both router.go call sites (pass nil for now; Phase 5 builds the real preference), the test doubles, and the distributed e2e caller. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(prefixcache): make Sync satisfy Provider with Evict Sync.Observe now returns whether the local index treated the assignment as new or extended, and Sync gains an Evict method that delegates to the wrapped index. Together these let SmartRouter hold a single prefixcache.Provider that broadcasts via NATS. Adds a compile-time Provider assertion and an Evict-delegates behavioral test. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nodes): prefix-cache-aware preference and observe in SmartRouter.Route Add a PrefixProvider + PrefixConfig to SmartRouterOptions/SmartRouter (nil keeps routing byte-for-byte the round-robin floor). On each request Route now calls buildPreference: it reads the prompt prefix chain from ctx (distributedhdr.PrefixChain), resolves the per-model policy/thresholds over the global config, loads candidate replica in-flight via a new registry read LoadedReplicaStats (deduped to one entry per node using the MIN in-flight across that node's replicas), asks the provider to Decide, and runs prefixcache.Select. The chosen node is passed as the RoutePreference to FindAndLockNodeWithModel on all three pick paths (cache hit, locked re-pick, cold scheduleAndLoad), and the served node is recorded via Observe only when the resolved policy is prefix_cache so round-robin models never pollute the tree. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nodes): invalidate prefix-cache entries on unload and stale removal UnloadModel and both staleness fall-through paths in Route (after a failed gRPC probe and RemoveNodeModel) now call prefixProvider.Invalidate(model, nodeID), guarded by a nil-provider check so the round-robin floor is unchanged. At runtime the provider is the *prefixcache.Sync, so invalidations also broadcast to peer frontends. Adds a test that a previously hot prefix no longer Decides to a node after UnloadModel. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(prefixcache): rolling forced-disturb pressure counter Add a concurrency-safe per-model rolling counter that tracks how many times a request had a usable hot prefix match but the load guard forced it off the warm node. Entries outside the window are dropped lazily on Count so the backing slice stays bounded. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nodes): autoscale on prefix-cache forced-disturb pressure Wire the rolling forced-disturb counter into the SmartRouter and the ReplicaReconciler. Router: in buildPreference, after Decide + Select, record a forced-disturb when a usable hot prefix match existed (d.HotNodeID != "" and d.MatchRatio >= cfg.MinPrefixMatch) but Select chose a different node (or nothing) because the load guard ruled the warm node out. This is the scale-worthy signal: the cache-warm replica is saturated. It deliberately does not fire for all-unique workloads (no hot match), avoiding false-positive scale-ups. Pressure is optional on SmartRouterOptions; nil keeps the path a no-op. Reconciler: read the same Pressure instance in reconcileModel as an extra scale-up reason, reusing the existing MaxReplicas + ClusterCapacityForModel guards and the UnsatisfiableUntil cooldown that gates the whole method. Pressure never overrides MaxReplicas and never force-evicts; a no-capacity model does not spin. Window and threshold come from prefixcache.Config (PressureWindow default 1m, PressureScaleThreshold default 1) and are configurable via ReplicaReconcilerOptions. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(prefixcache): bound Pressure slice in Record; drop dead reconciler pressureWindow Record now prunes entries older than the rolling window (the same prune Count does), via a shared pruneLocked helper, so a model that takes forced-disturb records but is never Counted (e.g. one with zero loaded replicas the reconciler skips) no longer grows its backing slice unbounded. Also removes the dead pressureWindow struct field and the ReplicaReconcilerOptions.PressureWindow option from the reconciler: they were stored but never read (the window lives inside the *prefixcache.Pressure instance). The scale block now reads pressure.Count once into a local. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(api): prefix-cache fields in scheduling endpoint DTO with validation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): prefix-cache routing controls in node scheduling form Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): wire prefix-cache index, NATS sync, and config Activates prefix-cache-aware routing in distributed mode. Builds the prefixcache Index + NATS-backed Sync + Pressure counter, installs the distributedhdr.PrefixChainHook so core/backend/llm.go attaches a prefix chain per request, subscribes to prefixcache.observe/prefixcache.invalidate to apply peers' events to the local index (no re-broadcast), threads PrefixProvider/PrefixConfig/Pressure into the SmartRouter and Pressure/PressureThreshold into the ReplicaReconciler, and runs a background eviction ticker (every TTL/2) bound to the app context. Enabled by default; --distributed-prefix-cache=false (LOCALAI_DISTRIBUTED_PREFIX_CACHE) opts out and leaves the provider/pressure nil so routing stays round-robin. --distributed-prefix-cache-ttl (LOCALAI_DISTRIBUTED_PREFIX_CACHE_TTL, default 5m) controls entry idle-timeout and eviction cadence. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(nodes): round-robin-floor invariant for prefix-cache routing Drives Select directly: a saturated hot node (in_flight 50 vs 0) is never picked even with a perfect prefix match (round-robin floor holds), while a balanced hot node within the load slack is reused. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(prefixcache): clear branch lint findings and em dashes Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): validate prefix-cache config at startup wiring Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * perf(radixtree): single-walk WeightsFor for batch value weights Add Tree.WeightsFor(values, now) which computes the recency-weighted weight for many values in a single O(N + len(values)) tree traversal, versus calling Weight once per value (O(len(values) * N)). Consumers that score K candidates against the tree under the read lock no longer pay K full walks. Extract the per-entry contribution math into an unexported helper shared by both Weight and WeightsFor so the metric stays identical (DRY). Weight's public behavior is unchanged. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(config): add ModelConfig.ModelID() single source of truth The c.Name fallback to c.Model was duplicated in core/backend/options.go (feeding model.WithModelID) and hand-copied into core/backend/llm.go (the prefix-chain salt). These MUST agree or the prefix-cache salt diverges silently from the id the model loader tracks. Consolidate both into a new config.ModelConfig.ModelID() helper and call it from both sites. Behavior is identical. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * perf(prefixcache): reuse one xxhash.Digest in ExtractChain ExtractChain allocated a fresh xxhash.New() Digest per block (up to MaxDepth per call) and grew the chain slice without preallocation. Reuse a single Digest via Reset() before each block and preallocate the chain to min(nBlocks, MaxDepth). xxhash seed 0 is stateless, so Reset()+Write produces the byte-identical value to a fresh New()+Write. Output hashes are unchanged, preserving the cross-process determinism that peers rely on over NATS. Verified by capturing ExtractChain output for the existing test inputs before and after the refactor: identical. Existing extractor tests pass unchanged. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(prefixcache): drop hot match when matched node is not a candidate; weigh cold candidates in one walk Index.Decide called radixtree.LongestMatch over the whole tree, so the deepest match could be a node that is offline, unloaded, or simply not in the passed candidate set. Honoring that as HotNodeID produced a false forced-disturb signal upstream (buildPreference records pressure when chosen != HotNodeID), making it look like a warm replica was load saturated when it was actually absent. Build the candidate set once and only set HotNodeID/MatchRatio when the matched node is an actual candidate; otherwise fall back to cold placement. A future refinement could ask the tree for the longest match restricted to the candidate nodes (shallower-but-valid) instead of dropping it. Also replace the per-candidate tree.Weight call in the cold-order sort with a single tree.WeightsFor walk, turning O(K*N) under the read lock into O(N + K). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(prefixcache): remove Select's unreachable deterministic fallback buildPreference always passes ColdOrder as a permutation of the full candidate set, so the cold-order loop hits every eligible candidate. The trailing best/bestIF scan was dead. Replace it with a plain "return """ and document that ColdOrder is guaranteed to cover all candidates, so "" means none were eligible. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): fetch model scheduling config once per Route GetModelScheduling was read three times per request - in resolveSelectorCandidates, buildPreference, and nodeMatchesScheduling - three DB round-trips for one row that is immutable for the life of the request, and not a consistent snapshot. Fetch it once near the top of Route and thread the *ModelSchedulingConfig (may be nil) into all three helpers. scheduleNewModel keeps its own fetch since it runs outside the Route snapshot. Behavior is identical for nil sched. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(autoscale): add Pressure.Reset to consume forced-disturb signal Pressure.Count is non-draining (it prunes only by age), so a single burst of forced-disturbs stays within the rolling window for the whole window and keeps Count >= threshold on every reconciler tick. The reconciler will use Reset to clear a model's events after acting on the signal so a fresh scale-up requires fresh forced-disturbs to accumulate, rather than one burst driving the model toward MaxReplicas. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(autoscale): at most one scale-up per reconcile tick, consume pressure Two autoscale bugs: 1. Over-scaling: the pressure scale-up block read Pressure.Count but never consumed it. With a non-draining counter a single forced-disturb burst kept Count >= threshold across the whole window, firing scaleUp on every tick and pushing the model toward MaxReplicas off one transient burst. After a successful pressure-triggered scale-up the reconciler now calls Pressure.Reset to consume the signal. 2. Double scale-up in one tick: the all-replicas-busy block and the pressure block could both fire in the same reconcileModel pass, each calling scaleUp(+1) against the same `current` read once at the top, so a model that was both busy and over threshold scaled +2 and could overshoot MaxReplicas by one. A scaledUp flag now enforces at most one scaleUp(+1) per tick: the pressure block is skipped if the busy block already scaled, and scale-down is skipped in any tick that scaled up. MinReplicas enforcement, UnsatisfiableUntil backoff, and capacity guards are unchanged. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(nodes): replica-removed chokepoint hook for prefix-cache invalidation Add SetReplicaRemovedHook to NodeRegistry and fire it from both RemoveNodeModel and RemoveAllNodeModelReplicas after a successful delete. This is the single chokepoint every replica-removal path funnels through (router eviction, reconciler scale-down, probe reaper, health-monitor node-down reap, RemoteUnloaderAdapter), so the prefix-cache index can be invalidated by construction rather than wiring each call site individually. The hook is stored in an atomic.Pointer so the startup wiring (setter) and the request/reconcile-time fire are race-free; it is nil-safe when unset. GORM Delete reports no error for a no-op delete, so the hook also fires when nothing was removed; the consumer's Invalidate(model, node) is idempotent so this is harmless. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): invalidate prefix-cache on any replica removal via registry hook Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(prefixcache): single source of truth for threshold bounds Extract ValidateThresholds into prefixcache/config.go so the per-model override validation (nodes.go endpoint) and Config.Validate share one implementation of the numeric bounds (min_prefix_match in [0,1], balance_abs_threshold >= 0, balance_rel_threshold == 0-or->= 1) instead of hard-coding them in two places. The route_policy allow-list stays explicit (not ParsePolicy, which maps typos to Default). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nodes): preserve prefix-cache settings on partial scheduling update A scheduling POST that omitted route_policy/thresholds (e.g. a min_replicas-only update) full-replaced every column and silently reset the model's previously-configured prefix-cache settings to empty/zero. Make the four prefix-cache request fields pointers so omitted is distinguishable from explicit zero, and merge PATCH-style in SetSchedulingEndpoint: a provided pointer wins, an omitted one preserves the existing config value (zero default when none). Non-prefix fields keep their full-replace PUT semantics. Validation now runs on the resolved values via prefixcache.ValidateThresholds. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(prefixcache): make Invalidate a no-op for uncached models and skip empty broadcasts A registry chokepoint fires Sync.Invalidate(model, nodeID) for every replica removal of every model, including round-robin models that never used the prefix cache. Index.Invalidate previously called tree(model), which lazily created and permanently retained an empty radix tree for any model that ever lost a replica, growing the trees map without bound. Sync.Invalidate also published a NATS PrefixCacheInvalidateEvent on every call, amplifying no-op removals across the cluster. Index.Invalidate now looks the tree up read-only via existingTree and returns without allocating when none exists. The Provider interface is unchanged; Sync gates the broadcast through an optional invalidateExisting(bool) capability type-asserted from the wrapped Index, falling back to the prior always-broadcast behavior for other Provider implementations. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * perf(prefixcache): derive Decide candidacy from WeightsFor and skip trivial sort WeightsFor already returns a map keyed by every requested candidate, so the separate candidates set built to validate the hot match was redundant: a node is a candidate iff it is a key in the weights map. Drop the extra map and gate the hot-match check on weights membership. Also skip the sort when there is at most one candidate, since the input order is already the cold order. Behavior is unchanged. Deferred follow-up: skipping the WeightsFor walk entirely when a hot match wins would need lazy cross-file changes and is out of scope here. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nodes): fire replica-removed hook on bulk node_models deletes; trim LoadedReplicaStats columns Bulk node-scoped node_models deletes (Register re-register cleanup, MarkOffline, MarkDraining, Deregister) removed rows directly without firing the replica-removed hook, so the prefix-cache index kept pointing at nodes whose models were gone. Capture the DISTINCT model names before each bulk delete and fire fireReplicaRemoved once per model after a successful delete, restoring the single-chokepoint invariant for all removal paths. The pre-query is skipped when no hook is set so the no-hook path stays cheap. Also narrow LoadedReplicaStats to SELECT only node_id and in_flight (the only fields the router consumer reads), dropping the JOIN-side available_vram fetch and unused columns while keeping the []ReplicaCandidate return type unchanged. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(reconciler): consume autoscale signals only on a real scale-up scaleUp was fire-and-forget (void) yet its callers unconditionally consumed the pressure signal (Pressure.Reset) and the MinReplicas hysteresis (ClearUnsatisfiable) right after calling it. If scaleUp added nothing (ScheduleAndLoadModel errored, or no node could be loaded) the saturated warm replica got no new replica AND its accumulated forced-disturb history was wiped, forcing the signal to re-accumulate over a full PressureWindow before the next attempt. Make scaleUp return whether at least one replica was actually scheduled, and gate the side effects on it: - pressure block (2b): set scaledUp and call Pressure.Reset only on success; on failure preserve the signal so the next tick retries off the same accumulated pressure. - busy-burst block (2): set scaledUp from the return value so a failed attempt does not suppress the pressure path or scale-down. - MinReplicas block: call ClearUnsatisfiable only on success so a failed attempt does not reset the unsatisfiable counter. All existing invariants (MaxReplicas, capacity gating, UnsatisfiableUntil cooldown, at-most-one-scale-up-per-tick) are preserved. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(nodes): drop router's redundant prefix-cache Invalidate calls The NodeRegistry removal chokepoint (RemoveNodeModel / RemoveAllNodeModelReplicas) now fires SetReplicaRemovedHook, which invalidates the prefix-cache index. The router was also calling prefixProvider.Invalidate explicitly right after each registry removal on the two stale-replica health-probe fall-throughs in Route and in UnloadModel, so every router-side eviction invalidated twice (double tree-prune + double NATS broadcast). Remove the three redundant explicit Invalidate calls and their empty nil-guards. Each removed call sat immediately after a registry removal that fires the hook, so invalidation is preserved via the chokepoint. Decide/Observe usage is untouched. Re-point the unit test (fake registry fires no hook) to assert the removal chokepoint is exercised on unload instead of the router's direct invalidation. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(prefixcache): broadcast invalidations unconditionally for cross-frontend coherence Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(prefixcache): reject TTL<=0 in Config.Validate (eviction ticker would panic) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(nodes): make capture+delete atomic in bulk node_models removal paths MarkOffline, MarkDraining, and the Register re-register cleanup ran the nodeModelNames SELECT and the bulk node_models DELETE as two separate statements on r.db with no transaction. A SetNodeModel landing between the two was deleted but its replica-removed hook never fired, leaving the prefix-cache index pointing at a removed replica until TTL or candidacy self-heal. Wrap the capture and the delete in a single db.Transaction in each path (mirroring how Deregister already does it). The captured model names are collected into a slice declared outside the closure; the replica-removed hook fires for each only after the transaction commits, so a rollback never invalidates the index for a removal that did not persist. The set of fired hooks now equals exactly the set of node_models rows actually deleted, with no interleaving gap. The status flip in MarkOffline/MarkDraining (setStatus) is a separate, pre-existing operation and routing already filters non-healthy nodes, so it stays outside the transaction; return contracts are unchanged. Deregister was already correct and is untouched. The cheap-path skip (no hook -> skip the SELECT) is preserved. Adds a spec asserting MarkOffline fires hooks for exactly the rows it deletes and leaves no node_models row behind (consistent snapshot). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(nodes): debug logging for prefix-cache routing decisions and observations Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(radixtree): match shared prefixes by valuing every node on insert Insert recorded the value (node id) only on the final node of the key chain, leaving every intermediate prefix node valueless. LongestMatch returns the deepest node that hasValue, so two chains that share a leading block but diverge in the tail never matched: only exact-repeat queries hit. That broke the prefix-cache routing core use cases (shared system prompt, multi-turn extension, volatile tail), all of which rely on prefix matching rather than exact-repeat. Set value/hasValue/lastSeen at every node along the chain so each prefix-block node remembers the node id that served that prefix (SGLang/vLLM-style). The deepest match wins, and the last writer owns a shared prefix node (a recency heuristic: the most recent chain through a block is the one most likely still warm). size now counts valued nodes, which is the intended meaning. Updated radixtree tests to the new semantics: deepest-prefix test uses non-overlapping chains, a new test asserts last-writer-owns-shared-node, Evict/Remove/MaxEntries expectations recomputed for per-prefix-node counting, and a shared-prefix LongestMatch red test added. Added a prefixcache Decide test proving a prefix-only query routes to the warm node. No prefixcache .go logic changed. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(distributed): lock in prefix-cache routing behavior end to end Add a DB-backed e2e spec that drives SmartRouter against a real NodeRegistry (Postgres testcontainer) and the real prefixcache.Index radix-tree provider, using a fake gRPC backend factory so no real inference runs. Covers the five behaviors validated by hand: 1. Cold miss + observe: an unseen prefix chain cold-places and is recorded. 2. Hot-match affinity: the same chain returns to its warm node X. 3. Shared-prefix match: a divergent chain sharing X's leading prefix still routes to X (the radix-tree regression we fixed). 4. Negative control: an unrelated chain is a cold miss, not a false hot match on X. 5. Failover + invalidation: removing X's replica fires the registry chokepoint hook to invalidate the prefix entry, and the chain fails over to surviving node Y and re-homes there. Replaces the need for manual docker-compose re-runs. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(prefixcache): make prefix-cache affinity replica-granular Track prefix-cache affinity per loaded replica (a backend process with its own KV cache) instead of per node, so multiple replicas of the same model on one node each keep distinct affinity and a hot prefix routes back to the exact replica that served it. - radixtree: add RemoveFunc(pred) and reimplement Remove on top of it. - prefixcache: introduce ReplicaKey{NodeID, Replica}; Index/Candidate/ PrefixDecision/Select/Provider now key on ReplicaKey. Add InvalidateNode to drop every replica of a node; Invalidate drops one replica. Select returns (ReplicaKey, bool) and gains a deterministic least-in-flight eligible fallback (tiebreak NodeID then Replica). - messaging: carry Replica on PrefixCacheObserveEvent and PrefixCacheInvalidateEvent (Replica < 0 means all replicas of the node). - Sync delegates + broadcasts with replica; InvalidateNode broadcasts Replica=-1; ApplyInvalidate routes negative replica to InvalidateNode. This is part 1 of 2; the registry/router/wiring consumers are updated separately. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): make prefix-cache routing replica-granular Wire the SmartRouter, NodeRegistry, and distributed startup to the replica-keyed prefixcache API. Affinity is now tracked per replica (each replica is a separate process with its own KV cache), so a prefix served by (node,0) no longer leaks onto the same-node sibling (node,1). - RoutePreference gains PreferredReplica; FindAndLockNodeWithModel locks the EXACT (node_id, replica_index) row, falling through to the default ORDER BY when that replica is not loaded. - SetReplicaRemovedHook now carries replicaIndex; RemoveNodeModel fires the specific replica, RemoveAllNodeModelReplicas and the four bulk node-scoped deletes fire replica<0 (all replicas of the node). - buildPreference builds one Candidate per loaded replica and locks the exact replica the policy chose; observePrefix records the served ReplicaKey at every call site. - distributed.go routes the hook to InvalidateNode (replica<0) or Invalidate(key). - Tests updated to the replica-keyed API plus new coverage: a hot prefix on (node,0) prefers replica 0 over the same-node sibling (router unit + e2e), and FindAndLock locks the exact preferred replica. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): derive prefix chain from messages for tokenizer-template models Prefix-cache-aware routing built its prompt-prefix chain from the rendered prompt string `s` in ModelInference. For models with TemplateConfig.UseTokenizerTemplate the frontend never renders a prompt - the backend tokenizes the structured messages itself - so `s` is empty, the chain is empty, and routing silently falls back to round-robin. That covers the bulk of modern chat models (qwen3, llama3, ...), so the feature effectively never engaged for them. Fall back to messagesPrefixSource(messages): a deterministic, prefix-stable head-first serialization of the conversation (role + content per turn). Two requests sharing a leading system prompt and early turns share a leading byte prefix, which ExtractChain maps to a shared chain prefix - landing both on the same cache-warm replica. The rendered `s` is still preferred when present (higher fidelity for non-template models). Found via the multi-replica-per-node e2e: zero "prefix-cache routing decision" logs despite per-request Route calls, traced to the empty-chain guard. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(distributed): document prefix-cache routing roadmap Add a routing-and-caching roadmap section to the distributed-mode guide, linking the epic (#10063) and the follow-up issues (#10064-#10070) surfaced from a survey of SGLang, vLLM production-stack, Ray Serve, llm-d, AIBrix, and NVIDIA Dynamo. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-30 23:24:22 +02:00
Richard Palethorpe	fbcd886a47	fix(application): stop backend processes synchronously on shutdown (#10058) ... application.New wires a fire-and-forget goroutine that runs StopAllGRPC + distributed.Shutdown when the app context is cancelled. Callers (tests, CLI signal handler) cancel the context and then exit immediately, so the test binary / process can terminate before that goroutine kills the spawned backend children. go-processmanager sets no Pdeathsig, so the orphans are reparented to init and survive — leaving dozens of stray mock-backend processes after an e2e run. Add Application.Shutdown(), which runs the same cleanup synchronously on the caller's stack and is idempotent via sync.Once. The context-cancel goroutine, the CLI signal handler, and the test suites all call it, so cleanup is deterministic and the duplicated teardown logic collapses to one place. The async goroutine remains as a safety net for callers that forget; sync.Once dedupes the double call. Wire e2e_suite_test and the two mock-backend Contexts in app_test to call Shutdown in their AfterSuite/AfterEach. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-05-29 11:40:43 +02:00
Richard Palethorpe	6a80e23733	feat(middleware): Model routing, PII filtering, Cloud model proxies (#9802) ... Add a routing middleware stack and a cloud-proxy backend. * cloud-proxy: a Go gRPC backend that forwards OpenAI- and Anthropic-shaped chat requests to upstream providers, with an optional translate mode (OpenAI request -> Anthropic /v1/messages -> OpenAI response) and full tool-calling support. * routing: admission control, content-aware model routing (embedding cache + classifier + rerank + Arch-Router score), PII detection/redaction (regex + NER) with streaming filter and OpenAI/Anthropic adapters, and a per-user/per-key billing recorder backed by GORM or in-memory storage. * middleware: UsageMiddleware records usage via the billing recorder, plus admission, route-model, usage-stamp and trace middlewares. * observability: BackendTrace ring buffer stores full request bodies (capped), MITM proxy emits structured trace events, and router classifier decisions surface at /api/router/decide. * gallery: Arch-Router-1.5B (Q4_K_M and Q8_0). * UI: cloud-proxy model-editor fields, classifier system-prompt and score-normalization config, and a Traces page rendering request bodies. Assisted-by: claude-code:claude-opus-4-7 [Read] [Edit] [Bash] Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-05-25 09:28:27 +02:00
LocalAI [bot]andEttore Di Giacinto	a0f3e26245	fix(distributed): make admin backend installs resilient and observable (#9958) ... * feat(distributed): add configurable NATS backend install/upgrade timeouts Adds BackendInstallTimeout and BackendUpgradeTimeout to DistributedConfig with 15m defaults, following the existing MCPToolTimeout / WorkerWaitTimeout pattern. These will replace the hardcoded literals in RemoteUnloaderAdapter so admin-driven backend installs across the cluster survive long OCI image pulls that previously timed out at 3m. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * style(distributed): gofmt alignment after timeout fields Re-aligns the Validate() negative-duration map and the Default* const block so the new BackendInstall/UpgradeTimeout entries do not leave the surrounding columns mis-padded. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(cli): surface LOCALAI_NATS_BACKEND_INSTALL_TIMEOUT and _UPGRADE_TIMEOUT Parses the two new env vars on the run CLI and threads them through the existing AppOption builder so DistributedConfig picks them up. Invalid duration strings now fail loudly at startup rather than silently falling back to the default. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): inject NATS install/upgrade timeouts into RemoteUnloaderAdapter Removes the hardcoded 3m / 15m literals from RemoteUnloaderAdapter and threads in DistributedConfig.BackendInstallTimeoutOrDefault() and BackendUpgradeTimeoutOrDefault() at construction. Install now defaults to 15m (was 3m); cold OCI image pulls on Jetson Wi-Fi routinely blew past the old ceiling. Scripted messaging client captures the timeout so tests can assert the configured value actually reaches the NATS request. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): introduce galleryop.ErrWorkerStillInstalling sentinel When the NATS request-reply for backend.install (or .upgrade) times out the worker is almost always still pulling the OCI image. Wrap the timeout in a typed sentinel so the manager above can distinguish "worker hung" from "worker still working" and leave the pending_backend_ops row in place for the reconciler to confirm via backend.list. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): treat NATS install timeout as in-progress, not failure When a worker times out replying to backend.install but the install is still running on the worker, enqueueAndDrainBackendOp now reports a running_on_worker status and pushes NextRetryAt out by the install timeout so the reconciler does not immediately re-fire another install while the worker is still pulling the image. The pending_backend_ops row stays in place for the next reconciler pass to confirm via backend.list. InstallBackend wraps the result in galleryop.ErrWorkerStillInstalling so callers can branch (galleryop renders yellow in-progress instead of red error). UpgradeBackend uses the same wrap. Adds RemoteUnloaderAdapter.InstallTimeout() so the manager can push NextRetryAt by the configured timeout without reaching into a private field, and NodeRegistry.RecordPendingBackendOpInFlight as the soft cousin of RecordPendingBackendOpFailure. Also includes incidental gofmt-driven struct-field alignment in registry.go on lines unrelated to the change (touched files are re-formatted to canonical form per project policy). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): don't increment Attempts on in-flight install timeout An in-flight timeout (worker still pulling the OCI image) is not a failed attempt, it's a delayed one. Incrementing Attempts let genuinely-progressing slow installs (e.g. 30 GB CUDA images on Wi-Fi) trip the reconciler's maxPendingBackendOpAttempts cap and dead-letter the queue row while the worker was still legitimately working. RecordPendingBackendOpInFlight now only updates LastError and NextRetryAt. Also documents "running_on_worker" in the NodeOpStatus.Status enum comment so Task 6 implementers see the full surface. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(galleryop): surface ErrWorkerStillInstalling as non-error OpStatus When the distributed backend manager returns an error that wraps ErrWorkerStillInstalling, backendHandler now completes the op with a "still installing in background" message rather than marking it as a red failure. Admin UI sees a yellow in-progress state; reconciler confirms completion on its next pass. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(distributed): end-to-end install-timeout-then-reconcile Wires Task 1-6 end-to-end so any seam mismatch surfaces in CI rather than during a real cluster install. NATS times out, the queue row stays alive with running_on_worker status, the worker eventually reports the backend installed via backend.list, the manager surfaces it via ListBackends. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(distributed): document LOCALAI_NATS_BACKEND_INSTALL_TIMEOUT / _UPGRADE_TIMEOUT Add the two new operator-tunable env vars to the Frontend Configuration table in the distributed-mode docs. Explains the 15m default, when to raise it (slow links pulling multi-GB OCI images), and the new "still installing in background" admin-UI state when the round-trip times out but the worker is still working. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): clear pending install rows when backend.list confirms DistributedBackendManager.ListBackends now proactively clears pending_backend_ops install rows whose (nodeID, backend) is reported installed by backend.list. Operator UI updates immediately instead of waiting up to installTimeout (default 15m) for the next reconciler tick after NextRetryAt. Only install rows are cleared; upgrade and delete intents are not satisfied by presence in backend.list and continue to drain through their normal reconciler paths. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(messaging): add BackendInstallProgressEvent wire type and subject New NATS subject nodes.<nodeID>.backend.install.<opID>.progress lets the worker publish transient progress events (file, current/total bytes, percentage, phase) while a long-running install pulls its OCI image. BackendInstallRequest gains an optional OpID field so the worker knows which subject to publish on. Transient pub/sub (not JetStream): the install reply remains ground truth for success/failure; dropped progress events are tolerable. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * style(messaging): drop em-dash from BackendInstallProgress test comment Per project convention (no em-dashes anywhere). Comment substance is unchanged. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): worker publishes debounced install progress over NATS When BackendInstallRequest.OpID is set, the worker's backend.install handler wires a debounced publisher (250ms window) into the gallery download callback. Each tick becomes a BackendInstallProgressEvent on nodes.<nodeID>.backend.install.<opID>.progress; the publisher always emits a final event on Flush so the UI sees the terminal percentage. Old masters that do not set OpID continue to run silent installs: no behavior change for them. Lock ordering: the publisher releases its mutex before calling messaging.Publish so a slow network never stalls the install loop. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): RemoteUnloaderAdapter subscribes to install progress InstallBackend gains opID + onProgress parameters. When both are set, the adapter subscribes to nodes.<nodeID>.backend.install.<opID>.progress BEFORE publishing the install request, decodes each message into the caller's onProgress callback in a goroutine (so a slow callback never stalls the NATS reader thread), and unsubscribes after RequestJSON returns. When onProgress is nil OR opID is empty (the reconciler retry path), subscription is skipped entirely - silent installs cost nothing extra. Subscribe failure is logged at Warn and the install proceeds without progress streaming; the NATS round-trip still owns terminal status. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): forward backend install progress into galleryop OpStatus DistributedBackendManager.InstallBackend now passes the gallery op ID and a progress bridge into the adapter call. Each BackendInstallProgressEvent from the worker becomes a galleryop.ProgressCallback tick - which the existing backendHandler already turns into OpStatus.UpdateStatus, so the admin UI/SSE polling sees per-byte progress for distributed installs without any UI-side change. UpgradeBackend is intentionally left silent for now: its wire request (BackendUpgradeRequest) does not carry OpID, and rolling-update fallback is the rarer path. Will be picked up in a follow-up if the worker upgrade path also gets a progress channel. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(distributed): InstallBackend tolerates silent (pre-Phase-2) workers A worker on pre-Phase-2 code never publishes progress events. The new master subscribes optimistically; this spec pins that a silent worker still produces a green install with no progressCb ticks. The install reply is the source of truth for terminal state; the progress stream is a best-effort UX enrichment. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(distributed): document install progress streaming Note the new nodes.<nodeID>.backend.install.<opID>.progress subject and the silent-worker compatibility behavior so operators know to expect real-time progress and what happens on a mixed-version cluster. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(distributed): note progress-event ordering trade-off in InstallBackend Document near the goroutine dispatch why ordering at the consumer is best-effort, why it rarely matters in practice (worker debounce >> goroutine jitter), and what a future hardening pass would look like (Seq field + stale-by-seq drop). Stops the next reader from accidentally "fixing" the goroutine pool away. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(galleryop): add NodeProgress + OpStatus.Nodes for per-node breakdown Adds the data model the UI needs to render an expandable per-node breakdown of a fanned-out backend install. NodeProgress carries node identity (ID + name), per-node status (queued / running_on_worker / success / error / downloading), the current file + bytes + percentage from the Phase 2 progress stream, and any per-node error. OpStatus.Nodes is the slice the /api/operations handler will surface in a follow-up. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(galleryop): UpdateNodeProgress merges per-node ticks by NodeID GalleryService.UpdateNodeProgress(opID, nodeID, np) merges a NodeProgress into OpStatus.Nodes (keyed by NodeID, no duplicates) and mirrors the latest tick into the aggregate Progress / FileName / DownloadedFileSize / TotalFileSize fields so the legacy single-bar OperationsBar view keeps working unchanged alongside the new per-node breakdown. Concurrent-safe via the existing g.Mutex. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): write per-node OpStatus entries during install fan-out DistributedBackendManager now accepts a nodeProgressSink and feeds it two streams: 1. enqueueAndDrainBackendOp emits a per-node terminal entry on each status it appends to BackendOpResult (queued, success, error, running_on_worker). The opID is threaded through the function so the sink gets the right gallery op identity. 2. The install apply closure fans each BackendInstallProgressEvent into the sink as a downloading entry, alongside the legacy progressCb path so the aggregate single-bar view stays correct. Production wiring passes the GalleryService (which implements UpdateNodeProgress via Task 2) as the sink. Single-node tests pass nil. DeleteBackend and UpgradeBackend pass an empty opID so the sink path no-ops for ops that aren't gallery-tracked the same way as Install. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(operations): expose per-node breakdown on /api/operations When an operation's OpStatus has Nodes entries (populated by the Phase 4 progress sink wiring), surface them as a "nodes" array on the /api/operations response, sorted by node_name for stable rendering. Backward compatible: legacy clients ignore the field; ops without any node entries (single-node mode, model installs) omit the array entirely thanks to the empty-slice guard. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): per-node breakdown in OperationsBar When an install op fans out to more than one worker, the operations bar now shows a "N nodes" chevron that expands into a per-node list. Each row carries the node's status (color-coded pill), the current file being downloaded, byte counts, percentage, and a thin per-node progress bar. Yellow "Worker busy" pill marks running_on_worker status with a tooltip explaining the NATS round-trip timed out but the worker is still installing in the background. Backward compatible: ops without a nodes field (legacy or single-node mode) render as before. State for expand/collapse is local to the component, keyed by jobID/id - reload starts collapsed. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(distributed): document per-node breakdown in the operations bar Adds a short subsection covering the expandable "N nodes" chevron in the OperationsBar admin UI, the meaning of each status pill, and how it relates to the /api/operations nodes array. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(galleryop): UpdateStatus preserves Nodes when caller sends none Real-world bug surfaced by the Phase 4 multi-worker smoke test: the nodes[] array in /api/operations flickered between a single node at a time on a 2-worker install. Root cause: the Phase 2 progress bridge also calls the legacy progressCb -> UpdateStatus(&OpStatus{...}) on every tick. UpdateStatus then overwrote the entire status pointer, wiping the Nodes slice that UpdateNodeProgress had just merged in. Fix: in UpdateStatus, if the incoming op has an empty Nodes slice, carry forward the previous status's Nodes before storing. Callers that explicitly populate Nodes still win (their slice replaces the prior one, no merge across the two code paths). Two regression specs added pinning both directions of the contract. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(distributed): strip implementation details from user-facing docs Trim the new install/upgrade timeout rows and the install-progress sections to focus on what the operator sees and tunes. Drops: - the NATS subject names and pub/sub mechanics - "round-trip" / reconciler / backend.list jargon - /api/operations polling cadence - "pre-2026-05-22" version references Reframes the breakdown text around the admin UI (Operations Bar, chevron, status pills, "Worker busy" tooltip). Implementation context lives in the agent notes and code comments. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(config): move DistributedConfig.Validate flag names to constants The negative-duration check map was a wall of literal kebab-case strings that had to stay in sync with the kong-derived CLI flag names manually. Move them to a Flag* const block alongside the existing Default* block so a rename of either the Go field or the CLI naming convention forces a compile error rather than silent drift. Sole consumer today is Validate; the constants are exported so future operator-facing surfaces (e.g. error messages on other validation paths) can reference them by name instead of repeating the literals. Tests pin both the literal values (so a future "let's just rename this" doesn't accidentally regress the CLI flag) and the negative- duration error message for the new BackendInstall / BackendUpgrade fields. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(distributed): extract NodeStatus and Phase enums to constants Sweep for the same literal-string-as-identifier pattern called out on the Validate flag names: the per-node install status enum ("queued" | "downloading" | "running_on_worker" | "success" | "error") appeared as raw literals across managers_distributed.go (10+ sites, including 3 separate `n.Status == "running_on_worker"` checks), operation.go, and the test suite. Same shape for the Phase enum ("resolving" | "downloading" | "extracting" | "starting") in the worker-side progress publisher. Promote both to exported const blocks: - galleryop.NodeStatus{Queued,Downloading,RunningOnWorker,Success,Error} shared between galleryop.NodeProgress.Status (the wire field) and nodes.NodeOpStatus.Status (the in-process per-node summary) - messaging.Phase{Resolving,Downloading,Extracting,Starting} shared between the worker publisher and any future consumer that needs to switch on phase Tests pin both the literal values (so a future "let's just rename" doesn't silently change the JSON wire) and use the constants in setup (so the producer side stays drift-protected). Wire-format assertions on the /api/operations JSON output keep their literals deliberately, so the constant value can never silently diverge from what the UI receives. Out of scope for this PR (separate cleanup): the finetune and quantization job-status enums have the same anti-pattern with 14+ literal sites each, but predate this PR's work. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-23 12:35:44 +02:00
LocalAI [bot]andEttore Di Giacinto	e5d7b84216	fix(distributed): split NATS backend.upgrade off install + dedup loads (#9717) ... * feat(messaging): add backend.upgrade NATS subject + payload types Splits the slow force-reinstall path off backend.install so it can run on its own subscription goroutine, eliminating head-of-line blocking between routine model loads and full gallery upgrades. Wire-level Force flag on BackendInstallRequest is kept for one release as the rolling-update fallback target; doc note marks it deprecated. Assisted-by: Claude:claude-sonnet-4-6 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed/worker): add per-backend mutex helper to backendSupervisor Different backend names lock independently; same backend serializes. This is the synchronization primitive used by the upcoming concurrent install handler — without it, wrapping the NATS callback in a goroutine would race the gallery directory when two requests target the same backend. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed/worker): run backend.install handler in a goroutine NATS subscriptions deliver messages serially on a single per-subscription goroutine. With a synchronous install handler, a multi-minute gallery download would head-of-line-block every other install request to the same worker — manifesting upstream as a 5-minute "nats: timeout" on unrelated routine model loads. The body now runs in its own goroutine, with a per-backend mutex (lockBackend) protecting the gallery directory from concurrent operations on the same backend. Different backend names install in parallel. Backward-compat: req.Force=true is still honored here, so an older master that hasn't been updated to send on backend.upgrade keeps working. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed/worker): subscribe to backend.upgrade as a separate path Slow force-reinstall now lives on its own NATS subscription, so a multi-minute gallery pull cannot head-of-line-block the routine backend.install handler on the same worker. Same per-backend mutex guards both — concurrent install + upgrade for the same backend serialize at the gallery directory; different backends are independent. upgradeBackend stops every live process for the backend, force-installs from gallery, and re-registers. It does not start a new process — the next backend.install will spawn one with the freshly-pulled binary. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(distributed): add UpgradeBackend on NodeCommandSender; drop Force from InstallBackend Master now sends to backend.upgrade for force-reinstall, with a nats.ErrNoResponders fallback to the legacy backend.install Force=true path so a rolling update with a new master + an old worker still converges. The Force parameter leaves the public Go API surface entirely — only the internal fallback sets it on the wire. InstallBackend timeout drops 5min -> 3min (most replies are sub-second since the worker short-circuits on already-running or already-installed). UpgradeBackend timeout is 15min, sized for real-world Jetson-on-WiFi gallery pulls. Updates the admin install HTTP endpoint (core/http/endpoints/localai/nodes.go) to the new signature too. router_test.go's fakeUnloader does not yet implement the new interface shape; Task 3.2 will catch it up before the next package-level test run. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(distributed): update fakeUnloader for new NodeCommandSender shape InstallBackend lost its force bool param (Force is not part of the public Go API anymore — only the internal upgrade-fallback path sets it on the wire). UpgradeBackend gained a method. Fake records both call slices and provides an installHook concurrency seam for upcoming singleflight tests. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(distributed): cover UpgradeBackend's new subject + rolling-update fallback Task 3.1 changed the master to publish UpgradeBackend on the new backend.upgrade subject; the existing UpgradeBackend tests scripted the old install subject and so all 3 began failing as expected. Updates them to script SubjectNodeBackendUpgrade with BackendUpgradeReply. Adds two new specs for the rolling-update fallback: - ErrNoResponders on backend.upgrade triggers a backend.install Force=true retry on the same node. - Non-NoResponders errors propagate to the caller unchanged. scriptedMessagingClient gains scriptNoResponders (real nats sentinel) and scriptReplyMatching (predicate-matched canned reply, used to assert that the fallback path actually sets Force=true on the install retry). Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(distributed): coalesce concurrent identical backend.install via singleflight Six simultaneous chat completions for the same not-yet-loaded model were observed firing six independent NATS install requests, each serializing through the worker's per-subscription goroutine and amplifying queue depth. SmartRouter now wraps the NATS round-trip in a singleflight.Group keyed by (nodeID, backend, modelID, replica): N concurrent identical loads share one round-trip and one reply. Distinct (modelID, replica) keys still fire independent calls, so multi-replica scaling and multi-model fan-out are unaffected. fakeUnloader gains a sync.Mutex around its recording slices to keep concurrent test goroutines race-clean. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e/distributed): drop force arg from InstallBackend test calls Two e2e test call sites still passed the trailing force bool that was removed from RemoteUnloaderAdapter.InstallBackend in 9bde76d7. Caught by golangci-lint typecheck on the upgrade-split branch (master CI was already green because these tests don't run in the standard test path). Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(distributed): extract worker business logic to core/services/worker core/cli/worker.go grew to 1212 lines after the backend.upgrade split. The CLI package was carrying backendSupervisor, NATS lifecycle handlers, gallery install/upgrade orchestration, S3 file staging, and registration helpers — all distributed-worker business logic that doesn't belong in the cobra surface. Move it to a new core/services/worker package, mirroring the existing core/services/{nodes,messaging,galleryop} pattern. core/cli/worker.go shrinks to ~19 lines: a kong-tagged shim that embeds worker.Config and delegates Run. No behavior change. All symbols stay unexported except Config and Run. The three worker-specific tests (addr/replica/concurrency) move with the code via git mv so history follows them. Files split as: worker.go - Run entry point config.go - Config struct (kong tags retained, kong not imported) supervisor.go - backendProcess, backendSupervisor, process lifecycle install.go - installBackend, upgradeBackend, findBackend, lockBackend lifecycle.go - subscribeLifecycleEvents (verbatim, decomposition is a follow-up commit) file_staging.go - subscribeFileStaging, isPathAllowed registration.go - advertiseAddr, registrationBody, heartbeatBody, etc. reply.go - replyJSON process_helpers.go - readLastLinesFromFile Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(distributed/worker): decompose subscribeLifecycleEvents into per-event handlers The 226-line subscribeLifecycleEvents method packed eight NATS subscriptions inline. Each grew context-shaped doc comments mixed with subscription plumbing, making it hard to read any one handler without scrolling past the others. Extract each handler into its own method on *backendSupervisor; the subscriber becomes a thin 8-line dispatcher. No behavior change: each method body is byte-equivalent to its corresponding inline goroutine + handler. Doc comments that were attached to the inline SubscribeReply calls migrate to the new method godocs. Adding the next NATS subject is now a 2-line patch to the dispatcher plus one new method, instead of grafting onto a monolith. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-08 16:24:54 +02:00
LocalAI [bot]andEttore Di Giacinto	2be07f61da	feat(whisper): honor client cancellation via ggml abort_callback (#9710) ... * refactor(transcription): propagate request ctx through ModelTranscription* Replaces context.Background() with the HTTP request ctx so client disconnects start cancelling the gRPC call. No backend-side abort wiring yet — that comes in a later commit. Pure plumbing. Assisted-by: Claude:claude-haiku-4-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(cli): pass ctx to backend.ModelTranscription Follow-up to e65d3e1f which threaded ctx through ModelTranscription but missed the CLI caller. CLI commands have no request-scoped ctx, so context.Background() is correct here. Assisted-by: Claude:claude-haiku-4-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(audio): propagate request ctx into TTS, sound-gen, audio-transform Same ctx-plumbing pattern applied to the rest of the audio path. CLI callers use context.Background() since there is no request scope; HTTP callers use c.Request().Context(). Assisted-by: Claude:claude-haiku-4-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(backend): propagate request ctx into biometric, detection, rerank, diarization paths Replaces remaining context.Background() sites in core/backend with the caller's ctx. After this commit, every core/backend/*.go entry point threads the request ctx end-to-end to the gRPC client. Assisted-by: Claude:claude-haiku-4-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(grpc): plumb ctx through AIModel.AudioTranscription{,Stream} Adds context.Context as first parameter to the AIModel interface methods that wrap whisper-style transcription. Server-side gRPC handler now forwards the per-RPC ctx (server-streaming uses stream.Context()). Whisper, Voxtral, vibevoice-cpp, and sherpa-onnx accept the parameter; none uses it yet — the actual cancellation primitive lands in the next commit so this is pure plumbing. Assisted-by: Claude:claude-sonnet-4-6 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(whisper): add abort_callback hook in the C++ bridge Installs a std::atomic<int> flag, wires it into whisper_full_params.abort_callback, and exposes a set_abort(int) C symbol so Go can flip the flag from a goroutine watching the request context. transcribe() now distinguishes abort (return 2) from real whisper_full failure (return 1). Assisted-by: Claude:claude-haiku-4-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(whisper): register set_abort symbol in the purego loader Adds the Go-side binding for the new C export so the next commit can call CppSetAbort(1) from a watcher goroutine on ctx.Done(). Assisted-by: Claude:claude-haiku-4-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(whisper): honor ctx cancellation and return codes.Canceled A watcher goroutine watches ctx.Done() during AudioTranscription and calls CppSetAbort(1) on cancel. whisper_full sees abort_callback return true at the next compute graph step, returns non-zero, and the bridge returns 2 -> AudioTranscription maps that to codes.Canceled. Adds an opt-in test (gated on WHISPER_MODEL_PATH / WHISPER_AUDIO_PATH) that asserts cancellation latency under 5s and proves the abort flag resets cleanly so the next transcription succeeds. Assisted-by: Claude:claude-sonnet-4-6 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(whisper): join the cancel watcher goroutine before returning Follow-up to 85edf9d2. The previous commit used `defer close(done)` and called the watcher "joined synchronously" — but close() only signals, it does not block until the goroutine exits. That left a window where a late CppSetAbort(1) from a cancelled call could land on the next call, after its C-side g_abort reset but before whisper_full() began polling the abort callback, corrupting the second transcription. Switch to a sync.WaitGroup join so wg.Wait() blocks until the watcher has actually returned from its select. Assisted-by: Claude:claude-sonnet-4-6 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(whisper): short-circuit pre-cancelled ctx in AudioTranscription If ctx is already Done() at entry, return codes.Canceled immediately instead of running the full transcription. The C-side g_abort reset happens at the start of transcribe() and would otherwise overwrite a watcher-set abort flag from an already-cancelled ctx, producing a spurious successful transcription on a request the client has already abandoned. Assisted-by: Claude:claude-haiku-4-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(tests/distributed): update testLLM mock for new AudioTranscription signature Phase B (93c48e19) added context.Context to AIModel.AudioTranscription but missed the testLLM mock in tests/e2e/distributed. CI golangci-lint caught it: *testLLM did not implement grpc.AIModel because the method signature lacked the ctx parameter, which broke the distributed test suite compilation and cascaded through every backend-build job that runs `go build ./...`. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(whisper): port cancellation test to Ginkgo/Gomega Project policy (.agents/coding-style.md, enforced by golangci-lint forbidigo) is that all Go tests must use Ginkgo v2 + Gomega — no stdlib testing patterns (t.Skip, t.Fatalf, etc.). Convert the cancellation test to a Describe/It block with Skip(...) for env gating and Expect/HaveOccurred for assertions. Same coverage: cancel mid-flight returns codes.Canceled within 5s and a follow-up transcription succeeds, proving the C-side g_abort flag resets cleanly. Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-08 01:44:47 +02:00
LocalAI [bot]andEttore Di Giacinto	447c186089	fix(distributed): make backend upgrade actually re-install on workers (#9708) ... * fix(distributed): make backend upgrade actually re-install on workers UpgradeBackend dispatched a vanilla backend.install NATS event to every node hosting the backend. The worker's installBackend short-circuits on "already running for this (model, replica) slot" and returns the existing address — so the gallery install path was skipped, no artifact was re-downloaded, no metadata was written. The frontend's drift detection then re-flagged the same backends every cycle (installedDigest stays empty → mismatch → "Backend upgrade available (new build)") while "Backend upgraded successfully" landed in the logs at the same time. The user-visible symptom: clicking "Upgrade All" silently does nothing and the same N backends sit on the upgrade list forever. Two coupled fixes, one PR: 1. Force flag on backend.install. Add `Force bool` to BackendInstallRequest and thread it through NodeCommandSender -> RemoteUnloaderAdapter. UpgradeBackend (and the reconciler's pending-op drain when retrying an upgrade) sets force=true; routine load events and admin install endpoints keep force=false. On the worker, force=true stops every live process that uses this backend (resolveProcessKeys for peer replicas, plus the exact request processKey), skips the findBackend short-circuit, and passes force=true into gallery.InstallBackendFromGallery so the on-disk artifact is overwritten. After the gallery install completes, startBackend brings up a fresh process at the same processKey on a new port. 2. Liveness check on the fast path. installBackend's "already running" branch read getAddr without verifying the process was alive, so a gRPC backend that died without the supervisor noticing left a stale (key, addr) entry. The reconciler then dialed that address, got ECONNREFUSED, marked the replica failed, retried install — and the supervisor said "already running addr=…" again. Loop forever, exactly what we observed on a node whose llama-cpp process had died but whose supervisor record persisted. Verify s.isRunning(processKey) before trusting getAddr; if the entry is stale, stopBackendExact cleans up and we fall through to a real install. Backwards-compatible: the new Force field is omitempty, older workers ignore it (their default behavior matches force=false). The signature change on NodeCommandSender.InstallBackend is internal-only. Verified: unit tests in core/services/nodes pass (108s suite). The pre-existing core/backend build break (proto regen pending for word-level timestamps) blocks core/cli and core/http/endpoints/localai package tests but is unrelated to this change. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-7 [Claude Code] * test(e2e/distributed): pass force=false to adapter.InstallBackend NodeCommandSender.InstallBackend gained a final force bool in the upgrade-force commit; the e2e distributed lifecycle tests still called the old 8-arg signature and broke compilation. These tests exercise the routine install path (single replica, default behavior), so force=false preserves their existing semantics. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-7 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-07 17:28:14 +02:00
LocalAI [bot]andEttore Di Giacinto	70cf8ac546	fix(backend): resolve relative draft_model paths against the models dir (#9680) ... * fix(backend): resolve relative draft_model paths against the models dir The main model file and mmproj are joined with the configured models directory before reaching the backend, but draft_model was sent verbatim. With a relative draft_model in the YAML config, llama.cpp opens the path from the backend process's CWD and fails with "No such file or directory", forcing users to hard-code an absolute path. Mirror the existing mmproj resolution: if draft_model is relative, join it with modelPath. Absolute paths are passed through unchanged. Adds an e2e regression test against the mock backend that asserts the main model file, mmproj, and draft_model all arrive at the backend resolved to absolute paths. Closes #9675 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-7-1m [Read] [Edit] [Bash] [Write] * fix(backend): always join draft_model with models dir (drop IsAbs shortcut) The previous commit kept absolute draft_model paths intact via an IsAbs check. That left a path-traversal vector open: a user-supplied YAML config could set draft_model to /etc/passwd (or any other host file the backend process can read) and the path would be sent through unchanged. filepath.Join cleans the leading slash from absolute components, so joining unconditionally — the way mmproj already does — keeps the result rooted at the configured models directory regardless of input. Adds a second e2e spec that feeds an absolute draft_model into the mock backend and asserts the path is clamped under modelsPath. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-7-1m [Read] [Edit] [Bash] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-06 00:58:38 +02:00
Richard Palethorpe	8e43842175	feat(vllm, distributed): tensor parallel distributed workers (#9612) ... * feat(vllm): build vllm from source for Intel XPU Upstream publishes no XPU wheels for vllm. The Intel profile was silently picking up a non-XPU wheel that imported but errored at engine init, and several runtime deps (pillow, charset-normalizer, chardet) were missing on Intel -- backend.py crashed at import time before the gRPC server came up. Switch the Intel profile to upstream's documented from-source procedure (docs/getting_started/installation/gpu.xpu.inc.md in vllm-project/vllm): - Bump portable Python to 3.12 -- vllm-xpu-kernels ships only a cp312 wheel. - Source /opt/intel/oneapi/setvars.sh so vllm's CMake build sees the dpcpp/sycl compiler from the oneapi-basekit base image. - Hide requirements-intel-after.txt during installRequirements (it used to 'pip install vllm'); install vllm's deps from a fresh git clone of vllm via 'uv pip install -r requirements/xpu.txt', swap stock triton for triton-xpu==3.7.0, then 'VLLM_TARGET_DEVICE=xpu uv pip install --no-deps .'. - requirements-intel.txt trimmed to LocalAI's direct deps (accelerate / transformers / bitsandbytes); torch-xpu, vllm, vllm_xpu_kernels and the rest come from upstream's xpu.txt during the source build. - requirements.txt: add pillow + charset-normalizer + chardet -- used by backend.py and missing on the Intel install profile. - run.sh: 'set -x' so backend startup is visible in container logs (the gRPC startup error path was previously opaque). Also adds a one-line docs example for engine_args.attention_backend under the vLLM section, since older XE-HPG GPUs (e.g. Arc A770) need TRITON_ATTN to bypass the cutlass path in vllm_xpu_kernels. Tested end-to-end on an Intel Arc A770 with Qwen2.5-0.5B-Instruct via LocalAI's /v1/chat/completions. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(vllm): add multi-node data-parallel follower worker vLLM v1's multi-node story is one process per node sharing a DP coordinator over ZMQ -- the head runs the API server with data_parallel_size > 1 and followers run `vllm serve --headless ...` with matching topology. Today LocalAI can already configure DP on the head via the engine_args YAML map, but there's no way to bring up the follower nodes -- so the head sits waiting for ranks that never handshake. Add `local-ai p2p-worker vllm`, mirroring MLXDistributed's structural precedent (operator-launched, static config, no NATS placement). The worker: - Optionally self-registers with the frontend as an agent-type node tagged `node.role=vllm-follower` so it's visible in the admin UI and operators can scope ordinary models away via inverse selectors. - Resolves the platform-specific vllm backend via the gallery's "vllm" meta-entry (cuda*, intel-vllm, rocm-vllm, ...). - Runs vLLM as a child process so the heartbeat goroutine survives until vLLM exits; forwards SIGINT/SIGTERM so vLLM can clean up its ZMQ sockets before we tear down. - Validates --headless + --start-rank 0 is rejected (rank 0 is the head and must serve the API). Backend run.sh dispatches `serve` as the first arg to vllm's own CLI instead of LocalAI's backend.py gRPC server -- the follower speaks ZMQ directly to the head, there is no LocalAI gRPC on the follower side. Single-node usage is unchanged. Generalises the gallery resolution helper into findBackendPath() shared by MLX and vLLM workers; extracts ParseNodeLabels for the comma-separated label parsing both use. Ships with two compose recipes (`docker-compose.vllm-multinode.yaml` for NVIDIA, `docker-compose.vllm-multinode.intel.yaml` for Intel XPU/xccl) plus `tests/e2e/vllm-multinode/smoke.sh`. Both vendors are supported (NCCL for CUDA/ROCm, xccl for XPU) but mixed-vendor DP is not -- PyTorch's process group requires every rank to use the same collective backend, and NCCL/xccl/gloo don't interoperate. Out of scope (deferred): SmartRouter-driven placement of follower ranks via NATS backend.install events, follower log streaming through /api/backend-logs, tensor-parallel across nodes, disaggregated prefill via KVTransferConfig. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * test(vllm): CPU-only end-to-end test for multi-node DP Adds tests/e2e/vllm-multinode/, a Ginkgo + testcontainers-go suite that brings up a head + headless follower from the locally-built local-ai:tests image, bind-mounts the cpu-vllm backend extracted by make extract-backend-vllm so it's seen as a system backend (no gallery fetch, no registry server), and asserts a chat completion across both DP ranks. New `make test-e2e-vllm-multinode` target wires the docker build, backend extract, and ginkgo run together; BuildKit caches both images so re-runs only rebuild what changed. Tagged Label("VLLMMultinode") so the existing distributed suite isn't pulled along. Two pre-existing bugs surfaced by the test: 1. extract-backend-% (Makefile) failed for every backend, because all backend images end with `FROM scratch` and `docker create` rejects an image with no CMD/ENTRYPOINT. Fixed by passing --entrypoint=/run.sh -- the container is never started, only docker-cp'd, so the path doesn't have to exist; we just need anything that satisfies the daemon's create-time validation. 2. backend/python/vllm/run.sh's `serve` shortcut for the multi-node DP follower exec'd ${EDIR}/venv/bin/vllm directly, but uv bakes an absolute build-time shebang (`#!/vllm/venv/bin/python3`) that no longer resolves once the backend is relocated to BackendsPath. _makeVenvPortable's shebang rewriter only matches paths that already point at ${EDIR}, so the original shebang slips through unchanged. Fixed by exec-ing ${EDIR}/venv/bin/python with the script as an argument -- Python ignores the script's shebang in that case. The test fixture caps memory aggressively (max_model_len=512, VLLM_CPU_KVCACHE_SPACE=1, TORCH_COMPILE_DISABLE=1) so two CPU engines fit on a 32 GB box. TORCH_COMPILE_DISABLE is currently mandatory for cpu-vllm: torch._inductor's CPU-ISA probe runs even with enforce_eager=True and needs g++ on PATH, which the LocalAI runtime image doesn't ship -- to be addressed in a follow-up that bundles a toolchain in the cpu-vllm backend. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(vllm): bundle a g++ toolchain in the cpu-vllm backend image torch._inductor's CPU-ISA probe (`cpu_model_runner.py:65 "Warming up model for the compilation"`) shells out to `g++` at vllm engine startup, regardless of `enforce_eager=True` -- the eager flag only disables CUDA graphs, not inductor's first-batch warmup. The LocalAI CPU runtime image (Dockerfile, unconditional apt list) does not ship build-essential, and the cpu-vllm backend image is `FROM scratch`, so any non-trivial inference on cpu-vllm crashes with: torch._inductor.exc.InductorError: InvalidCxxCompiler: No working C++ compiler found in torch._inductor.config.cpp.cxx: (None, 'g++') Bundling the toolchain in the CPU runtime image would bloat every non-vllm-CPU deployment and force a single GCC version on backends that may want clang or a different version. So this lives in the backend, gated to BUILD_TYPE=='' (the CPU profile). `package.sh` snapshots g++ + binutils + cc1plus + libstdc++ + libc6 (runtime + dev) + the math libs cc1plus links (libisl/libmpc/libmpfr/ libjansson) into ${BACKEND}/toolchain/, mirroring /usr/... layout. The unversioned binaries on Debian/Ubuntu are symlink chains pointing into multiarch packages (`g++` -> `g++-13` -> `x86_64-linux-gnu-g++-13`, the latter in `g++-13-x86-64-linux-gnu`), so the package list resolves both the version and the arch-triplet variant. Symlinks /lib -> usr/lib and /lib64 -> usr/lib64 are recreated under the toolchain root because Ubuntu's UsrMerge keeps them at /, and ld scripts (`libc.so`, `libm.so`) hardcode `/lib/...` paths that --sysroot re-roots into the toolchain. The unversioned `g++`/`gcc`/`cpp` symlinks are replaced with wrapper shell scripts that resolve their own location at runtime and pass `--sysroot=<toolchain>` and `-B <toolchain>/usr/lib/gcc/<triplet>/<ver>/` to the underlying versioned binary. That's how torch's bare `g++ foo.cpp -o foo` invocation finds cc1plus (-B), system headers (--sysroot), and the bundled libstdc++ (--sysroot, --sysroot is recursive into linker). `run.sh` adds the toolchain bin dir to PATH and the toolchain's shared-lib dir to LD_LIBRARY_PATH -- everything else (header search, linker search, executable search) is encapsulated in the wrappers. No-op for non-CPU builds, the dir doesn't exist there. The cpu-vllm image grows by ~217 MB. Tradeoff is acceptable -- cpu-vllm is already a niche profile (few users compared to GPU vllm) and the alternative is a backend that crashes at first inference unless the operator manually sets TORCH_COMPILE_DISABLE=1, which silently disables all torch.compile optimizations. Drops `TORCH_COMPILE_DISABLE=1` from tests/e2e/vllm-multinode -- the smoke now exercises the real compile path through the bundled toolchain. Test runtime is +20s for the warmup compile, still <90s end to end. Assisted-by: Claude:claude-opus-4-7 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(vllm): scope jetson-ai-lab index to L4T-specific wheels via pyproject.toml The L4T arm64 build resolves dependencies through pypi.jetson-ai-lab.io, which hosts the L4T-specific torch / vllm / flash-attn wheels but also transparently proxies the rest of PyPI through `/+f/<sha>/<filename>` URLs. With `--extra-index-url` + `--index-strategy=unsafe-best-match` uv would pick those proxy URLs for ordinary PyPI packages — anthropic/openai/propcache/annotated-types — and fail when the proxy 503s. Master is hitting the same bug on its own l4t-vllm matrix entry. Switch the l4t13 install path to a pyproject.toml that marks the jetson-ai-lab index `explicit = true` and pins only torch, torchvision, torchaudio, flash-attn, and vllm to it via [tool.uv.sources]. uv won't consult the L4T mirror for anything else, so transitive deps fall back to PyPI as the default index — no exposure to the proxy 503s. `uv pip install -r requirements.txt` ignores [tool.uv.sources], so the l4t13 branch in install.sh now invokes `uv pip install --requirement pyproject.toml` directly, replacing the old requirements-l4t13*.txt files. Other BUILD_PROFILEs continue using libbackend.sh's installRequirements and never read pyproject.toml. Local resolution test (x86_64, dry-run) confirms uv hits the L4T index for torch and falls through to PyPI for everything else. Assisted-by: claude-code:claude-opus-4-7-1m [Read] [Edit] [Bash] [Write] Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-05-06 00:22:50 +02:00