26 Commits

Author	SHA1	Message	Date
mudler's LocalAI [bot]	05e16e0fa8	chore: remove local pre-commit gates (#11116) ... Remove the versioned pre-commit hook and its installer while retaining CI coverage and conformance checks. Assisted-by: Codex:gpt-5 Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-25 01:19:32 +02:00
mudler's LocalAI [bot]	83a0f16a21	feat(gallery): let one gallery entry offer several builds of the same model (#10943) ... * feat(system): expose raw detected capability for model meta resolution Model meta gallery entries express hardware fallback through candidate ordering rather than a capability map, so they need the undecorated detected capability string without Capability's default/cpu fallback chain. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(system): drop duplicate capability accessor, cover DetectedCapability ReportedCapability was added with a body identical to the existing DetectedCapability. Keep one accessor and move the specs onto it, since DetectedCapability had no direct coverage of its no-fallback behavior. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(vram): parse IEC binary size suffixes (KiB..PiB) ParseSizeString accepted only SI suffixes, so a "20GiB" floor was rejected outright. Model and VRAM sizes are conventionally quoted in IEC units, and silently reading GiB as GB would understate a floor by about 7%. Purely additive: these inputs previously returned an unknown-suffix error. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add Candidate type for meta model entries Candidate is one option in a meta entry's ordered variant list. It names a concrete gallery entry and declares when that entry suits the host. EffectiveMinVRAM resolves the VRAM floor, letting an authored min_vram win over a nightly-inferred one. An unparseable floor errors instead of being treated as absent: swallowing a typo would turn a constrained candidate into an unconstrained one and select a too-large variant rather than fail loudly. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add hardware-aware model variant resolver Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): allow gallery model entries to declare variant candidates A gallery entry with a non-empty candidates list is a meta entry: it names an ordered list of concrete entries and resolves to the first one the host can satisfy, instead of describing model files directly. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): resolve meta model entries to hardware-appropriate variants at install Meta gallery entries carry an ordered candidate list; at install time the first candidate the host satisfies is resolved and its payload installed under the meta's name, so the model keeps a stable name regardless of which variant backs it. The resolution is recorded in the installed gallery config so a reinstall honors a prior pin and operators can see the backing variant. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): key meta pin recall on the installed name and detach resolved entries Six review findings on the meta-entry install path. Pin recall was keyed on the gallery entry name while applyModel writes the record under the install name (req.Name when supplied), so a meta installed under a custom name with a pin lost that pin on reinstall and was silently re-resolved onto a different variant, possibly swapping its backend. Compute the install name with applyModel's own precedence before the recall. ResolveMetaModel returned a shallow struct copy, so the resolved entry's Overrides aliased the gallery entry's map and the install path's in-place mergo merge wrote the caller's request into the shared catalog. Detach Overrides, ConfigFile, AdditionalFiles, URLs and Tags. Not exploitable today only because this path re-unmarshals the gallery per call, which is a property nobody should have to rely on. Also: overlay the meta's name onto the persisted config for meta installs so the gallery file no longer records the variant's name; move the pinned-VRAM warning below the variant validation so a pin naming a nonexistent entry does not warn about VRAM before failing for an unrelated reason; and stop seeding config.URLs in the config_file branch, which duplicated every declared URL. Add seven network-free specs driving InstallModelFromGallery with a meta entry: variant payload wins over the meta's legacy url fallback, the resolution record round-trips to disk, a pin is recorded and honored on reinstall including under a custom install name, and the resolved entry does not alias the gallery's maps. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): deep-copy meta overrides and make two specs functional ResolveMetaModel detached the resolved entry's Overrides and ConfigFile with maps.Clone, which only copies the top level. Gallery overrides are nested in practice (parameters.model is near-universal) and the install path merges the caller's request with mergo.WithOverride, which recurses into nested maps and overwrites them in place, so the gallery entry's own inner maps were still reachable and still got rewritten by the last caller to install. Copy both maps all the way down instead, recursing through the container shapes a YAML decoder produces. ConfigFile is not mutated on the install path today, but it carries the same kind of nested payload and leaving it shallowly cloned would invite the bug back. Also fix two specs that passed whether or not their target fix was present: - "does not write the caller's overrides back into the gallery entry" re-read the catalog from disk, which re-unmarshals fresh structs and so cannot observe in-memory aliasing. It now asserts against the in-memory gallery entry and drives the real mergo merge. - "round-trips the resolution record to disk under the meta's name" asserted a name that is already correct in the config_file branch. It now drives the url branch via a file:// fixture, where the meta-name overlay actually applies. Both were verified red by reverting their fix. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(gallery): lint meta model entry invariants in index.yaml Adds Ginkgo specs that parse the shipped gallery/index.yaml and enforce the invariants that keep meta entries safe: a legacy url fallback equal to the final candidate's url, references only to existing non-meta entries, a min_vram floor on every candidate but the last-resort one, a capability drawn only from the vocabulary the system can report, and descending VRAM floors within a capability group. The capability check is the only compensating control for a typo there. Candidate matching is a case-sensitive exact comparison against SystemState.DetectedCapability(), so an unknown value never matches and falls through silently instead of erroring. The vocabulary therefore mirrors the raw return set of getSystemCapabilities(), which notably excludes "cpu": that is a fallback key inside Capability(capMap) on the meta backend path, never a reported capability. A CPU-only host reports "default". These pass vacuously until the pilot meta entry lands; the guard is intentionally in place before the thing it guards. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(gallery): close coverage gaps in the meta entry lint The ordering invariant grouped candidates by capability and asserted floors descend within a group. A candidate with an EMPTY capability matches every host, so it does not belong in its own group: it dominates every later candidate whose floor is at or above its own, across capability groups. Track a running minimum floor over the unconditional candidates instead, which subsumes the old same-group check for the empty capability. Every spec skipped non-meta entries, so with zero meta entries in the index all five bodies were no-ops. Aligning GalleryModel.IsMeta() with GalleryBackend.IsMeta(), whose semantics are deliberately opposite, would have made all of them pass while checking nothing. Extract each invariant into a helper over a slice of entries returning the violations it finds, and cover those helpers with synthetic fixtures so the logic stays tested at zero meta entries. The index-driven specs are now a thin application of already proven logic. Also assert the index parses non-empty, report every violation in one run rather than aborting on the first, and parse the index once for the suite. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(gallery): add nightly denormalization of meta model candidates Fills the read-only backend, quantization and inferred_min_vram fields on meta gallery candidates and opens a PR, modeled on the existing checksum_checker job. Computing these needs network access, so it happens nightly rather than at install time. An authored min_vram is never modified: a human who measured a real load knows more than a pre-download estimate does. The index is rewritten via yaml.Node rather than a document round-trip. A full round-trip reflows all ~26k lines of gallery/index.yaml, which would bury the computed values and make the nightly PR unreviewable. The rewrite touches only the three derived keys, so authored styling survives and a run that computes nothing leaves the file untouched. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ci): keep the gallery denormalize diff reviewable and self-healing The nightly denormalization job edits YAML nodes instead of round-tripping structs so its PR stays small enough for a human to review, but the write path undid that: yaml.Marshal re-encoded the node tree at yaml.v3's default 4-space indent and dropped the leading document marker, reflowing roughly 6000 lines around the handful of real changes. Encode through yaml.NewEncoder at the index's authored 2-space indent and restore the header. A write that changes three fields now changes three lines. Stale inferred_min_vram values were also never cleared. Both skip paths (an authored min_vram is present, or the candidate is the last resort) returned before touching the field, so a candidate that gained a floor or became the last resort after a reorder kept an inferred value that EffectiveMinVRAM reported as a real constraint, failing the meta lint with no way for the job to self-heal. Clear the field before both skips. The workflow discarded a whole night's work on any single failure: the program exits 1 when a candidate cannot be estimated, which aborted the job before the PR step, so one unreachable candidate blocked every other refresh indefinitely. Capture the status, open the PR with what was computed, mark the PR body as partial, and fail the run afterwards so the problem still surfaces. Also preserve the index's existing file mode instead of forcing 0644, and drop the redundant //go:build ignore tag, since Go already skips dot directories and the sibling modelslist.go carries no tag. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add nanbeige4.1-3b meta entry with hardware-resolved variants Adds the first real meta entry to the gallery index. It resolves to the Q8_0 build on hosts with at least 6GiB of VRAM and to the Q4_K_M build everywhere else, installing either payload under the stable name nanbeige4.1-3b. The entry carries a url equal to its final candidate's url. LocalAI releases that predate candidates support parse the index non-strictly and drop the key silently, so without that url they would list the entry and install nothing. A regression spec parses the index the way those releases do and asserts every meta entry stays installable for them. Also teaches core/schema/gallery-model.schema.json about candidates. The schema sets additionalProperties: false at the top level, so an author following CONTRIBUTING.md and adding the yaml-language-server comment would otherwise get a validation error on this entry. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): make candidate entries complete, installable entries Reworks hardware-resolved gallery variants after a design pivot. There is no longer a separate "meta" entry kind. A gallery entry is a normal, complete entry that may additionally carry candidates:, a list of hardware-gated upgrades over itself, and the entry is itself the last-resort candidate. The previous design relied on a bare url: as the fallback for LocalAI releases that predate candidates support. That fallback is empty in practice: none of the 80 gallery/*.yaml files carry a top-level files:, and 1216 of 1281 index entries carry their payload in the index entry itself, so a url alone yields a config template with nothing to download. Since every released LocalAI reads gallery/index.yaml live from master, merging a payload-less entry would have shown every existing user a model that installs to a broken state. Making the entry its own base candidate removes the problem at the root: old clients drop the candidates key and install the entry exactly as they do today. Resolution order is now explicit pin, then capability plus VRAM over the declared upgrades, then the entry itself. The entry ALWAYS installs: when its own min_vram or capability is unmet the installer warns and installs it anyway, because there is nothing below it and refusing would make the gallery behave worse the newer the client is. A pin naming the entry's own name is valid and is how an operator declines an upgrade. IsMeta() becomes HasCandidates(), ResolveMetaModel becomes ResolveVariant, and the persisted meta_name record key becomes entry_name. GalleryBackend.IsMeta() is a separate concept and is untouched. The lint drops the three rules the pivot makes wrong (url equality with the final candidate, no inline payload, unconstrained final candidate) and gains one: the entry's own floor must sit strictly below every candidate's, since a base that outranks a candidate makes that candidate unreachable. The pilot entry is now the existing nanbeige4.1-3b-q4, which gains a 2GiB floor of its own and a single 6GiB upgrade to nanbeige4.1-3b-q8, replacing the separate nanbeige4.1-3b entry added in d0d441bb4. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): select model variants by hardware fit, not authored order Gallery entries could already carry a list of alternatives, but selection was an authored, ordered, first-match policy: every candidate declared a `capability` string and the VRAM floors had to descend in a hand-tuned order. That pushed hardware knowledge onto whoever edits the gallery and made ordering load-bearing, so a reordered list silently changed what users installed. None of it was necessary. SystemState.IsBackendCompatible already derives hardware support from a backend name alone: it knows MLX and metal are Darwin-only, CUDA is NVIDIA-only, ROCm AMD-only, SYCL Intel-only. Selection can read that instead of asking authors to restate it. Authoring is now just a list of names: - name: qwen3.6-27b min_memory: 4GiB variants: - model: qwen3.6-27b-mlx-8bit - model: qwen3.6-27b-gguf-q8 min_memory: 28GiB and all the intelligence moved into the selector. Given a host it drops the variants whose backend cannot run here, drops those whose known memory requirement exceeds what the host has, and takes the LARGEST of what is left, because a bigger footprint is a higher quality quantization of the same model. A variant of unknown size is kept, since nothing proves it does not fit, but it ranks last so a proven fit always beats a guess. An explicit pin still wins outright, and if nothing survives the entry installs its own payload: the base always installs, this never refuses. Available memory is VRAM when a GPU was detected and system RAM otherwise, read through xsysinfo so a cgroup limit is honored and a container gets its own limit rather than the node's RAM. Capability disappears entirely, from the types, the schema and the lint. VRAM and RAM collapse into one `min_memory`, because a model's footprint is roughly the same wherever it lives and one figure is compared against whichever applies. The lint rules about ordering, the capability vocabulary and floor relationships are deleted with the hazards they described; what remains is that every variant names an entry that exists and does not itself declare variants, plus that any memory figure actually parses. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * gallery: size model variants with a live probe, drop the nightly denormalizer Selection needs each variant's size to decide whether it fits and to rank largest-first. That figure was written into the index by a nightly job, which made the gallery carry a derived value that could drift from the entry it was derived from. Derive it at install time instead. pkg/vram already sizes a model without downloading it, and the gallery UI already uses it: a remote GGUF header range-fetch, then an HTTP HEAD for the content length, then any declared size:. It caches its results, so reuse it rather than writing a second probing path. A probe failure must never fail an install, so an unprobeable variant is treated as unknown: it survives the memory filter, because nothing proves it does not fit, and it ranks last, so a known-good fit always beats a guess. If every probe fails, selection still terminates on the base entry. The probe is injected through ResolveEnv rather than called directly, for the same reason the backend compatibility check is: specs pin an exact size, or an exact failure, without reaching the network. With that in place three things are dead weight and go: - The nightly job and the fields it populated. Variant.Backend was redundant because the backend is resolved live from the referenced entry during selection, and Quantization was display-only that nothing read. - min_memory on the base entry. The base always installs and its floor could only warn, so it could not change any outcome. - The lint rules and schema entries for both. min_memory on individual variants stays, as the override for when the probed size is wrong. An authored figure now suppresses the probe entirely rather than merely outranking it, so it costs no round trip. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): expose model variants for selection over API, CLI and MCP A gallery entry may carry `variants:`, alternative builds of the same model. Selection already worked at install time, but nothing could see what an entry offered or ask for a specific build, so the feature was undrivable. Listing: `GET /api/models` now reports `variants` and `auto_variant` for the entries that declare variants. Each variant carries its resolved backend, its measured size and whether it fits this host. `auto_variant` is what installing without a choice would pick right now. The new gallery.DescribeVariants runs the same variantOptions + SelectVariant pass the installer runs, so the reported default cannot drift from what installing actually does, and HostResolveEnv is extracted so both derive the host and share pkg/vram's probe cache from one place. Performance: an entry that declares no variants returns early without touching the probe, so the ~1280 ordinary entries cost exactly what they cost before. Selection: `variant` is accepted on POST /models/apply, as a query param on POST /api/models/install/:id, on the gallery apply file/string request, as `local-ai models install --variant`, and as a parameter on the install_model MCP tool (both the httpapi and inproc clients). Empty means auto-select. An unknown variant name now fails the install naming what was requested. This closes a real hole: an entry declaring no variants short-circuits before selection runs, so a requested variant was previously dropped silently and the install reported success. startup.InstallModels ends in a variadic model list, so install options could not be appended to it; InstallModelsWithOptions is added alongside and InstallModels delegates to it. No caller signature changed. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): drop the redundant variant min_memory field Variant.MinMemory was an authored override for when the live probe misreads a variant's footprint. It duplicated an existing field: probeEntryMemory already passes the entry's declared size: into EstimateModelMultiContext, whose cascade prefers that declared size over its own guesswork. Correcting size: on the referenced entry fixes the figure for every consumer rather than only for variant selection, so min_memory shadowed the right answer. A variant is now nothing but a name. Its effective size is exactly the probe result, and an unknown stays unknown: it survives the filter and ranks last. EffectiveMemory loses its error return along with the field. The authored string was the only thing that could fail to parse, so the error had no remaining source and was propagating dead nil-checks through SelectVariant, DescribeVariants and the pin warning. Selection behaviour is unchanged. The specs covering probe-derived sizing, ranking, filtering, the unknown-size path, pin recall, entry/variant metadata split and deep-copy isolation all survive; the three install specs that needed a definite size now declare it through the referenced entry's own size:, which exercises the documented escape hatch directly. gallery/index.yaml is untouched: no entry ever carried the key. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): rank the entry's own build against its variants Variant selection pulled the declaring entry's own payload, the base, out of the candidate set and consulted it only once every declared variant had been rejected. Two real failures followed. A variant whose size the probe cannot determine deliberately survives the memory filter, because nothing proves it does not fit. As the only survivor it then won outright on any host, however small: a 2GiB machine installed an unmeasured variant in preference to the 4GiB build the entry itself ships, with no warning. 241 of the 1280 current index entries carry no files and no size, which is exactly that shape. "Largest wins" also broke whenever the base was the largest. An author writing a Q8 entry that offers a Q4 downgrade for small hosts, a natural shape that nothing in the lint, schema or docs discourages, had the Q4 installed on every large host instead. Make the base an ordinary participant. It is still exempt from both filters, so selection always terminates on something installable, but it is now ranked against the variants: a proven fit first and largest, then the base, then any variant whose size nothing could measure. Both failures disappear together. The base is probed for its size accordingly, which it was not before, because an unsized base would lose every contest to an unmeasurable variant. FellBackToBase is kept but narrowed to "no declared variant survived", rather than "the base was chosen", since the base now also wins on merit and that is not worth warning about. A recalled variant pin also became a permanent install failure. A pin the caller supplies on this request must stay fatal, but one recalled from ._gallery_<name>.yaml can be invalidated by any later gallery edit, and failing on it turned one rename into a model that could never be reinstalled or upgraded again short of deleting a dotfile the user has never heard of. A stale recalled pin is now dropped with a warning naming it, and selection runs as if it had never been recorded. Also drop the last textual reference to two abandoned designs from the DetectedCapability comment, correct the documented variants JSON example, which showed a memory_bytes of 0 that omitempty makes impossible, and remove an em dash from the install skill. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): budget variant memory from RAM when a GPU reports no VRAM Variant selection read its memory budget from VRAM whenever a GPU capability was detected, and from system RAM only when none was. Apple Silicon satisfies the first branch and fails the premise: arm64 macs report the metal capability unconditionally, without probing anything, while TotalAvailableVRAM has no discrete VRAM pool to find and returns zero. The budget therefore came out as zero on every Mac. Zero drops every variant carrying a known size, so the base build was installed on all of them however much memory the machine had. The feature was inert on the platform, and silently: falling back to the base is a legitimate outcome, so nothing looked wrong. Take VRAM only when it is actually a number, and fall back to RAM otherwise. On a unified-memory host RAM is not an approximation of the budget, it is the budget, since the GPU shares it. A discrete GPU whose VRAM could not be read also lands on RAM, which overstates what the card holds but understates nothing the host has; the previous zero understated both. An unreadable RAM figure still yields zero and still installs the base, so a genuinely unknown host is not talked into a larger download. This is what turned tests-apple red: "installs a fitting variant's payload under the entry's own name" asserts on selection, and the runner resolved to the base because its budget was zero. The added specs pin the branch directly rather than relying on a macOS runner to notice again. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): add a model variant picker to the models gallery PR #10943 shipped the server side: a gallery entry may declare `variants:`, `GET /api/models` attaches `variants` and `auto_variant` to declaring entries, and `POST /api/models/install/:id` accepts a `variant` query parameter. Nothing in the UI consumed any of it, so the feature was not reachable from the browser. This wires it up. modelsApi.install takes an optional second argument and appends an encoded `?variant=` only when one is given, so every existing call site keeps sending exactly the request it sent before. On the models table, an entry that declares variants gets a split button. The primary Install still installs the auto-selected build, because auto is the default and the point of the feature; the chevron opens a menu for a deliberate override. It follows the Backends.jsx precedent: one shared Popover re-anchored per row, rendering .action-menu items, which brings Escape, outside-click and focus return along with it. An entry that declares no variants renders exactly as it did before. A variant that does not fit is dimmed but stays selectable, since the server honors an explicit choice with a warning rather than refusing it. memory_bytes is omitempty on the wire, so an absent key means the size is unknown and never zero. A single helper guards both the menu and the detail row, because formatBytes would otherwise render a falsy value as "0 B", which reads as "needs nothing". The expanded detail row gains a Variants section listing each build's backend, size, whether it fits, which is the entry's own build, and which one auto-selection would pick, built from the existing DetailRow helper and .badge classes. Eight Playwright specs cover the picker, including that plain Install sends no variant parameter and that choosing one sends it. One pre-existing assertion was scoped with .first(): the Variants section legitimately adds more llama-cpp badges to the detail row, which tripped strict mode. UI line coverage 49.42% -> 49.36% against a 40.0 baseline and 0.8pp tolerance; branch coverage rose 72.04% -> 72.66%. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): describe model variants from a companion endpoint Variant description probes each referenced entry's weight files over the network: an HTTP HEAD plus a ranged GET, serial, five seconds per probe with no aggregate deadline. Running it inline in GET /api/models made one listing cost (entries x variants) round trips. The Manage page fetches with items=9999, so at 200 declaring entries that is ~1000 serial probes, minutes of a blocked handler and gigabytes of range traffic for a single page load. Only one entry declares variants today, but the feature exists so that many will. Follow the precedent already set for VRAM estimates. The listing now reports only has_variants, a length check on loaded metadata that touches nothing, and GET /api/models/variants/:id returns the description for one entry, mirroring estimate/:id in route shape, auth and error handling. DescribeVariants itself is unchanged; only its caller moved. The picker fetches lazily at the two points where a user asks to see variants, opening the split-button menu and expanding the detail row, and caches per entry for the page session. An entry declaring no variants issues no request at all. A spec counts real HTTP hits on the weight files, so it goes red if description becomes reachable from the listing path again through any caller. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): filter the model gallery to entries that declare variants The gallery is heading towards showing parent entries and hiding the individual builds they reference, so a user sees one row per model rather than six quantizations of it. Adoption is a single entry today, so defaulting to that would leave a one-row gallery. This ships the migration-phase inverse instead: the default is untouched, and a toggle narrows the list to only the entries that declare variants. It previews the end state and changes nothing until someone asks for it. The filter is server-side, next to term/tag/backend/capability and above the pagination arithmetic. The listing paginates at 9 items, so narrowing on the client would leave totalPages and availableModels describing the unfiltered set and hand the user empty pages. It selects on HasVariants(), which reads already-loaded metadata, so it issues no variant probes. The parameter is named has_variants after the listing field it selects on, and is compared against "true" like the other boolean query params (all_users, save_checkpoint), so has_variants=false reads as absent. With it omitted the response is byte-for-byte what it was before. The control is the shared Toggle component, matching the fitsFilter toggle already on this page: same wrapper class, same icon and label shape, same localStorage persistence. Unlike fitsFilter it resets to page 1 on change, which a server-side filter has to do. Stacking the toggle with a tag or backend filter easily yields nothing while one entry declares variants, so the empty state now names the variants filter as the cause rather than leaving a user to conclude the gallery is broken. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): render gallery model descriptions as Markdown Gallery descriptions are Markdown, but the React UI dumped them raw, so a model whose description opens with an ATX heading showed a literal "# Qwen3.6-27B [](https://chat.qwen.ai)" in the list. Full-description areas now render through renderMarkdown (marked + DOMPurify), matching how Backends.jsx and the Manage detail panels already handle the same content: - Models.jsx expanded detail row - VoiceLibrary.jsx voice detail header The truncated one-line previews must not render block Markdown: a leading "#" would become an <h1> and wreck the row height and rhythm. They get a new stripMarkdown() helper instead, which reduces Markdown to a single line of readable plain text. It is used for the cell text and for the title tooltip, since a tooltip full of "[](url)" is no better than a cell full of it: - Models.jsx gallery table description cell - Manage.jsx model and backend resource-row descriptions stripMarkdown walks marked's lexer output rather than running regexes over the source, so what it strips is by construction what renderMarkdown would have rendered, and it needs no new dependency. Output lands in JSX text nodes, so React escapes it; no new dangerouslySetInnerHTML beyond the two full-description sites, both of which run DOMPurify. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ui): strip Markdown from the backends table description cell Commit b35d630cf fixed this for gallery models but left the Backends admin page with the same asymmetry: its detail panel renders the description through renderMarkdown, while the collapsed table row dumped the raw gallery string into both the cell body and the title tooltip. That is user-visible. 40 of the 949 entries in backend/index.yaml carry Markdown - insightface uses inline code backticks, others use lists and links - and backend descriptions also contain embedded newlines, so the one-line cell showed literal syntax. The cell now runs stripMarkdown over the description once and uses the result for the text and the title, matching Models.jsx and the ResourceRowDesc component in Manage.jsx. The '-' placeholder is preserved, and now also fires when a description reduces to nothing after stripping. The detail panel is untouched and no new dangerouslySetInnerHTML is introduced: stripMarkdown output lands in a JSX text node, so React escapes it. Three Playwright specs cover it: a description with a heading, inline code and a link renders as clean text with no literal syntax and no block element in the cell, the title tooltip carries the same stripped text, and a backend without a description still shows the placeholder. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ui(models): polish the variant detail view and scope rendered Markdown The gallery detail pane rendered every field through the same two-column label/value row, including the description. Multi-paragraph prose in a value cell ran eight rows tall at the top of the pane on a ~1200px measure, breaking the grid's rhythm exactly where the eye enters. Move it into its own full-width block above the table, capped at a 68ch measure, keeping the label. Rendered Markdown had no scoped typography anywhere in the app, so a description opening with `#` inherited the browser default 2em inside a 13px surface while a `##` further down was indistinguishable from body text. Add a reusable .markdown-body block mapping h1-h6, paragraphs, lists, links, code, blockquotes, images and tables onto the existing type scale, and apply it to every renderMarkdown() consumer: the models detail, the backends detail, both Manage details and the voice library detail. Rebalance the variants list so the name leads. Backend and size drop from badge/secondary weight to muted metadata; the FITS badge goes entirely, since it was true of nearly every row and so said nothing, while the variant that does not fit keeps a warning badge and a dimmed name. AUTO-SELECTED stays marked because it answers what a plain Install produces. Rows share the parent's grid tracks via subgrid so name, backend, size and status line up down the list instead of raggedly following name length. Finally, make each variant row actionable. It looked like a list of choices but was inert text, with per-variant install hidden behind the split-button chevron elsewhere; each row is now a button onto the existing handleInstall(modelId, variant) path, with hover, keyboard focus and a disabled state while an install is in flight. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): collapse the listing to one row per model The listing supported has_variants=true, which narrowed to entries that DECLARE variants. With adoption at three entries that showed three rows, which is useless; it was always a placeholder. Replace it with the view that is actually useful: the deduplicated gallery. Show every entry installable in its own right and nothing twice, which means the parents plus every entry nobody references, and hide only the builds another entry already offers as a variant, since those are reachable through their parent. The parameter is renamed to collapse_variants accordingly: the filter is no longer a predicate on a row's own metadata but a view over the whole gallery. Default stays off, so the response with the parameter absent is unchanged. VariantReferencedIDs never reports an entry that declares variants of its own, so parents are always visible. That guarantees every hidden entry has a visible entry offering it, and no chain can strand a row. Variant resolution already refuses to install such a reference, but the listing has to stay coherent in the presence of a gallery that has one rather than silently swallowing entries. Self-references and dangling references hide nothing. The referenced set is computed over the whole gallery rather than over what the other filters left, so an entry is hidden because a parent offers it and never because of what the user searched for. The pass is over metadata already in memory: it resolves nothing over the network and triggers no variant description or size probe, so the listing's zero-probe contract still holds. The UI toggle keeps its behaviour (persistence, page reset, clear filters) and becomes "One row per model", which says what the user gets. Its localStorage key moves too, since the stored value meant a different filter. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): show the collapsed model listing by default The gallery listing is what a user reaches for to answer "what can I install". Answering that with several rows for the same model, one per build, makes the reader do the deduplication the collapsed view already does, so the collapsed view is the one to land on. The UI now asks for collapse_variants=true unless the toggle says otherwise. The server default is deliberately untouched: a request with the parameter absent still returns the full listing, because other API clients depend on that response and collapsing it under them would be a breaking change. Opting out omits the parameter rather than sending false, so it asks for exactly the listing everyone else gets. The stored preference changes vocabulary from '1'/'0' to 'on'/'off'. The previous build wrote it from an effect that runs on mount, so a stored '0' recorded that the page had been opened rather than that anyone chose the expanded view, and honouring it would pin every earlier visitor to a default they never picked. Only the new vocabulary counts as a choice; a legacy '1' meant the collapsed view and is what the new default gives anyway, so no earlier deliberate choice is lost. Collapsing being the default also changes what the empty state may say about it. An opted-into filter can be named as the cause of an empty result; a default cannot, so the filters keep the top line and the collapsed view drops to a hint below it, shown only once filters are narrowing the set. For the same reason "Clear filters" now restores the collapsed default instead of switching it off, and the toggle alone no longer counts as a filter worth offering to clear. The label stays "One row per model": it describes the view the user is looking at rather than an action, so it reads the same whether it is opted into or out of. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * gallery: group alternative builds of the same weights under variants Sweep the gallery for entries that are alternative builds of the same weights (different quantization, precision, or runtime format) and declare them as variants of a single parent row, so the listing offers one row per model instead of one row per quantization and the installer picks the largest build that this host can actually run. 41 families over 95 entries, turning 54 entries into variants. The parent is the bare-named entry wherever one exists, so nothing changes about what any existing entry installs. Ranking already selects the largest fitting build regardless of which entry is nominally the parent, so the parent only decides the pathological case where nothing fits. For the ten families that have no bare-named entry, the smallest build is the parent, since that is the one that has to install when nothing fits. Grouping was verified against the actual model filenames rather than the entry names alone. Different parameter sizes, languages, finetunes, and products that merely share a name prefix are left as separate rows: the qwen3.6 APEX and pi-tune finetunes, the DFlash and MTP speculative-decoding pairings, English-only versus multilingual Whisper, the QAT versus non-QAT Gemma 4 weights, and the abliterated FLUX build are all distinct models. Six parents define YAML anchors that other entries pull in with a merge key, which would have handed their variants to every merging child. For the two depth-anything anchors that would have made fourteen unrelated entries advertise the base model's builds as their own. All 26 merging children therefore carry an explicit empty variants list, which overrides the merged key and is equivalent to the key being absent. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): rank model variants by host backend preference Variant auto-selection filtered candidates by whether their backend can run on the host, then ranked the survivors by size alone. The backend never influenced the choice beyond that gate, so a Mac offered both an MLX build and a llama.cpp build kept neither filtered and installed whichever was larger, leaving the native accelerated runtime unused. The same held for CUDA against CPU on NVIDIA and ROCm against Vulkan on AMD. Rank by the host's backend preference between the fit tier and size: fit stays a filter, preference decides among the builds the host can equally hold, and size still separates builds on equally preferred runtimes. The preference data stays in one declarative table in pkg/system, now read by a prefix lookup instead of a switch, so adding a capability or reordering one host's runtimes is a one-line edit and the gallery's ranking code carries no per-backend branching. MLX joins the metal rule ahead of metal itself, which is inert for the existing alias-resolution consumer because no alias group holds a candidate named for mlx. An unrecognised backend, an unrecognised capability and an absent preference list all collapse to the previous size-only ordering rather than erroring or dropping candidates. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): rank variants by engine name, not backend build tag Variant auto-selection ranked candidates with SystemState.BackendPreferenceTokens, but that function and the variant ranker speak different vocabularies. BackendPreferenceTokens returns BUILD TAGS ("cuda", "rocm", "sycl", "vulkan", "metal", "cpu"). It exists to match installed backend build directory names like "llama-cpp-cuda-12" during alias resolution in ListSystemBackends. Variant ranking instead matches a gallery entry's `backend:` value, which is an ENGINE NAME: "llama-cpp", "vllm", "vllm-omni", "sglang", "mlx" and the rest. No engine name in gallery/index.yaml contains "cuda", "rocm", "sycl" or "vulkan". preferenceRank matches by substring, so on an NVIDIA host the tokens [cuda, vulkan, cpu] matched neither "llama-cpp" nor "vllm", every candidate scored identically and size alone decided. The NVIDIA, AMD, Intel, darwin-x86 and vulkan rules were all inert. Only metal appeared to work, and only because the token "mlx" happens to equal an engine name. The mismatch does not error, it silently deletes the feature. Separate the two vocabularies. backendBuildTagPreferenceRules keeps the build tags and its original output for every capability, including metal, whose "mlx" token is removed again; its alias-resolution consumer is byte-identical to before. engineNamePreferenceRules is new, holds engine names, and is read by the new EnginePreferenceTokens, which HostResolveEnv wires into the renamed ResolveEnv.EnginePreference. Both tables sit adjacent under one block comment naming each vocabulary and each consumer, and share one lookup helper so their semantics cannot drift. On NVIDIA the order is vLLM, then SGLang, then llama-cpp: vLLM is the throughput engine and a model published with a vLLM build is published that way because that build is the one worth running. AMD and Intel get the same order, since rocm and intel builds of both serving engines ship. Metal prefers mlx over llama-cpp. Vulkan prefers llama-cpp, the only LLM engine with a Vulkan build. darwin-x86 and unknown capabilities are deliberately absent rather than guessed at, degrading to the size-only ordering that predates preference. preferenceRank stays generic and names no engine and no capability, so adding a runtime remains a one-line table edit. Specs pin the NVIDIA and metal rules through the live table and the real HostResolveEnv wiring, so emptying the engine table or wiring the build tag source back in both go red. A regression table asserts BackendPreferenceTokens' original output per capability, and mirrored locks assert neither table carries the other's vocabulary. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: record that variant selection ranks by engine before size A gallery entry can now declare variants, and selection ranks the builds a host can run by engine preference before size. Nothing told a contributor adding a backend that engineNamePreferenceRules exists, so a new engine would silently rank below every known one and lose to whatever build happened to be larger on hosts where it should have won. Document the step where a backend is added, warn against the sibling backendBuildTagPreferenceRules table (build tags, not engine names: the wrong table matches nothing, scores every candidate equally and disables the preference without erroring), and index it from AGENTS.md. Fix the authoring and user docs, which still claimed the largest surviving build wins. An author grouping builds under one entry has to be able to predict what a user gets, and size alone no longer decides it. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(cli,mcp): describe variant auto-selection as preference before size The CLI flag help and the install_model tool schema both still said auto-selection takes the largest build that runs. Ranking now puts engine preference ahead of size, so on NVIDIA a vLLM build wins over a larger llama.cpp one. An assistant reading the old schema would tell users the wrong thing. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): prefer llama.cpp over GPU serving engines on hosts with no GPU engineNamePreferenceRules had no row for the "default" capability, which getSystemCapabilities() returns both when no GPU is detected and when a GPU is present but under the 4 GiB VRAM floor. A missing row yields an empty preference list, which preferenceRank reads as "score everything equally", collapsing variant selection to size alone. That would be harmless if the hardware filter dropped GPU serving engines on such a host, but it does not. IsBackendCompatible derives support from the engine NAME, and "vllm" and "sglang" contain none of the darwin, cuda, rocm or sycl tokens it keys on, so they fall through to its closing "return true". A vLLM variant therefore survives on a CPU-only box and wins whenever its build is the larger of the two on offer: the machine installs vLLM in preference to llama.cpp. darwin-x86 had the identical hole. It was documented as a deliberate omission because nothing accelerates on an Intel Mac, which is true about acceleration and wrong about consequence: with every engine tied, download size decides. Add rows for both putting llama-cpp first. The GPU engines are enumerated behind it rather than left unmatched: an unmatched engine already ranks below every listed one, so llama.cpp would win either way, but unmatched engines also tie with each other and let size decide among them. Naming them fixes that order. MLX is left off the darwin-x86 row on purpose so it ranks last, since IsBackendCompatible admits darwin-tokened engines on that capability even though MLX needs Apple silicon. Preference orders survivors and never filters, so a model published only as a vLLM build is still installed on a host with no GPU; there is a spec for it. Surveyed every other value getSystemCapabilities() can return. nvidia, amd, intel and vulkan have rows; the l4t and cuda-refined values reach the nvidia row by prefix; "apple" and "" cannot reach the vendor fallthrough because the darwin and no-GPU branches return earlier. default and darwin-x86 were the only live holes. BackendPreferenceTokens and its build-tag table are untouched, and preferenceRank stays generic, naming no engine and no capability. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * gallery: prefer speculative-decoding builds when they fit Rank serving features between engine preference and size, so a host that can hold a DFlash or MTP build of a model's weights installs it instead of the plain build. Both answer faster for the same output, so whenever one survives the filters there is no reason to take the plain build. Precedence is now fit, then engine, then serving feature, then size. Engine outranks the feature deliberately: a serving feature makes the right engine faster, it does not make a wrong engine right, so a plain vLLM build still beats a DFlash llama.cpp build on NVIDIA. Fit outranks both, and a drafter pairing is strictly larger than the plain build, so the existing size filter drops it on a host too small for it before this axis is consulted. The order lives in a third preference table in pkg/system, alongside the build tag and engine name tables. It is the odd one of the three: not keyed by capability, because no hardware prefers a plain build over an equivalent faster one, and matched against whole segments of a gallery ENTRY NAME rather than as a substring of a backend value. Nothing on a gallery entry declares a serving feature, and tags are not a usable substitute: gemma-4-e2b-it:sglang-mtp carries an mtp tag while ornith-1.0-9b-mtp and qwen3.6-27b-nvfp4-mtp carry none. Entry names are author-supplied free text, unlike the closed engine vocabulary, so a short marker can turn up inside an unrelated word and whole segment matching is what keeps smtp-assistant from ranking as an MTP build. The block comment over the tables now documents all three together and states what each is matched against; the ranking code names no feature, so adding one stays a one-line edit to the table. 29c49203b rejected these entries as serving configurations rather than alternative builds of the same weights. The definition is now "alternative ways to serve the same model", which includes them, so regroup 14 entries under 12 parents. Judged by the files each entry points at: the qwen3.6, qwen3.5, qwen3 and deepseek pairings are the base GGUF plus a drafter, the gemma-4 QAT MTP entries are the same QAT weights at a different quantization plus an MTP drafter, and the two sglang MTP entries describe themselves as the same model served with speculative decoding. Left separate: qwen3.6-27b-mtp-pi-tune, a finetune with its own weights, and every entry whose base model LocalAI does not ship as its own row, which is the whole Qwopus line plus gemmable-4-12b-mtp, mimo-7b-mtp:sglang and qwen3.5-4b-dflash. None of the twelve parents defines a YAML anchor, so no variants key can leak through a merge key and no empty override was needed this time. The index was edited by line insertion only. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test: check env restore errors in capability and variant specs errcheck flagged ten unchecked os.Setenv and os.Unsetenv returns in the specs added while the pre-commit hook was being skipped. Restoring an env var is exactly the place a silent failure leaks state into the next spec, so assert on it rather than suppressing the linter. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): make the mtp tag authoritative for serving-feature ranking Variant auto-selection ranks survivors by fit, then engine, then serving feature, then size. The serving-feature lookup read only whole alphanumeric segments of a variant's entry name, because tags were inconsistent: every dflash entry carried a dflash tag, but only 7 of 20 MTP entries carried an mtp tag. Tag the 13 untagged MTP entries, then teach the lookup to read tags as well as names. A tag is now the authoritative signal and is compared whole and case-insensitively, which is safe precisely because a tag is a deliberate declaration rather than free text: there is no word-inside-a-word failure mode, so the segment splitting the name half needs is unnecessary there. The name check stays as a fallback rather than being replaced. Switching to tags only would have regressed the six already-grouped entries on the day it shipped, and would depend on tagging discipline that does not exist yet. The lookup still names no feature, so adding one remains a one-line edit to servingFeaturePreferenceTokens. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): make a declared tag the sole serving-feature signal Variant auto-selection ranks survivors by fit, then engine, then serving feature, then size. The serving-feature lookup recognised a speculative build by either a declared tag or a whole segment of its entry name. Drop the name half: a tag is now the only signal. A name is author-supplied free text and a naming convention is not a contract, so reading a marker out of one infers a capability nobody declared. The gallery already had the failure in it: the four NVFP4 entries name MTP-bearing weights while setting no option that enables speculative decoding, and being live variants they were winning the feature axis without answering any faster. overrides.options was considered as the replacement and rejected. It carries spec_type:draft-mtp / spec_type:draft-dflash, which is what actually turns the feature on, but that spelling is llama.cpp's config vocabulary: ds4 spells the same feature mtp_path and sglang spells it speculative_algorithm in a referenced config. Keying a cross-backend ranking decision on one backend's option syntax would rank the other backends' builds as plain. Options are the curation-time check instead, and never reach the selection logic. With no fallback left, tag correctness is load bearing, so audit every entry against the rule "tagged when the entry configures that feature, in whatever vocabulary its backend uses". Three entries configure MTP untagged and gain the tag (hy3, glm-5.2, qwythos-9b-claude-mythos-5-1m, all spec_type:draft-mtp with no marker in their names). Four carry the tag while configuring nothing and lose it: qwen3.6-27b-nvfp4-mtp, qwen3.6-35b-a3b-nvfp4-mtp, qwopus3.6-27b-coder-mtp-nvfp4 and qwopus3.6-27b-v2-mtp-nvfp4, whose only option is use_jinja:true. The dflash side was checked independently rather than assumed consistent: all five dflash entries declare spec_type:draft-dflash and all five are tagged, so it needed no edits. Four entries keep a tag that a literal spec_type-only reading would strip, because they configure MTP through a different backend: deepseek-v4-flash-q2-mtp via ds4's mtp_path/mtp_draft, and the three sglang entries via speculative_algorithm in their referenced configs. Stripping those would contradict the reason spec_type was rejected as the signal and would demote four genuinely faster builds to plain. The index was edited by line insertion and deletion only, never round-tripped through a serializer. A resolved-tag diff across all 1272 named entries, taken after merge keys are applied, shows exactly these 7 changing and no entry gaining or losing a tag through an anchor. The two specs that pinned the name fallback are inverted rather than deleted, since a name silently promoting a build is the regression worth guarding. The whole-token guard survives on the tag path, where smtp must still not match mtp. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): make deepseek-v4-flash variant targets installable Clicking install on deepseek-v4-flash failed with "invalid gallery model". The parent entry is fine, but all four entries it was grouped with declared neither url: nor config_file:, and applyModel needs one of the two to have anything to build a config from. They carry urls: (plural), the informational HuggingFace link list, which is a different field. None of the four was ever independently installable, so grouping them routed a previously-working install into a broken entry. Give each the url: the parent already resolves through. virtual.yaml is a no-op base, and applyModel passes overrides to InstallModel separately from the fetched config, so backend: ds4, the parameters and the ssd/mtp options all still land exactly as authored. This is the same pattern the parent and many other GGUF entries in the index already use. Add the lint rule that should have caught this. checkVariantReferences only proved a target exists and is not itself a parent, which is structural validity: an entry can exist, declare no variants, and still be uninstallable. checkVariantTargetsInstallable mirrors applyModel's precondition instead, and names the parent, the target and the missing fields, because whoever hits it is reading a gallery entry and has no reason to know applyModel exists. The two index-driven resolution specs live in their own Ordered container: an Ordered container stops at its first failure, so sharing one with the lint rules let a lint breach skip them silently. Nine further entries gallery-wide have the same defect and are unrelated to variants, so they are broken installs that predate this branch. They are left alone here rather than buried in a regression fix, and widening the rule to cover every entry is deferred with them so the gate can ratchet up in one step instead of needing a skip list. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(gallery): install entries with no url or config_file on an empty base applyModel had three branches: fetch a base config from url:, build one from an inline config_file:, or fail with "invalid gallery model". An entry declaring neither is now installed on an empty base config, with overrides: and files: supplying everything. This is what the ~345 entries pointing at gallery/virtual.yaml were already getting. That stub is five lines carrying name, description and license. description and license are overwritten from the gallery entry immediately after the fetch, and the name never reaches disk because InstallModel prefers the install name. Crucially applyModel passes model.Overrides to InstallModel as a separate argument rather than merging it into the fetched config, so nothing an author writes depends on that base existing. The fetch bought a round trip to GitHub and nothing else. That makes f4ef80173 the wrong fix, so it is unwound. The four url: lines it added to the deepseek-v4-flash variants are reverted: they are a pointless network fetch now, and the family installs without them. Relaxing the branch would hide a real authoring mistake, so a payload rule replaces the base-config rule. An entry with no url, no config_file, no overrides and no files installs nothing and would leave an empty model directory while reporting success, so it is refused by name. The caller's request counts toward the payload, because its overrides and files are merged into the install exactly as the entry's own are. urls: (plural) is the informational link list and does not count, which is what the four entries that shipped broken had and why they were still uninstallable. checkVariantTargetsInstallable asserted every variant target declares a url: or a config_file:, which is no longer true and would now reject correct authoring. checkEntriesInstallSomething pins what survives instead, and covers every entry rather than only variant targets: the hazard is a half-written stanza and a parent can be one as easily as a target. The old rule was scoped to targets precisely because nine unrelated entries would have failed a gallery-wide version; those nine are valid now, so the deferred ratchet happens here in one step. 1280 entries, zero violations. Those nine (aurore-reveil_koto-small-7b-it, lfm2-1.2b, the six liquidai_lfm2 entries and deepseek-v4-pro-q2-ssd) become installable for free. Each carries overrides: and files:, and one of them is driven through the real install path in a spec. The no-fetch spec is paired rather than bare: an assertion that nothing was fetched proves nothing unless something could have been, so a control runs the same fixture with a url: pointing at a base config that is not there and asserts the install fails. Only then does the identical fixture without the url passing mean the read was skipped. Follow-up, deliberately not here: the ~345 entries still naming virtual.yaml can drop their url:. That is 345 index edits with their own risk, and mixing them in would bury this change. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ui(models): let search bypass the variant collapse, drop the toggle The models page collapsed the gallery to one row per model by default and offered a toggle to see every individual build. Because the collapse composed with the search term, a build another entry offers as a variant could not be found by typing its name, so the toggle was the only way to reach those builds in the UI. A user who typed a name they knew existed got "no models found", which reads as "that model does not exist". Collapse is for browsing; search is for finding. An explicit search term now bypasses the collapse in the listing handler, so a name lookup returns matching entries whether or not a parent offers them. The term is trimmed once at the top of the handler, so whitespace is neither a search nor a bypass; previously an untrimmed blank term also narrowed the listing to whatever contained a space. Tag and backend deliberately do not bypass: they refine a listing the user is still reading rather than name an entry already known to exist. That makes the toggle redundant, so it goes, along with its i18n strings in all six locales, its localStorage persistence, its participation in "Clear filters" and the empty-state hint telling users to turn it off. The hint was doubly stale: it pointed at a control that no longer exists, and it was untrue exactly when a user has a search term, since searching now sees every build. The page always requests the collapsed listing. The stored preference key is left inert rather than cleaned up: nothing reads it, so a user who had the toggle off simply gets the collapsed view. collapse_variants stays on the API, off by default, because other clients want either view and the UI dropping its control is no reason to remove a working parameter. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): give the models gallery filter form a deliberate structure The filter area had accreted controls into one undifferentiated flow. The "Fits in GPU" toggle and the backend select were direct children of .filter-bar, the same wrapping container as the 18 taxonomy chips, so their position was decided by how many chips happened to wrap at the current width rather than by any layout intent. At narrow widths they were pushed past the right edge of that container's horizontal scroll and became unreachable entirely. Restructure into three bands inside the house .filter-bar-group wrapper that components/FilterBar.jsx already uses on Backends and the System tabs: 1. query scope: search plus the backend select 2. taxonomy: the chip row, alone, free to wrap 3. refinements: fits-in-GPU and context size, under a hairline rule The backend select leads the chips rather than trailing them because picking a backend disables the use cases that backend cannot serve, so it gates the row below it. Fits-in-GPU and context size share a band because they are one control group: the context size is the length the VRAM estimate is computed at, and that estimate is what the fits filter tests against. Chips had no visible keyboard focus indicator. The global focus ring is wrapped in :where(), so it carries the specificity of a bare :focus-visible, ties with .filter-btn and loses on source order, leaving focused chips showing their resting drop shadow. Restate the ring where it outranks both resting and hover. Also: aria-pressed on the chips, a real label association and aria-valuetext on the context slider (it steps over an index, so it announced "2"), disabled chip styling moved off inline styles, a prefers-reduced-motion block for the chip transition, and the hard-coded English "Context:" moved into all seven locales. No behaviour change: same filters, same state, same requests. Page reset on change, localStorage persistence and "Clear filters" verified unchanged. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): let the models recommendations panel fade into the background The "Recommended for your hardware" strip rendered at full height on every visit regardless of how many models were already installed, costing 186px at 1600px wide (287px at 1100px, where its cards wrapped to two rows) and pushing the first gallery row to y=554 / y=703. Make its prominence track how much the user still needs it. The panel now defaults to a one-line summary once anything is installed, and both the collapse choice and the existing dismissal persist: collapsed = explicit user choice, if one exists : installedCount > 0 The preference is three-valued on purpose. A boolean cannot tell "the user expanded it" apart from "the user has never chosen", and those need opposite handling when the installed count later crosses zero: someone who deliberately opened the panel on an empty instance should not have it collapse out from under them when their first model finishes installing. Collapsed keeps the card, icon, title and a suggestion count, so the panel is recovered by clicking what you are already looking at rather than by hunting. Expanded is unchanged, because for a user with nothing installed it was never the problem. Collapsed reclaims 145px at 1600 and 420, and 246px at 1100. Models.jsx gains a statsLoaded flag: stats initializes to installed:0, so reading it before the fetch resolves would render expanded and collapse a frame later, which is exactly the layout shove this removes. The dismissal key moves to the page's localai-models-* convention; the old localai_rec_models_dismissed is still read, never written, so an existing dismissal is honoured rather than resurrected by the rename. Accessibility: the disclosure is a real button whose accessible name is the visible title alone, with state on aria-expanded and aria-controls resolving in both states, because the grid is hidden via the hidden attribute rather than unmounted. That also keeps the four install buttons out of the tab order while collapsed. The app's global focus ring applies; no per-component outline is added, per the warning in App.css. Reveal animates opacity and transform only, never height, and both it and the chevron rotation are disabled under prefers-reduced-motion. Only en had a recommended block, so the other six locales were falling back to English for the whole panel. Translated the complete block rather than adding one orphaned key to files that would still render the title in English. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(downloader): recover from a leftover .partial on non-HTTP URIs An interrupted download leaves a `<file>.partial` behind. The partial handling in DownloadFileWithContext gated resume on `err == nil && uri.LooksLikeHTTPURL()`, so for any URI that is not literally http(s) the branch fell through to `else if !errors.Is(err, os.ErrNotExist)`, which with a nil err is true. The download then failed with an error wrapping nil: failed to check file ".../Ternary-Bonsai-27B-Q2_g64.gguf" existence: <nil> Every gallery file URI uses `huggingface://`, so a single interrupted download made that model permanently uninstallable until someone deleted the partial by hand. The `<nil>` in the message compounded it by pointing debugging at a filesystem failure that never happened. Restructure the handling as an explicit switch over the four real states: partial exists and is resumable, partial exists and is not resumable (discard and restart, as already done for an HTTP server without range support), no partial, and a genuine stat failure. The error branch is now only reachable with a non-nil error, names the path that was actually stat'd, and wraps with %w. Discarding is required for correctness and not merely convenience: the writer opens the partial with O_APPEND, so an un-resumed download would concatenate a fresh body onto stale bytes. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): tell the models gallery's variant rows apart, and let browsing see every build Both variant surfaces rendered name, backend and size. For two builds of one model that is close to no information: a variant exists precisely because the same weights are offered another way, so the backend usually matches and the sizes usually land within a few hundred megabytes. Comparing ternary-bonsai-27b-pq2 against ternary-bonsai-27b-q2-g64 meant reading two names that differ by a suffix nobody has defined anywhere in the UI. Report the quantization and the serving features on VariantView, and derive both server-side from the referenced entry rather than parsing names in the browser, so every client reads the same format out of the same file the installer will hand the backend. Quantization comes from overrides.parameters.model first, falling back to the file list. That order is load bearing: entries routinely ship a vision tower alongside the language model at a different quantization, so reading the file list first reports the mmproj's format. Matching walks `-` and `.` delimited segments right to left; `_` deliberately does not split, because it separates the parts INSIDE a quant token and splitting on it reports Q4 for a Q4_K_M build. A second, looser pass takes a segment's `_`-delimited tail, which catches the gemma-4-E2B_q4_0-it.gguf style; it runs second so a precise match can never lose to a fuzzy one further right in the name. An entry naming no format reports nothing, which is the honest answer for a backend served from a directory of weights. Features are the same tag-against-vocabulary match servingFeatureRank already ranks on, over the same host preference list. A build can therefore never be shown as faster than one selection did not actually reward, nor rewarded without being shown; a spec pins that agreement rather than trusting it. The compact dropdown gets the quantization on its meta line and the bare feature token. The detail row, which has the room, gets the quantization as its own monospaced column so precision lines up down the list, and the feature spelled out, because DFLASH names nothing to a user who has not met it. The referenced entry's description stays out of both: the detail row already renders the parent's prose above the table, and a second block per variant would push a three-variant list past a screen to restate what the columns now say precisely. The collapse toggle comes back. 462583f38 dropped it once search bypassed the collapse, on the reasoning that nothing was unreachable any more. That holds for finding a build whose name you know and does not hold for browsing: no sequence of actions enumerated the 68 builds the default view hides. Collapse is for browsing and search is for finding, and the toggle was the browsing half. It goes in the refinements band 0d4823362 established, not back among the taxonomy chips where its position depended on how many chips happened to wrap. It leads that band because it decides how many rows the other two refine over, and because unlike fits-in-GPU it is unconditional: a host with no GPU still browses. The search bypass is untouched and re-checked by a spec in the toggle's default state, since restoring the control must not restore the dead end it replaced. The empty-state hint returns but only without a search term, because a term bypasses the collapse and the hint would otherwise point at a control that cannot change the result. The stored preference reads 'on'/'off' only: an older build wrote '1'/'0' from an effect that ran on mount, so those record that the page was opened, not that anyone chose a view. Also fixes a latent flake it exposed. The collapse_variants spec compared whole response bodies byte for byte, and the listing envelope carries live host telemetry that drifts between two calls milliseconds apart, so it was asserting on the machine's memory pressure. It now compares everything the parameter governs -- the entries, their serialization and the paging -- and is green 25/25 where it was failing about one run in three. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): let the models gallery show a variant's full details The variant list in an entry's expanded detail row says how the builds differ: name, backend, quantization, size, and the auto-selected, base and serving-feature markers. It cannot say what any one of them is. A variant's own description, tags, license, source links and file list are unreachable anywhere in the UI, because while the collapse is on a variant has no gallery row of its own. Give each variant row an info control that reveals its entry, rendered by the same ModelDetail a top-level row gets, so a field added to the detail view appears here too. variantData is withheld from the nested render: a variant may declare variants of its own, and recursing would nest a picker inside a picker two levels deep already. An inline disclosure rather than a modal. The control sits inside a table row that is already expanded, inside a variant list within that; a dialog opened from there stacks a dismissal on a dismissal for a handful of extra fields about the entry the user is already reading, and breaks the page's own expand idiom. The third level is carried by an inset and a left rule instead of another card. The entry is fetched by exact name from the listing, once, on first use. The listing already returns every field the detail view renders, and a search term bypasses the variant collapse server-side, so no new endpoint is needed and neither the listing nor DescribeVariants gains any work. Expanding a row costs nothing; a variant nobody opens costs nothing. A name the listing no longer returns is stated, not blanked: an empty panel reads as a rendering fault rather than as a lookup that came back empty. The control is a sibling of the install button, not a descendant, so asking about a build can never install it. The variant list keeps its content-sized columns via a trailing filler track instead of max-content sizing, so the rows are unchanged while the panel spanning them gets the pane width its file table needs. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): let search respect the collapse instead of switching it off The models listing collapsed to one row per model, and an explicit search term turned that off wholesale. Searching while collapsed therefore answered with the individual builds a parent already offers, which are exactly the rows the view the user asked for has no place for: typing "mtp" returned qwen3.6-27b-nvfp4-mtp, a row that is invisible the moment the box is cleared. The bypass was the right shape of fix for the wrong half of the problem. What a search must not do is answer "no models found" for a build the gallery does hold; that does not require abandoning the grouping the user asked for. So the term is now matched against every entry either way, hidden builds included, and the collapse decides how a match is reported rather than which matches exist. Collapsing stops being a filter that drops rows and becomes a substitution: a match on a build another entry offers is reported as that entry, the one installable in its own right. Nothing becomes unfindable and nothing comes back that the requested view cannot show. Substitution happens after search, tag and backend, so every filter is judged against the build that really carries the name, tag or backend rather than against a parent that merely offers it; the other order would let backend=vllm match a parent whose own backend is something else. The price is that the surfaced row shows the parent's own metadata while the match was on a variant, which is what grouping means, and the alternative is claiming the gallery holds no such build. It happens before the count and the page math, so both describe the rows actually handed out rather than the matches that produced them. A parent already in the result keeps its own position and absorbs its matching variants there, which is what leaves the browsing listing ordered exactly as it was; a parent surfaced only by a variant takes the position of the first variant that surfaced it. Either way it appears once, however many of its builds matched and whether or not it matched itself. Search preserves gallery order rather than scoring, so a surfaced parent has a real position rather than an invented one. VariantParents never reports an entry that declares variants of its own, so a parent is never itself hidden and one hop always lands on a visible row. The handler follows exactly one anyway: refusing the second is what makes a gallery the linter would have rejected terminate rather than loop. The empty-state hint pointing at the toggle goes with it for every server-side filter. Substitution means a match is always reported as some row, so the collapse can no longer be why a term, a chip or a backend came back empty, and naming it there sends the user to a control that cannot change the result. It survives for the fits filter alone, which runs in the browser after the substitution and judges the surfaced entry's own size: there the build that fits really can be filtered out along with a parent that does not. Searching a build's exact name while collapsed now answers with its parent, so the result no longer contains the string the user typed. That is intended, and the row is the one they can act on, but it is a real rough edge: nothing on the row explains the connection. Closing it properly means reporting which variant matched so the UI can say so, which the listing does not do today. ResetGalleryModelCache is added for tests. The model cache is a package global keyed by nothing, so a background refresh one spec triggers can land in the middle of the next and answer it with the previous spec's gallery; the extra specs here made that fail about one run in five. It waits for the in-flight refresh to publish before clearing, since clearing alone only narrows the window. Assisted-by: Claude:claude-opus-4-8 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-20 18:43:02 +02:00
mudler's LocalAI [bot]	40d35c0385	docs: onboarding overhaul, dedup, and error docs (#7711) (#10895) ... * docs: fix CPU image tag (latest, not latest-cpu) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: use canonical localai/localai registry in models guide Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: replace dead llama-stable backend with llama-cpp Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: correct mitm-proxy intercept config and redaction tier Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: fix text-to-audio endpoint and broken notice block Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: fix VAD example, stale FAQ, broken link, CLI list, whats-new dump Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: render advanced/reference section indexes (consolidate _index) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: remove duplicate getting-started build/kubernetes pages Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: fold container image reference into installation/containers Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: remove stale advanced fine-tuning page (superseded by features/fine-tuning) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: fold distribution/longcat/sound pages into their parents Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: make getting-started index accurate and complete Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: carry one concrete model through the getting-started path Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add end-to-end 'build your first agent' walkthrough Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add runtime errors reference; consolidate troubleshooting from FAQ Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add agent actions catalog Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: agent-scoped MCP, skills walkthrough, agentic disambiguation Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add concrete gallery install lines to media feature pages Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: merge installation into getting-started (URLs preserved via aliases) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add Operations section; move operator pages and P2P API reference Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: journey-ordered top nav and grouped feature sections Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: add docs-with-code process gate (PR template + agent instructions) Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs: remove em/en dashes from documentation prose Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-07-17 22:08:20 +02:00
LocalAI [bot]	f2ed63e39a	docs(backends): make OS coverage explicit + require darwin support for new backends (#10516) ... docs(backends): make OS coverage explicit + require darwin for new backends The backend matrix is the source of truth for which OS a backend ships on, but that was never written down, so backends were landing Linux-only by default even when the engine builds fine on macOS. - .github/backend-matrix.yml: header block documenting the two matrices (include = Linux, includeDarwin = macOS/Apple Silicon) and the policy that new backends target every OS they can build for. - .agents/adding-backends.md: a 'Cover every OS' subsection in step 2 (full darwin wiring: includeDarwin entry, index.yaml metal: + metal-<backend> entries, run.sh DYLD branch + inferBackendPathDarwin case for C++ backends, the hw_grpc_proto protobuf/grpc link gotcha, and the path-filter touch) plus a verification-checklist item. - AGENTS.md (CLAUDE.md): Quick Reference pointer so it surfaces every session. Assisted-by: Claude:opus-4.8 [Claude Code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-06-25 23:26:39 +02:00
Richard Palethorpe	5a0013defe	test(react-ui): add page render-smoke specs, reset the coverage gate (#10122) ... The UI coverage gate was tightened to 0.1pp against a fast-local measurement (39.86% baseline); CI's slower runners measure ~0.9pp lower, so tests-ui-e2e failed there. UI e2e coverage is diffusely non-deterministic and tracks machine speed — a 0.1pp band can't hold across environments. Rather than loosen the gate, raise the floor under it: a render-smoke spec mounts each lazy page (navigate + assert the header renders), covering a dozen previously-untested pages and lifting coverage from ~39% to ~42.7% locally. Restore the tolerance to 0.8pp and set the baseline conservatively (40.0), below the slow-CI floor, so the ratchet holds without flapping. Document the coverage policy — install the git hooks and don't bypass them (no --no-verify, no hand-lowering the baseline or widening the tolerance); raise coverage by adding tests instead; set the UI baseline below the slow-CI floor — in AGENTS.md, CONTRIBUTING.md and .agents/building-and-testing.md. Assisted-by: Claude:claude-opus-4-8 [Claude Code] Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-06-01 14:24:36 +02:00
Richard Palethorpe	5d0b549049	feat(gallery): verify backend OCI images with keyless cosign (#9823) ... * feat(gallery): verify backend OCI images with keyless cosign Close a trust gap where a registry compromise or MITM could silently replace a backend image: the gallery YAML tells LocalAI which image to pull, but until now nothing verified the bytes came from our CI. Consumer (pkg/oci/cosignverify): - New package using sigstore-go to verify keyless-cosign signatures. - OCI 1.1 referrers API + new bundle format (no legacy :tag.sig). - Policy fields: Issuer / IssuerRegex / Identity / IdentityRegex / NotBefore. NotBefore is the revocation lever — keyless Fulcio certs are ephemeral so revocation is policy-side; advancing not_before in the gallery YAML invalidates every signature predating the cutoff. - TUF trusted root cached process-wide so N backends from one gallery do 1 fetch, not N. Plumbing: - pkg/downloader: ImageVerifier interface + WithImageVerifier option threaded through DownloadFileWithContext. Verification runs between oci.GetImage and oci.ExtractOCIImage, with digest pinning via pinnedImageRef to close the TOCTOU window. Skips the verifier's HEAD when the ref is already digest-pinned. - core/config: Gallery.Verification YAML block. - core/gallery: backendDownloadOptions builds the verifier from the policy; applied on initial URI, mirrors, and tag fallbacks. - core/gallery/upgrade: the upgrade path now routes through the same options builder. A regression Ginkgo spec pins this contract — without it, UpgradeBackend silently bypassed verification. - core/cli: --require-backend-integrity (LOCALAI_REQUIRE_BACKEND_INTEGRITY) escalates missing policy / empty SHA256 from warn to hard-fail. Producer (.github/workflows/backend_merge.yml): - id-token: write at job scope (PR-fork-safe via existing event gate). - sigstore/cosign-installer@v3 pinned to v2.4.1. - After each docker buildx imagetools create, resolve the manifest list digest and run cosign sign --recursive --new-bundle-format --registry-referrers-mode=oci-1-1 against repo@digest. --recursive signs the index and every per-arch entry, matching how the consumer resolves a tag to a platform-specific manifest before verifying. Rollout: backend/index.yaml has no `verification:` block yet, so this PR is backward-compatible — installs proceed with a warning until the gallery is populated. Strict mode is opt-in. Assisted-by: claude-code:claude-opus-4-7 [Bash] [Edit] [Read] [Write] [WebSearch] [WebFetch] Signed-off-by: Richard Palethorpe <io@richiejp.com> * refactor(gallery): plumb RequireBackendIntegrity through config instead of env The previous implementation re-exported the --require-backend-integrity CLI flag into LOCALAI_REQUIRE_BACKEND_INTEGRITY via os.Setenv, then re-read it in core/gallery via os.Getenv. This leaked process state into the gallery package and made the flag impossible to override per-call or test without touching the env. Add RequireBackendIntegrity to ApplicationConfig (with a matching WithRequireBackendIntegrity AppOption) and thread the bool through every install/upgrade path: InstallBackend, InstallBackendFromGallery, UpgradeBackend, InstallModelFromGallery, InstallExternalBackend, ApplyGalleryFromString/File, startup.InstallModels. Worker subcommands gain the same env-bound flag on WorkerFlags so distributed-worker installs honor it consistently with the worker daemon path. Add a forbidigo lint rule against os.Getenv / os.LookupEnv / os.Environ to keep the env-leak pattern from creeping back. Existing offenders (p2p, config loaders, etc.) are baseline-grandfathered by the existing new-from-merge-base: origin/master setting; targeted path exclusions cover the legitimate cases — kong CLI entry points, backend subprocesses, system capability probes, gRPC AUTH_TOKEN inheritance, test gating env vars. Assisted-by: claude-code:claude-opus-4-7 Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-05-18 08:02:20 +02:00
LocalAI [bot]	d892e4af80	feat: add ds4 backend (DeepSeek V4 Flash) with tool calls, thinking, KV cache (#9758) ... * test(e2e-backends): allow BACKEND_BINARY for native-built backends Adds an escape hatch for hardware-gated backends (e.g. ds4) where the model is too large for Docker build context. When BACKEND_BINARY points at a run.sh produced by 'make -C backend/cpp/<name> package', the suite skips docker image extraction and drives the binary directly. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(e2e-backends): validate BACKEND_BINARY basename + log actual source Two follow-ups from the cbcf5148 code review: - BACKEND_BINARY now requires a path whose basename is `run.sh`. Without this check, `filepath.Dir(binary)` silently discarded the filename, so pointing the env var at an arbitrary binary failed later with a confusing assertion that named a path the user never typed. - The "Testing image=..." debug line printed an empty string when the binary path was used, hiding the actual source in CI logs. The line now reports whichever of BACKEND_IMAGE / BACKEND_BINARY is in effect as `src=...`. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): scaffold ds4 backend dir Adds prepare.sh, run.sh, and a .gitignore. CMakeLists, Makefile, and the implementation arrive in follow-up commits. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): add backend Makefile Drives ds4's upstream Makefile to produce engine .o files (CUDA on Linux when BUILD_TYPE=cublas, Metal on Darwin, otherwise CPU debug path), then invokes CMake on our wrapper. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): add CMakeLists for grpc-server Generates protoc stubs from backend.proto, links grpc-server.cpp + dsml_parser.cpp + dsml_renderer.cpp + kv_cache.cpp against pre-built ds4 engine .o files. DS4_GPU=cuda|metal|cpu selects the backend. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): grpc-server skeleton + module stubs The minimum that links: Backend service with Health + Free; other RPCs default to UNIMPLEMENTED. Stub headers/sources for dsml_parser, dsml_renderer, and kv_cache are in place so CMake links cleanly even before those modules ship. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): implement LoadModel Opens engine + creates session sized to ContextSize (default 32768). Backend is compile-time: CPU when DS4_NO_GPU, Metal on __APPLE__, else CUDA. MTP/speculative options are accepted via ModelOptions.Options[] (mtp_path, mtp_draft, mtp_margin). kv_cache_dir option is captured into g_kv_cache_dir for the cache module (Task 19 wires it in). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): implement TokenizeString Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): implement Predict (plain text) Tool calls + thinking-mode split arrive in Task 13 once dsml_parser is in. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): implement PredictStream (plain text) ChatDelta + reasoning/tool_calls split arrives in Task 14. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): implement Status RPC Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): add DSML streaming parser Classifies raw model-emitted token text into CONTENT / REASONING / TOOL_START / TOOL_ARGS / TOOL_END events. Markers it watches for are the literal DSML strings rendered by ds4_server.c's prompt template (<｜DSML｜tool_calls>, <｜DSML｜invoke name=...>, <think>, etc.) - these are plain text the model emits, not special tokens. Partial markers split across token chunks are buffered until a full marker or a definitively-not-a-marker '<' is observed. RandomToolId() generates the API-side tool call id (call_xxx) that exact-replay would key on. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(backend/cpp/ds4): split hex escapes in DSML markers + add cstring/cstdio includes C++ \x hex escapes have no length cap. '\x9cD' was read as a single escape producing byte 0xCD, eating the 'D'. The markers were never actually matching the DSML text the model emits. Split each escape with adjacent string literal concatenation so the byte sequence is exactly EF BD 9C 44 (｜D) at runtime. Also adds <cstring> and <cstdio> includes (libstdc++ 13 does not transitively expose std::strlen / std::snprintf via <string>). The local plan file (uncommitted) was also updated with the same fixes so Task 16's dsml_renderer.cpp does not re-introduce the bug. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): wire DsmlParser into Predict (ChatDelta) Non-streaming Predict now emits one ChatDelta carrying content, reasoning_content, and tool_calls[] parsed from the model's DSML output. Reply.message still carries the raw model bytes for backends that prefer the regex fallback path. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): wire DsmlParser into PredictStream Per-token ChatDelta writes: content/reasoning_content go incrementally, tool_calls emit TOOL_START as one delta (id + name) followed by TOOL_ARGS deltas with incremental JSON. The Go-side aggregator (pkg/functions/chat_deltas.go) reassembles them. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): chat template + reasoning_effort mapping UseTokenizerTemplate=true + Messages -> ds4_chat_begin / append / assistant_prefix. PredictOptions.Metadata['enable_thinking'] and ['reasoning_effort'] map to ds4_think_mode (DS4_THINK_HIGH default; 'max'/'xhigh' -> DS4_THINK_MAX; disabled -> DS4_THINK_NONE). Tool-call rendering for assistant turns with tool_calls JSON arrives in the next commit (dsml_renderer). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): render assistant tool_calls + tool results to DSML Closes the round-trip: when an OpenAI client sends a multi-turn chat where prior turns contain tool_calls or role=tool messages, build_prompt serializes them back to the DSML shape the model was trained on. Mirrors ds4_server.c's prompt renderer; uses nlohmann::json for parsing the OpenAI tool_calls payload. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): disk KV cache module Dir-based cache keyed by SHA1(rendered prompt prefix). File format: 'DS4G' magic + version + ctx_size + prefix_len + prefix + payload_bytes + ds4_session_save_payload output. NOT bit-compatible with ds4-server's KVC files - that interop is a follow-up plan. LoadLongestPrefix walks the dir picking the longest stored prefix that prefixes the incoming prompt. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): wire KvCache into Predict/PredictStream LoadModel reads 'kv_cache_dir' from ModelOptions.Options[], passes it to g_kv_cache.SetDir. Each Predict/PredictStream computes a render text for the request, tries LoadLongestPrefix to recover state, then Saves the new state after generation. ds4_session_sync handles the live-cache fast path internally, so the disk cache only matters for cold-starts and cross-session reuse. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): add package.sh Linux: bundles libc + ld + libstdc++ + libgomp + GPU runtime libs into package/lib so the FROM scratch image boots without a host libc. Darwin is handled by scripts/build/ds4-darwin.sh which uses otool -L. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(backend/cpp/ds4): rename namespace ds4_backend -> ds4cpp ds4.h defines 'typedef enum {...} ds4_backend' which collides with our C++ 'namespace ds4_backend' anywhere a TU includes both. kv_cache.h includes ds4.h directly and surfaces the conflict immediately; other TUs would hit it once gRPC dev headers are available. Renames the C++ namespace to ds4cpp across all wrapper files and the plan, leaving the upstream ds4 typedef untouched. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend): add Dockerfile.ds4 Single-stage builder (CUDA devel image for cublas, ubuntu:24.04 for cpu) -> FROM scratch with packaged grpc-server + bundled runtime libs. nlohmann-json3-dev is required for dsml_renderer's JSON handling. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(make): wire backend/cpp/ds4 + ds4-darwin into root Makefile BACKEND_DS4 entry + generate-docker-build-target eval + docker-build-ds4 in docker-build-backends + .NOTPARALLEL guards. Also adds the backends/ds4-darwin target which delegates to scripts/build/ds4-darwin.sh (landed in Task 24). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci: add backend-matrix entries for ds4 (cpu + cuda13, per-arch) Two entries per build (amd64 + arm64) so backend-merge-jobs assembles a multi-arch manifest. Skipping cuda12 - ds4 was validated against CUDA 13. Darwin Metal is handled outside this matrix by backend_build_darwin.yml. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/index): add ds4 meta + image entries cpu + cuda13 x latest + master. Darwin Metal builds publish under ds4-darwin via the existing llama-cpp-darwin OCI pipeline. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(scripts/build): add ds4-darwin.sh Native macOS/Metal build for the ds4 backend. Mirrors llama-cpp-darwin.sh: make grpc-server -> otool -L for dylib bundling -> OCI tar that 'local-ai backends install' consumes via the backends/ds4-darwin Makefile target. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * ci(darwin): build ds4-darwin in backend_build_darwin Adds a 'Build ds4 backend (Darwin Metal)' step that runs the backends/ds4-darwin Makefile target on the macOS runner. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(import): auto-detect ds4 weights via DS4Importer Adds core/gallery/importers/ds4.go which matches on the antirez/deepseek-v4-gguf repo URI and the DeepSeek-V4-Flash-*.gguf filename pattern. Registered before LlamaCPPImporter so ds4 weights route to backend: ds4 instead of falling through to llama-cpp. Also lists ds4 in /backends/known so the /import-model UI surfaces it as a manual choice for users who want to force the backend on a non-canonical URI. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add deepseek-v4-flash-q2 (ds4 backend) One-click install of the q2 weights with backend: ds4. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * docs(.agents): add ds4-backend.md Documents the backend shape, DSML state machine, thinking-mode mapping, disk KV cache, build matrix (cpu/cuda13/Darwin), and the BACKEND_BINARY hardware-validation path. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(backend/cpp/ds4): pass UBUNTU_VERSION + arch env vars to install-base-deps The .docker/install-base-deps.sh script needs UBUNTU_VERSION (defaults to 2404), TARGETARCH, SKIP_DRIVERS, and APT_MIRROR/APT_PORTS_MIRROR exported into the environment so it can pick the right cuda-keyring / cudss / nvpl debs and apt mirrors. Dockerfile.ds4 was declaring some of the ARGs but not re-exporting them via ENV. Mirrors Dockerfile.llama-cpp's pattern. Without this fix 'make docker-build-ds4 BUILD_TYPE=cublas CUDA_MAJOR_VERSION=13' failed at: /usr/local/sbin/install-base-deps: line 120: UBUNTU_VERSION: unbound variable Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/index): add Metal image entries for ds4 Adds metal-ds4 + metal-ds4-development image entries pointing at quay.io/go-skynet/local-ai-backends:{latest,master}-metal-darwin-arm64-ds4 (built by scripts/build/ds4-darwin.sh on macOS arm64 runners), plus the 'metal' and 'metal-darwin-arm64' capability mappings on the ds4 meta and ds4-development variant. Closes a gap from the initial Task 23 landing - the Darwin Metal build script and CI workflow step were already wired (Tasks 24-25), but the gallery had no image entry for users to install the Metal variant. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(ci): use ubuntu:24.04 base for ds4 cuda13 matrix entries The initial Task 22 matrix landing used base-image: 'nvidia/cuda:13.0.0-devel-ubuntu24.04' which clashes with install-base-deps.sh's cuda-keyring step: E: Conflicting values set for option Signed-By regarding source https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/ The canonical pattern (llama-cpp, ik-llama-cpp, turboquant) uses plain 'ubuntu:24.04' + 'skip-drivers: false' so install-base-deps installs CUDA from scratch via its own keyring setup. Adopting that here. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(backend/cpp/ds4): drop install-base-deps.sh dependency The .docker/install-base-deps.sh pipeline is built around the llama-cpp needs: NVIDIA keyring + cuda-toolkit apt + gRPC-from-source build at /opt/grpc. For ds4 we don't need any of that: - CUDA: nvidia/cuda:13.0.0-devel-ubuntu24.04 ships /usr/local/cuda ready to go; install-base-deps's keyring step then conflicts with the pre-installed Signed-By. - gRPC: ds4's grpc-server.cpp only links against grpc++; system libgrpc++-dev (apt) is sufficient, no source build needed. Replaced the install-base-deps invocation in Dockerfile.ds4 with a direct 'apt-get install libgrpc++-dev libprotobuf-dev protobuf-compiler-grpc nlohmann-json3-dev cmake build-essential pkg-config git'. Matrix entries back to nvidia/cuda base + skip-drivers=true so install-base-deps would no-op even if some downstream tooling calls it. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(backend/cpp/ds4): correct proto accessors + alias grpc::Status as GStatus Two compile bugs caught by the docker build: 1. proto::Message uses snake_case accessors. The build_prompt loop called m.toolcalls() / m.toolcallid() - the protoc-generated names are m.tool_calls() / m.tool_call_id(). Plan-text bug propagated to the wrapper. 2. The Status RPC method shadowed the 'using grpc::Status' alias, so any later method declaration using Status as a return type failed to parse ('Status does not name a type' starting at LoadModel). Solution: alias grpc::Status as GStatus instead, with no 'using' clause that would conflict. All RPC method declarations and return-statement constructions now use GStatus. Pre-existing code reviewer flagged the Status-shadow concern as 'minor' in the original Task 10 commit; it turned out to be a real compile blocker under libstdc++ 13 once the surrounding methods were filled in. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(backend/cpp/ds4): preserve TOOL_ARGS content in dsml_parser Flush When the model emitted a parameter value that arrived in the same buffer as the surrounding tool_call markers (e.g. the buffered tail after a literal '</think>' opened the model output), the parser deferred all buffered bytes to Flush() because looks_like_prefix() always returns true while buf starts with '<'. Flush() then drained the buffer as plain CONTENT/REASONING regardless of parser state, so the bytes between the parameter open and close markers were classified as CONTENT instead of TOOL_ARGS. Symptom: the model emitted <|DSML|parameter name="location" string="true">Paris, France</|DSML|parameter> and the assembled tool_call arguments came out as {"location":""} - the opener and closer were emitted into the args stream but the "Paris, France" content went to the assistant message instead. Fix: 1. Flush() now uses the same state-aware emit logic as DrainPlain: PARAM_VALUE bytes become TOOL_ARGS (json-escaped when string), THINK bytes become REASONING, TEXT bytes become CONTENT, and INVOKE / TOOL_CALLS structural whitespace is discarded. 2. looks_like_prefix() restricts its leading-'<' fallback to buffers that have not yet seen a '>'. Without that change, char-by-char feeds would discard the '<' of '<|DSML|invoke name="..."' once the marker prefix length was reached but the closing quote/'>' were still in flight. Verified with a standalone harness that runs the failing input three ways (single Feed, split-after-'>', and char-by-char) and aggregates TOOL_ARGS for tool index 0: all three now produce {"location":"Paris, France"}. Assisted-by: Claude:opus-4.7 [Read,Edit,Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(backend/cpp/ds4): use ds4_session_sync + manual generation loop for KV persistence ds4_engine_generate_argmax() is a self-contained helper that doesn't take or update a ds4_session - it manages its own internal state. Our Predict and PredictStream methods created g_session via ds4_session_create() but then called ds4_engine_generate_argmax(), so g_session's KV state never advanced. ds4_session_payload_bytes(g_session) returned 0 and the disk KV cache save correctly rejected with 'session has no valid checkpoint to save'. Switch both RPCs to the proper session API: ds4_session_sync(g_session, &prompt, ...) loop: int token = ds4_session_argmax(g_session) if token == eos: break emit(token) ds4_session_eval(g_session, token, ...) After the loop the session has a real checkpoint and ds4_session_save_payload writes the KV state to disk. Verified end-to-end on a DGX Spark GB10: three .kv files (15-30 MB each) are written when BACKEND_TEST_OPTIONS sets kv_cache_dir, and the e2e tool-call assertion still passes. Also added stderr diagnostics to KvCache (enabled/disabled at SetDir; per-save path + payload_bytes + result) so future failures are visible instead of silent. The 'wrote ok' lines are low-volume - one per Predict/PredictStream when the cache is enabled - and skipped entirely when the option is unset. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): use ds4_session_eval_speculative_argmax when MTP loaded Wires MTP (Multi-Token Prediction) speculative decoding into the manual generation loop in both Predict and PredictStream. When the upstream MTP weights are loaded via 'mtp_path:' option AND we're on CUDA / Metal, ds4_engine_mtp_draft_tokens() returns >0 and we switch the inner loop to ds4_session_eval_speculative_argmax(), which can accept N>1 tokens per verifier step. When MTP is not loaded (no option, CPU backend, or weights absent), we fall through to the simple ds4_session_argmax + ds4_session_eval path with no behavior change. Validated on a DGX Spark GB10 with the optional MTP GGUF (DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf, ~3.6 GB). LoadModel logs 'ds4: MTP support model loaded ... (draft=2)' on stderr. Caveat per upstream README: 'currently provides at most a slight speedup, not a meaningful generation-speed win'. Wired now mainly to track the upstream API; bigger speedups arrive when ds4 improves the speculative path. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(backend/cpp/ds4): honor PredictOptions sampling with DSML-aware override Mirrors ds4_server.c:7102-7115 sampling-policy semantics on the LocalAI gRPC side. The generation loop now consults compute_sample_params() per token to pick the effective (temperature, top_k, top_p, min_p), based on: 1. Request defaults: PredictOptions.temperature / .topk / .topp / .minp 2. Thinking-mode override: when enable_thinking != false, force T=1.0, top_k=0, top_p=1.0, min_p=0.0 (creativity for the reasoning pass and the trailing content) 3. DSML structural override: when DsmlParser::IsInDsmlStructural() returns true (we are between tool-call markers but NOT in a param value payload), force T=0.0 so protocol bytes parse cleanly When the effective temperature is 0, we keep using ds4_session_argmax + MTP speculative path (matches ds4-server's gate that only enables MTP for greedy positions). When > 0, we call ds4_session_sample(s, T, ...) with a per-thread RNG seeded from system_clock and fall back to single-token ds4_session_eval. New public method on DsmlParser: IsInDsmlStructural() encodes which states need protocol-byte determinism. PARAM_VALUE is excluded (payload uses user sampling); TEXT and THINK are excluded (no tool-call context to protect). Verified on the DGX Spark GB10: the e2e suite still passes with all 5 specs including tools, and the Predict output now varies between runs (creative sampling active) while the tool-call args remain a clean '{"location":"Paris, France"}' because the parser-state check forces greedy on the structural bytes. UX note: thinking mode is ON by default (matching ds4-server). Users who want deterministic output should set Metadata.enable_thinking = false. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(gallery): add sha256 to deepseek-v4-flash-q2 entry Per HF LFS metadata for antirez/deepseek-v4-gguf: size: 86720111200 bytes (~80.76 GiB) sha256: 31598c67c8b8744d3bcebcd19aa62253c6dc43cef3b8adf9f593656c9e86fd8c LocalAI's downloader verifies sha256 when present, so users who install deepseek-v4-flash-q2 from the gallery get integrity-checked weights and the partial-download issue (an 81 GB file is easy to truncate) becomes recoverable instead of silently producing a broken backend. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-11 22:15:47 +02:00
LocalAI [bot]	6fd21d5cf3	docs(agents): update CI caching docs after the GHA-free-tier migration (#9742) ... The migration shipped over a sequence of PRs (#9726 → #9727 → #9730 → #9731 → #9737 → #9738 plus a handful of direct-to-master fixes) and left the .agents/ docs significantly out of date. Updated: - .agents/ci-caching.md (significant rewrite) - Cache key shape: now includes per-arch suffix (cache<suffix>-<arch>). - New "Workflow surfaces" overview table. - New "Pre-built base images (base-grpc-*)" section covering the 10 quay.io/go-skynet/ci-cache:base-grpc-* tags, the multi-target Dockerfile pattern (builder-fromsource / builder-prebuilt / aliasing FROM), the BUILDER_BASE_IMAGE → BUILDER_TARGET derivation, the bootstrap-on-branch order for new variants. - New "Per-arch native builds + manifest merge" section: split matrix entries, push-by-digest, backend_merge.yml, why provenance: false matters. - New "Path filter on master push" section: changed-backends.js handles push events via the Compare API; weekly Sunday cron is the safety net for unpinned Python deps. - New "ccache for C++ backend builds" section. - New "Composite actions" section: free-disk-space and setup-build-disk. - New "Concurrency" section documenting the per-PR-per-commit group fix. - Darwin section gains the brew link --overwrite note (after- cache-restore symlinks weren't restored) and the llama-cpp-darwin consolidation context. - "Self-hosted runners" section confirming the matrix is free of arc-runner-set / bigger-runner references except the residual test-extra.yml vibevoice case. - "Touching the cache pipeline" rule list extended (provenance, install-base-deps.sh single-source-of-truth, base-images bootstrap order). - .agents/adding-backends.md - Section 2 title: backend.yml -> backend-matrix.yml (path moved). - New paragraph on per-arch entries (platform-tag + paired matrix rows + auto-firing merge job). - New paragraph on builder-base-image for llama-cpp / ik-llama-cpp / turboquant. - Final checklist line updated accordingly. - .agents/building-and-testing.md - Reference: backend.yml -> backend-matrix.yml. - Note about builder-base-image and BUILDER_TARGET defaulting to builder-fromsource for local builds. - AGENTS.md - One-line description update for ci-caching.md to mention the new infrastructure (per-arch keys, base-grpc-*, manifest-merge, setup-build-disk, path filter). Assisted-by: Claude:claude-opus-4-7 Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>	2026-05-10 00:28:57 +02:00
Richard Palethorpe	c894d9c826	feat(sglang): wire engine_args, add cuda13 build, ship MTP gallery demos (#9686) ... Bring the sglang Python backend up to feature parity with vllm by adding the same engine_args:-map plumbing the vLLM backend already has. Any ServerArgs field (~380 in sglang 0.5.11) becomes settable from a model YAML, including the speculative-decoding flags needed for Multi-Token Prediction. Validation matches the vllm backend's: keys are checked against dataclasses.fields(ServerArgs), unknown keys raise ValueError with a difflib close-match suggestion at LoadModel time, and the typed ModelOptions fields keep their existing meaning with engine_args overriding them. Backend code: * backend/python/sglang/backend.py: add _apply_engine_args, import dataclasses/difflib/ServerArgs, call from LoadModel; rename Seed -> sampling_seed (sglang 0.5.11 renamed the SamplingParams field). * backend/python/sglang/test.py + test.sh + Makefile: six unit tests exercising the helper directly (no engine load required). Build / CI / backend gallery (cuda13 + l4t13 paths are now first-class): * backend/python/sglang/install.sh: add --prerelease=allow because sglang 0.5.11 hard-pins flash-attn-4 which only ships beta wheels; add --index-strategy=unsafe-best-match for cublas12 so the cu128 torch index wins over default-PyPI's cu130; new pyproject.toml-driven l4t13 install path so [tool.uv.sources] can pin torch/torchvision/ torchaudio/sglang to the jetson-ai-lab index without forcing every transitive PyPI dep through the L4T mirror's flaky proxy (mirrors the equivalent fix in backend/python/vllm/install.sh). * backend/python/sglang/pyproject.toml (new): L4T project spec with explicit-source jetson-ai-lab index. Replaces requirements-l4t13.txt for the l4t13 BUILD_PROFILE; other profiles still go through the requirements-*.txt pipeline via libbackend.sh's installRequirements. * backend/python/sglang/requirements-l4t13.txt: removed; superseded by pyproject.toml. * backend/python/sglang/requirements-cublas{12,13}{,-after}.txt: pin sglang>=0.5.11 (Gemma 4 floor); add cu130 torch index for cublas13 (new files) and cu128 torch index for cublas12 (default PyPI now ships cu130 torch wheels by default and breaks cu12 hosts). * backend/index.yaml: add cuda13-sglang and cuda13-sglang-development capability mappings + image entries pointing at quay.io/.../-gpu-nvidia-cuda-13-sglang. * .github/workflows/backend.yml: new cublas13 sglang matrix entry, mirroring vllm's cuda13 build. Model gallery + docs: * gallery/sglang.yaml: base sglang config template, mirrors vllm.yaml. * gallery/sglang-gemma-4-{e2b,e4b}-mtp.yaml: Gemma 4 MTP demos transcribed verbatim from the SGLang Gemma 4 cookbook MTP commands. * gallery/sglang-mimo-7b-mtp.yaml: MiMo-7B-RL with built-in MTP heads + online fp8 weight quantization, verified end-to-end on a 16 GB RTX 5070 Ti at ~88 tok/s. Uses mem_fraction_static: 0.7 because the MTP draft worker's vocab embedding is loaded unquantised and OOMs the static reservation at sglang's 0.85 default. * gallery/index.yaml: three new entries (gemma-4-e2b-it:sglang-mtp, gemma-4-e4b-it:sglang-mtp, mimo-7b-mtp:sglang). * docs/content/features/text-generation.md: new SGLang section with setup, engine_args reference, MTP demos, version requirements. * .agents/sglang-backend.md (new): agent one-pager covering the flat ServerArgs structure, the typed-vs-engine_args precedence, the speculative-decoding cheatsheet, and the mem_fraction_static gotcha documented above. * AGENTS.md: index entry for the new agent doc. Known limitation: the two Gemma 4 MTP gallery entries ship a recipe that doesn't yet run on stock libraries. The drafter checkpoints (google/gemma-4-{E2B,E4B}-it-assistant) declare model_type: gemma4_assistant / Gemma4AssistantForCausalLM, which neither transformers (<=5.6.0, including the SGLang cookbook's pinned commit 91b1ab1f... and main HEAD) nor sglang's own model registry (<=0.5.11) registers as of 2026-05-06. They will start working when HF or sglang upstream registers the architecture -- no LocalAI changes needed. The MiMo MTP demo and the non-MTP Gemma 4 paths work today on this build (verified on RTX 5070 Ti, 16 GB). Assisted-by: Claude:claude-opus-4-7 [Read] [Edit] [Bash] [WebFetch] [WebSearch] Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-05-07 17:27:29 +02:00
Ettore Di Giacinto	bcef72b9c1	feat: localai assistant chat modality (#9602) ... * fix(tests): inline model_test fixtures after tests/models_fixtures removal The previous reorg removed tests/models_fixtures/ but core/config/model_test.go still read CONFIG_FILE/MODELS_PATH env vars pointing into that directory, so `make test` failed with "open : no such file or directory" on the readConfigFile spec (the suite ran with --fail-fast and bailed before openresponses_test). Inline the YAMLs (config/embeddings/grpc/rwkv/whisper) directly into the test file, materialise them into a per-test tmpdir via BeforeEach, and drop the env-var lookups. The test no longer depends on Makefile plumbing. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: claude-code:claude-opus-4-7 [Edit] [Write] [Bash] * refactor(modeladmin): extract model-admin helpers into a service package Lift the bodies of EditModelEndpoint, PatchConfigEndpoint, ToggleStateModelEndpoint, TogglePinnedModelEndpoint and VRAMEstimateEndpoint into core/services/modeladmin so the same logic can be called by non-HTTP clients (notably the in-process MCP server that backs the LocalAI Assistant chat modality, landing in a follow-up commit). The HTTP handlers shrink to thin shells that parse echo inputs, call the matching helper, map typed errors (ErrNotFound, ErrConflict, ErrPathNotTrusted, ErrBadAction, ...) to the existing HTTP status codes, and render the existing response shapes. No REST-surface behaviour change; the existing localai endpoint tests cover the regression net. Adds focused unit tests for each helper against tmp-dir-backed ModelConfigLoader fixtures (deep-merge patch, rename + conflict, path separator guard, toggle/pin enable/disable, sync callback). Assisted-by: Claude:claude-opus-4-7 [Read] [Edit] [Write] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(assistant): LocalAI Assistant chat modality with in-memory MCP server Adds a chat modality, admin-only, that wires the chat session to an in-memory MCP server exposing LocalAI's own admin/management surface as tools. An admin can install models, manage backends, edit configs and check status by chatting; the LLM calls tools like gallery_search, install_model, import_model_uri, list_installed_models, edit_model_config and surfaces the results. Same Go package powers two modes: pkg/mcp/localaitools/ NewServer(client, opts) builds an MCP server that registers the 19-tool admin catalog. The LocalAIClient interface has two impls: - inproc.Client — calls services directly (no HTTP loopback, no synthetic admin API key). Used in-process by the chat handler. - httpapi.Client — calls the LocalAI REST API. Used by the new `local-ai mcp-server --target=…` subcommand to control a remote LocalAI from a stdio MCP host. Tools and their embedded skill prompts are agnostic to which client backs them. Skill prompts are markdown files under prompts/, embedded via go:embed and assembled into the system prompt at server init. Wiring: - core/http/endpoints/mcp/localai_assistant.go — process-wide holder that spins up the in-memory MCP server once at Application start using paired net.Pipe transports, then reuses LocalToolExecutor (no fork) for every chat request that opts in. - core/http/endpoints/openai/chat.go — small branch ahead of the existing MCP block: when metadata.localai_assistant=true, defense-in-depth admin check + executor swap + system-prompt injection. All downstream tool dispatch is unchanged. - core/http/auth/{permissions,features}.go — adds FeatureLocalAIAssistant; gating happens at the chat handler entry plus admin-only `/api/settings`. - core/cli/{run.go,cli.go,mcp_server.go} — LOCALAI_DISABLE_ASSISTANT flag (runtime-toggleable via Settings, no restart), plus `local-ai mcp-server` stdio subcommand. - core/config/runtime_settings.go — `localai_assistant_enabled` runtime setting; the chat handler reads `DisableLocalAIAssistant` live at request entry. UI: - Home.jsx — prominent self-explanatory CTA card on first run ("Manage LocalAI by chatting"); collapses to a compact "Manage by chat" button in the quick-links row once used, persisted via localStorage. - Chat.jsx — admin-only "Manage" toggle in the chat header, "Manage mode" badge, dedicated empty-state copy, starter chips. - Settings.jsx — "LocalAI Assistant" section with the runtime enable toggle. - useChat.js — `localaiAssistant` flag on the chat schema; injects `metadata.localai_assistant=true` on requests when active. Distributed mode: the in-memory MCP server lives only on the head node; inproc.Client wraps already-distributed-aware services so installs propagate to workers via the existing GalleryService machinery. Documentation: `.agents/localai-assistant-mcp.md` is the contributor contract — when adding an admin REST endpoint, also add a LocalAIClient method, an inproc + httpapi impl, a tool registration, and a skill prompt update; the AGENTS.md index links to it. Out of scope (follow-ups): per-tool RBAC granularity for non-admin read-only access; streaming mcp_tool_progress for long installs; React Vitest rig for the UI changes. Assisted-by: Claude:claude-opus-4-7 [Read] [Edit] [Write] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(assistant): extract tool/capability/MiB/server-name constants The MCP tool surface, capability tag set, server-name default, and the chat-handler metadata key were repeated as bare string literals across seven files. Renaming any one required hand-editing every call site and risked code/test/prompt drift. This pulls them into typed constants: - pkg/mcp/localaitools/tools.go — Tool* constants for the 19 MCP tools, plus DefaultServerName. - pkg/mcp/localaitools/capability.go — typed Capability + constants for the capability tag set the LLM passes to list_installed_models. The type rides through LocalAIClient.ListInstalledModels and replaces the triplet of "embed"/"embedding"/"embeddings" with the single CapabilityEmbeddings. - pkg/mcp/localaitools/inproc/client.go — bytesPerMiB constant for the VRAMEstimate byte→MB conversion. - core/http/endpoints/mcp/tools.go — MetadataKeyLocalAIAssistant for the "localai_assistant" request-metadata key consumed by the chat handler. Tool registrations, the test catalog, the dispatch table, the validation fixtures, and the fake/stub clients all reference the constants. The embedded skill prompts under prompts/ keep their bare strings (go:embed markdown can't import Go constants); the existing TestPromptsContain SafetyAnchors guards the alignment. No behaviour change. All tests pass with -race. Assisted-by: Claude:claude-opus-4-7 [Read] [Edit] [Write] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(modeladmin): typed Action for ToggleState/TogglePinned The toggle/pin verbs were bare strings everywhere — handler signatures, service implementations, MCP tool args, the fake/stub clients, the inproc and httpapi LocalAIClient impls, plus 4 test files. A typo in any caller silently fell through to the runtime "must be 'enable' or 'disable'" check. Introduce core/services/modeladmin.Action (string alias) with ActionEnable, ActionDisable, ActionPin, ActionUnpin and a small Valid helper. The compiler now catches mismatches at every boundary; renames ripple through one source of truth. LocalAIClient.ToggleModelState/Pinned signatures change to take modeladmin.Action. The package is brand-new and unreleased so this is a free public-API tightening. Assisted-by: Claude:claude-opus-4-7 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(assistant): respect ctx cancellation on gallery channel sends InstallModel, DeleteModel, ImportModelURI, InstallBackend and UpgradeBackend all pushed onto galleryop channels with bare sends. If the worker was paused or the buffer full, the chat-handler goroutine blocked forever — the LLM kept polling and the request leaked. Wrap the five sends in a sendModelOp/sendBackendOp helper that selects on ctx.Done() so a cancelled chat completion surfaces context.Canceled back to the LLM instead of hanging. Adds inproc/client_test.go with a pre-cancelled-ctx regression test on InstallModel; the helpers are shared so the same guarantee covers the other four call sites. Assisted-by: Claude:claude-opus-4-7 [Edit] [Write] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(assistant): graceful shutdown for in-memory holder and stdio CLI Two related leaks: - Application.start() built the LocalAIAssistantHolder but never wired Close() into the graceful-termination chain — the in-memory MCP transport pair stayed alive until process exit, and the goroutines behind net.Pipe() didn't drain. Hook into the existing signals.RegisterGracefulTerminationHandler chain (same pattern as core/http/endpoints/mcp/tools.go:770). - core/cli/mcp_server.go ran srv.Run with context.Background(); a Ctrl-C from the host (Claude Desktop, mcphost, npx inspector) or a SIGTERM from process supervision left the stdio loop reading from a closed pipe. Switch to signal.NotifyContext to surface the signal through ctx and let srv.Run drain. Assisted-by: Claude:claude-opus-4-7 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(assistant): typed HTTPError + propagate prompt walk error The httpapi client detected "no such job" by substring-matching on the error string ("404", "could not find") — brittle to status-code formatting changes and to LocalAI fixing /models/jobs/:uuid to return a proper 404. Replace with a typed *HTTPError whose Is() method honours errors.Is(err, ErrHTTPNotFound). The 500-with-"could not find" branch stays as a transitional fallback documented in Is(). Same change covers ListNodes' 404 fallback for the /api/nodes endpoint. Adds httptest tests for both 404 and the legacy 500 path, plus a direct errors.Is exposure test so external callers (the standalone stdio CLI host) can match without re-string-parsing. Also tightens prompts.SystemPrompt: panic when fs.WalkDir on the embedded FS fails. The only realistic cause is a build-time //go:embed misconfiguration; serving an empty system prompt to the LLM is much worse than crashing init. TestSystemPromptIncludesAllEmbeddedFiles catches regressions in CI. Assisted-by: Claude:claude-opus-4-7 [Edit] [Write] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(modeladmin): atomic writes for model config files The five sites that wrote model YAML used os.WriteFile, which opens with O_TRUNC|O_WRONLY|O_CREATE. A crash mid-write left the destination truncated and the model unloadable until manual repair. Pre-existing behaviour inherited from the original endpoint handlers — fix once now that there's a single helper. Adds writeFileAtomic: writes to a sibling temp file, chmods, syncs via Close(), then os.Rename. Same-directory temp keeps the rename atomic on the same filesystem; cleanup runs on every error path so stray temps don't accumulate. No new dependency. Applied to: - ConfigService.PatchConfig - ConfigService.EditYAML (both rename and in-place branches) - mutateYAMLBoolFlag (drives ToggleState + TogglePinned) atomic_test.go covers the happy path plus a read-only-dir failure case that asserts the original file is preserved (skipped on Windows where the chmod trick is POSIX-specific). Assisted-by: Claude:claude-opus-4-7 [Edit] [Write] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore(assistant): prune dead code, mark stub, document conventions Three small cleanups landing together: - Drop the unused errNotImplemented sentinel from inproc/client.go. All five methods that used to return it are wired to modeladmin helpers since the Phase B commit; the package var is dead. - Annotate httpapi.Client.GetModelConfig as a known stub. LocalAI's /models/edit/:name returns rendered HTML, not JSON, so the standalone CLI's get_model_config tool surfaces a clear error to the LLM. A future JSON-only /api/models/config-yaml/:name endpoint is tracked in the agent contract; FIXME points at it. - Extend `.agents/localai-assistant-mcp.md` with a "Code conventions" section that documents the audit-driven rules: tool/Capability/Action constants, errors.Is over substring matching, ctx-aware channel sends, atomic writes, and graceful shutdown. Refresh the file map so it lists tools.go and capability.go and drops the removed tools_bootstrap.go. The tools_models.go diff is a comment-only change explaining why the ModelName empty-string check stays at the tool layer (consistency across LocalAIClient implementations, since the SDK schema validator only enforces presence, not non-empty). Assisted-by: Claude:claude-opus-4-7 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test(assistant): convert test files to ginkgo + gomega The repo convention (per core/http/endpoints/localai/*_test.go, core/gallery/**, etc.) is Ginkgo v2 with Gomega assertions. The tests I introduced for the assistant feature used vanilla testing.T, which made them stand out and stripped the BDD structure the rest of the suite relies on. Convert every test file in the assistant scope to Ginkgo: pkg/mcp/localaitools/ dto_test.go — Describe("DTOs round-trip through JSON") prompts_test.go — Describe("SystemPrompt assembler") server_test.go — Describe("Server tool catalog"), Describe("Tool dispatch"), Describe("Tool error surfacing"), Describe("Argument validation"), Describe("Concurrent tool calls") parity_test.go — Describe("LocalAIClient parity"), hosts the suite's single RunSpecs (the file is package localaitools_test so it can import httpapi without an import cycle; Ginkgo aggregates Describes from both the internal and external test packages into one run). httpapi/client_test.go — Describe("httpapi.Client against the LocalAI admin REST surface"), Describe("ErrHTTPNotFound"), Describe("Bearer token") inproc/client_test.go — Describe("inproc.Client cancellation") core/services/modeladmin/ config_test.go — Describe("ConfigService") with sub-Describes for GetConfig, PatchConfig, EditYAML state_test.go — Describe("ConfigService.ToggleState") pinned_test.go — Describe("ConfigService.TogglePinned") atomic_test.go — Describe("writeFileAtomic") core/http/endpoints/mcp/ localai_assistant_test.go — Describe("LocalAIAssistantHolder") Each package gets a `*_suite_test.go` with the standard `RegisterFailHandler(Fail) + RunSpecs(t, "...")` boilerplate. Helpers that previously took *testing.T (newTestService, writeModelYAML, readMap, sortedStrings, sortGalleries, etc.) drop the *T receiver and use Gomega Expectations directly. tmp dirs come from GinkgoT().TempDir(). No semantic change to test coverage — every original assertion has a direct Gomega counterpart. All suites pass with -race. Assisted-by: Claude:claude-opus-4-7 [Read] [Edit] [Write] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * test+docs(assistant): drift detector for Tool ↔ REST route mapping Honest gap from the audit: the parity_test.go suite only checks four methods, and uses the same httpapi.Client for both sides — it asserts stability of the DTO shapes, not equivalence between in-process and HTTP. If a contributor adds an admin REST endpoint without an MCP tool, or a tool without a matching httpapi route, both surfaces silently diverge. Add a coverage test plus stronger docs: - pkg/mcp/localaitools/coverage_test.go introduces a hand-maintained toolToHTTPRoute map: every Tool* constant must list the REST endpoint the httpapi.Client hits (or "(none)" with a documented reason). Two Ginkgo specs assert the map and the published catalog stay in sync — one fails when a Tool is added without a route entry, the other fails when a route entry references a tool that no longer exists. Verified by removing the ToolDeleteModel entry locally; the test fired with a clear message pointing the contributor at the file. Deliberate non-test: we don't enumerate live admin REST routes from here. Walking the route registry requires booting Application; parsing core/http/routes/localai.go is brittle. The "new admin REST endpoint → MCP tool" direction stays a PR checklist item — see below. - AGENTS.md gets a new Quick Reference bullet that calls out the rule and points at the test by name. - .agents/api-endpoints-and-auth.md tightens the existing "Companion: MCP admin tool surface" subsection from "if useful, consider..." to "MUST be considered, with three concrete outcomes (tool added, deliberately skipped with documented reason, or forgot — which breaks the contract)". Adds a checklist item at the bottom of the file's authoritative checklist. Assisted-by: Claude:claude-opus-4-7 [Read] [Edit] [Write] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(assistant): drop duplicate DTOs, surface canonical types Audit feedback: localaitools/dto.go reinvented several types that already existed in the codebase. Replace the duplicates with the canonical types so the LLM-visible wire format stays aligned with the rest of LocalAI by construction (no parallel structs to keep in sync). Removed (and the canonical type now used by the LocalAIClient interface): localaitools.Gallery → config.Gallery localaitools.GalleryModelHit → gallery.Metadata localaitools.VRAMEstimate → vram.EstimateResult Tightened scope: localaitools.Backend → kept, but reduced to {Name, Installed}. ListKnownBackends now returns []schema.KnownBackend (the canonical type already used by REST /backends/known). Kept with documented rationale: localaitools.JobStatus — galleryop.OpStatus has Error error which marshals to "{}". JobStatus is the JSON-friendly mirror. localaitools.Node — nodes.BackendNode carries gorm internals + token hash; we expose only the LLM-relevant fields. ImportModelURIRequest/Response — schema.ImportModelRequest and GalleryResponse are wire-shaped, mine are LLM-shaped (BackendPreference flat, AmbiguousBackend exposed). Side wins: - Drop bytesPerMiB; vram.EstimateResult already carries human-readable display strings (size_display, vram_display) the LLM uses directly. - Drop the handler-private vramEstimateRequest in core/http/endpoints/localai/vram.go and bind directly into modeladmin.VRAMRequest (now JSON-tagged). Both clients pass through these types now where possible (e.g. ListGalleries in inproc.Client is a one-liner returning AppConfig.Galleries; httpapi.Client.GallerySearch decodes straight into []gallery.Metadata). All tests green with -race. Assisted-by: Claude:claude-opus-4-7 [Read] [Edit] [Bash] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * refactor(assistant): extract REST route paths into named constants httpapi.Client had 18 bare-string path sites scattered across methods. Pull them into pkg/mcp/localaitools/httpapi/routes.go: static paths as package-private constants, dynamic paths as small builders that handle url.PathEscape on segment values. No behaviour change. Drops the now-unused net/url import from client.go since path escaping moved into routes.go alongside the path it applies to. Local-only by design: the server-side registrations in core/http/routes/localai.go remain bare strings. Sharing constants across the pkg/ ↔ core/ boundary would invert the layering today; the existing Tool↔REST drift-detector in coverage_test.go is the safety net for that direction. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-7 [Claude Code] * docs(assistant): align with shipped UI and dropped bootstrap env vars The LocalAI Assistant doc still described the older iteration: - The in-chat toggle was renamed from "Admin" to "Manage" (the badge is now "Manage mode" and the home page exposes a "Manage by chat" CTA). - LOCALAI_ASSISTANT_BOOTSTRAP_MODEL / --localai-assistant-bootstrap-model and the bootstrap_default_model tool were removed — admins pick a model from the existing selector instead, no env-var configuration required. - The shipped tool catalog includes import_model_uri but didn't appear in the doc; bootstrap_default_model appeared but no longer exists. - The Settings → LocalAI Assistant runtime toggle wasn't mentioned as the preferred way to disable without restart. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude:claude-opus-4-7 [Claude Code] --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-04-28 19:29:27 +02:00
Ettore Di Giacinto	f4036fa83f	ci(python-backends): add weekly DEPS_REFRESH cache-buster ... The shared backend/Dockerfile.python ends in: RUN cd /${BACKEND} && PORTABLE_PYTHON=true make which `pip install`s each backend's requirements*.txt. A scan of all 34 Python backends shows every single one ships at least some unpinned deps (torch, transformers, vllm, diffusers, ...). With the registry cache now enabled, that `make` layer's BuildKit hash depends only on Dockerfile instructions + COPYed source — not on what pip resolves at runtime — so a warm cache would freeze upstream versions indefinitely. DEPS_REFRESH is an ARG declared right before that RUN. backend_build.yml computes `date -u +%Y-W%V` (ISO week, e.g. `2026-W17`) and passes it as a build-arg, so the install layer invalidates at most once per week and re-resolves PyPI / nightly indexes. Within a week, builds stay warm. Only Dockerfile.python is affected: Go (go.sum) and Rust (Cargo.lock) already lock their deps, and the C++ backends pull gRPC at a pinned tag and llama.cpp at a pinned commit. Add .agents/ci-caching.md documenting the cache layout (quay.io/go-skynet/ci-cache:cache<tag-suffix>), read/write semantics (master writes, PRs read-only), DEPS_REFRESH semantics, and how to manually evict tags. Index it from AGENTS.md (CLAUDE.md is a symlink). Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: claude-code:claude-opus-4-7-1m	2026-04-27 14:21:11 +00:00
Ettore Di Giacinto	f0c92610a1	feat(importer): expand importer flow to almost all backends (#9466) ... * docs(agents): require importer integration when adding backends Document the importer registry workflow so contributors know that adding a new backend also requires updating the /import-model dropdown source: either a new importer in core/gallery/importers/, extending an existing one for drop-in replacements, or the pref-only slice for backends with no reliable auto-detect signal. Always covered by a table-driven test. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for Batch 0 primitives Introduce failing tests that drive Batch 0 of the importer expansion: - pkg/huggingface-api: assert GetModelDetails populates PipelineTag and LibraryName from /api/models/{repo}, and that a failing metadata endpoint still returns file details (best-effort fetch). - core/gallery/importers/helpers_test.go: new table-driven coverage for HasFile, HasExtension, HasONNX, HasONNXConfigPair, HasGGMLFile. - core/gallery/importers/importers_test.go: assert ErrAmbiguousImport sentinel exists and round-trips through errors.Is. - core/gallery/importers/local_test.go: extend with detection cases for ggml-*.bin (whisper), silero_vad.onnx (silero-vad), and the piper .onnx + .onnx.json pair. - core/http/endpoints/localai/import_model_test.go: assert ImportModelURIEndpoint returns HTTP 400 with a structured {error, detail, hint} body when ErrAmbiguousImport surfaces. All tests fail in the expected places (missing fields, missing helpers, missing sentinel, endpoint still wraps as 500). Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): Batch 0 foundation — helpers, sentinel, local detection Implements the Batch 0 primitives that subsequent importer batches build on: - pkg/huggingface-api: ModelDetails gains PipelineTag and LibraryName. GetModelDetails now layers a best-effort GET /api/models/{repo} fetch on top of ListFiles — a metadata outage leaves the fields empty but still returns full file details. Uses a dedicated response struct because the single-model endpoint uses snake_case keys while the list endpoint historically returned camelCase. - core/gallery/importers/helpers.go: generic HasFile, HasExtension, HasONNX, HasONNXConfigPair, HasGGMLFile helpers working on []hfapi.ModelFile so per-backend importers can detect artefact patterns without duplicating string wrangling. - core/gallery/importers/importers.go: adds the ErrAmbiguousImport sentinel. DiscoverModelConfig now returns it (wrapped with fmt.Errorf("%w: ...")) when no importer matched AND the HF pipeline_tag falls in a whitelist of narrow modalities (ASR, TTS, sentence-similarity, text-classification, object-detection). The whitelist is intentionally narrow — unknown tags keep the previous "no importer matched" behaviour to avoid blocking rare repos. - core/gallery/importers/local.go: three new local-path detections, inserted before the existing merged-transformers branch: * ggml-*.bin → whisper * silero*.onnx → silero-vad * *.onnx + *.onnx.json pair → piper - core/http/endpoints/localai/import_model.go: ImportModelURIEndpoint surfaces ErrAmbiguousImport as HTTP 400 with {error, detail, hint} JSON, preserving existing behaviour for unrelated errors. Green tests: go test ./core/gallery/importers/... ./pkg/huggingface-api/... \ ./core/http/endpoints/localai/... Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(importers): red tests for KnownBackend endpoint and importer metadata Add failing tests that drive Batch UI-Dropdown: - importers_test.go: assert importers expose Name/Modality/AutoDetects and that LlamaCPPImporter advertises drop-in replacements via a new AdditionalBackendsProvider interface. A Registry() accessor is also expected. - backend_test.go (new): assert GET /backends/known returns []schema.KnownBackend, covers every importer, exposes drop-in llama-cpp replacements, includes curated pref-only backends, has no duplicates, and is sorted by Modality+Name. These tests fail at compile time against master; they are intentionally red so the follow-up green commit is reviewable. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery): add /backends/known endpoint for importer-aware backend list Extend the Importer interface with Name/Modality/AutoDetects so the import system can self-describe its registry, and introduce the AdditionalBackendsProvider interface so importers can advertise drop-in replacements (llama-cpp advertises ik-llama-cpp and turboquant). Expose the new GET /backends/known endpoint that merges: - the importer registry (auto-detect supported), - drop-in replacements hosted by importers (preference-only), - a curated knownPrefOnlyBackends slice for backends with no dedicated importer (sglang, tinygrad, trl, mlx-vlm, whisperx, kokoros, Qwen TTS variants, sam3-cpp) — kept at the top of backend.go so contributors adding a new pref-only backend have one obvious place to edit, - backends installed on disk but unknown to the importer (marked AutoDetect=false, empty Modality). The endpoint deliberately does NOT filter by gallery membership or host capability (unlike /backends/available): LocalAI may auto-install a backend that is not yet present, so the import form dropdown must show everything the importer knows about. Response is deduplicated (importer wins over pref-only) and sorted by Modality+Name for deterministic output. Registered in core/http/routes/localai.go next to /backends/available under the same admin middleware. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(ui): source import form backend dropdown from /backends/known Replace the hard-coded BACKENDS constant in ImportModel.jsx with a live fetch of /backends/known on mount. Users now see every backend the importer layer knows about (including preference-only entries) grouped by modality, not a stale subset. Changes: - config.js: add backendsKnown endpoint constant next to backendsAvailable. - api.js: add backendsApi.listKnown() wrapper. - ImportModel.jsx: remove BACKENDS constant, fetch the list via useEffect, and derive grouped options via buildBackendOptions. Preference-only entries render with a " (preference-only)" suffix. Loading state disables the dropdown with a "Loading backends…" placeholder; on fetch failure the form falls back to auto-detect only and surfaces a non-blocking toast. - SearchableSelect.jsx: accept items flagged isHeader=true and render them as non-selectable section dividers. Keyboard navigation skips headers and search queries hide them so filtered output stays relevant. Vitest is not set up in this project (devDependencies ship Playwright only). Per the brief's guard-rail, no frontend test framework is introduced; coverage is provided by the Go handler tests that assert the /backends/known contract consumed by the React form. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for whisper importer Asserts detection on ggerganov/whisper.cpp (via ggml-*.bin filename), the preferences.backend=whisper override path for arbitrary URIs, and the Importer interface metadata (name/modality/autodetect). Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add whisper importer Recognises whisper.cpp GGML models by the "ggml-*.bin" filename convention (direct URL or HF repo member) and by the explicit preferences.backend="whisper" override. Emits backend: whisper with the transcript use-case. Registered before llama-cpp so the narrow filename signal wins before any generic GGUF match is attempted. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for moonshine importer Asserts detection on UsefulSensors/moonshine-tiny via owner + ONNX files, the preferences.backend=moonshine override for arbitrary URIs, and the Importer interface metadata (name/modality/autodetect). Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add moonshine importer Matches UsefulSensors-owned HF repos whose artefacts or metadata identify them as ASR: on-disk .onnx files (the canonical Moonshine packaging) OR pipeline_tag=automatic-speech-recognition (covers transformers/safetensors-only sibling repos). preferences.backend= moonshine overrides detection. Test uses the live moonshine-tiny repo because the canonical UsefulSensors/moonshine repo currently hits a recursive-subfolder bug in pkg/huggingface-api ListFiles. Registered after WhisperImporter but before LlamaCPPImporter and TransformersImporter so the narrower owner+ASR signal wins before the generic tokenizer.json check routes the repo to transformers. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for nemo importer Asserts detection on nvidia/parakeet-tdt-0.6b-v3 via owner + .nemo file, the preferences.backend=nemo override for arbitrary URIs, and the Importer interface metadata (name/modality/autodetect). Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add nemo importer Matches nvidia-owned HF repos that ship a .nemo checkpoint archive, the canonical NeMo ASR packaging. preferences.backend=nemo forces detection. Registered between moonshine and llama-cpp so the narrow owner + extension signal wins before any downstream generic matcher. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for faster-whisper importer Asserts detection on Systran/faster-whisper-large-v3 (owner + model.bin + config.json + ASR pipeline), the preferences.backend= faster-whisper override for arbitrary URIs, and the Importer interface metadata. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add faster-whisper importer Recognises CTranslate2-packaged whisper checkpoints distributed for the faster-whisper runtime: model.bin + config.json + ASR pipeline_tag, narrowed to Systran-owned repos or repo names containing "faster-whisper" to avoid falsely claiming vanilla OpenAI whisper HF repos. preferences.backend=faster-whisper overrides detection. Registered before llama-cpp and transformers so the narrow signal wins before tokenizer.json routes the repo to the generic transformers importer. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for qwen-asr importer Asserts detection on Qwen/Qwen3-ASR-1.7B via owner + ASR substring in the repo name, the preferences.backend=qwen-asr override for arbitrary URIs, and the Importer interface metadata. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add qwen-asr importer Matches Qwen-owned HF repos whose name contains "ASR" (case-insensitive), routing them to the qwen-asr backend rather than the generic transformers/vllm path. The substring check scans the repo portion only so the owner field cannot leak a false match. preferences.backend=qwen-asr forces detection. Registered before llama-cpp and transformers so the narrow owner+name signal wins. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): ASR ambiguity surfaces ErrAmbiguousImport Locks in the behaviour added in Batch 0: an HF repo whose pipeline_tag marks it as automatic-speech-recognition but whose artefacts match no ASR importer (and no generic importer) must fail with ErrAmbiguousImport so callers know to pass preferences.backend rather than silently guess. pyannote/voice-activity-detection is the fixture — its file list is only config.yaml + README, leaving every importer's artefact check negative. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for piper importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add piper importer Detects piper TTS voices by the canonical <voice>.onnx + <voice>.onnx.json pair packaging (via HasONNXConfigPair). Narrow enough to skip generic ONNX repos used by other backends (Moonshine ASR, sentence-transformers). Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for bark importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add bark importer Detects Suno's Bark TTS checkpoints by HF owner "suno" + repo name prefix "bark". Adds HFOwnerRepoFromURI() helper so importers can fall back to URI parsing when pkg/huggingface-api's recursive tree listing errors on repos with nested subdirectories (suno/bark ships a speaker_embeddings/v2 subtree that trips a pre-existing path-doubling bug in the listFilesInPath recursion). Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for fish-speech importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add fish-speech importer Detects Fish Audio TTS releases by HF owner "fishaudio" with a URI-based fallback for repos whose tree recursion trips the pre-existing hfapi path-doubling bug. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for outetts importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add outetts importer Detects OuteAI's OuteTTS releases by HF owner "OuteAI" or a case- insensitive "OuteTTS" substring in the repo name, with a URI-based fallback for recursion-bugged repos. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for voxcpm importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add voxcpm importer Detects OpenBMB's VoxCPM TTS family by repo-name substring (community mirrors re-host the weights under many owners — mlx-community, bluryar, callgg, etc). Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for kokoro importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add kokoro importer Detects hexgrad's Kokoro TTS by the "Kokoro" repo-name substring paired with a PyTorch .pth/.pt checkpoint — the pairing excludes ONNX-only mirrors (handled by the pref-only `kokoros` Rust runtime) and GGUF mirrors (handled by llama-cpp). Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for kitten-tts importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add kitten-tts importer Detects KittenML's kitten-tts releases by owner or "kitten-tts" repo-name substring, with URI-parsing fallback. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for neutts importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add neutts importer Detects Neuphonic's NeuTTS releases by owner "neuphonic" or "neutts" repo-name substring, with URI-parsing fallback. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for chatterbox importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add chatterbox importer Detects Resemble AI's Chatterbox TTS by owner "ResembleAI" or "chatterbox" repo-name substring, with URI-parsing fallback. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for vibevoice importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add vibevoice importer Detects Microsoft's VibeVoice TTS by "vibevoice" repo-name substring (case-insensitive) so community mirrors still route here. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for coqui importer Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add coqui importer Detects Coqui AI's TTS releases (XTTS-v2, YourTTS, …) by the authoritative `coqui` HF owner, with URI-parsing fallback. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): TTS ambiguity surfaces ErrAmbiguousImport Adds a Ginkgo spec that imports nari-labs/Dia-1.6B — a real HF repo carrying pipeline_tag="text-to-speech" whose artefacts (*.pth, one safetensors shard, preprocessor_config.json, config.json) match none of the Batch-2 TTS importers nor the generic text/image importers — and asserts DiscoverModelConfig wraps ErrAmbiguousImport via errors.Is. Also pivots the endpoint-level ambiguity fixture from hexgrad/Kokoro-82M to nari-labs/Dia-1.6B. Batch 2 added a dedicated kokoro importer that now claims the original fixture; Dia remains genuinely unclaimed and so exercises the same ambiguity code path at the HTTP layer. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for stablediffusion-ggml importer Covers HF repo detection (city96/FLUX.1-dev-gguf), raw .gguf URL matching on filename arch tokens, preference override, and Importer interface metadata. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add stablediffusion-ggml importer Detects GGUF-packed Stable Diffusion and FLUX checkpoints (leejet owner, city96 FLUX mirrors, second-state SD dumps, raw .gguf URLs with arch tokens) and routes them to the stablediffusion-ggml backend. Registered BEFORE LlamaCPPImporter so .gguf image checkpoints are not stolen by llama-cpp's generic .gguf match. Reuses HFOwnerRepoFromURI for the hfapi-recursion-bug fallback. preferences.backend overrides detection. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for ace-step importer Covers HF repo-name detection (ACE-Step/ACE-Step-v1-3.5B), preference override, and Importer interface metadata. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add ace-step importer Routes ACE-Step music generation checkpoints (ACE-Step/ACE-Step-v1-3.5B, ACE-Step/Ace-Step1.5, community mirrors) to the ace-step backend. Matching is case-insensitive on the "ace-step" repo-name substring and owner, with an HFOwnerRepoFromURI fallback for the hfapi recursion bug. KnownUsecaseStrings mirrors the gallery's ace-step-turbo entry (sound_generation, tts). preferences.backend overrides. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): surface ErrAmbiguousImport on text-to-image misses Adds text-to-image to ambiguousModalities whitelist and covers the h94/IP-Adapter-FaceID case — pipeline_tag=text-to-image but ships only .bin/.safetensors so diffusers, stablediffusion-ggml, llama-cpp, transformers, vllm, mlx, and ace-step all miss. DiscoverModelConfig now surfaces ErrAmbiguousImport for that shape instead of the opaque "no importer matched" error. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for vllm-omni importer Introduces the test surface for the forthcoming VLLMOmniImporter: detection via preferences.backend, Qwen owner + Omni repo token, URI-only fallback, negative cases (plain Qwen, random OmniX repo), and Import() emitting backend: vllm-omni with chat + multimodal usecases. Includes a registration-order assertion via DiscoverModelConfig to pin the requirement that vllm-omni wins over vllm for Qwen Omni repos (tokenizer files are usually present too). Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add vllm-omni importer Adds VLLMOmniImporter for Qwen Omni-style multimodal checkpoints (Qwen3-Omni, Qwen2.5-Omni, …). Detection is narrow: HF owner "Qwen" combined with "omni" in the repo name, or a repo name matching the -Omni-/Omni- naming pattern. preferences.backend="vllm-omni" always wins; HFOwnerRepoFromURI provides a URI-only fallback for the hfapi recursion-bug edge case. Emitted YAML sets backend: vllm-omni and known_usecases: [chat, multimodal], matching the gallery/index.yaml vllm-omni entries. The importer is registered ahead of VLLMImporter so Qwen Omni repos — which also carry tokenizer files — route to vllm-omni rather than the plain vllm backend. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for llama-cpp drop-in preferences Pins the expected drop-in replacement behaviour: preferences.backend of ik-llama-cpp or turboquant must swap the emitted YAML backend field while keeping the llama-cpp file layout identical. Also covers the unknown-backend case (must stay llama-cpp) and re-asserts AdditionalBackends() returns the two curated entries with non-empty descriptions. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): llama-cpp honours ik-llama-cpp and turboquant drop-in preferences preferences.backend set to ik-llama-cpp or turboquant now swaps the emitted YAML backend field while leaving the file layout, model path, mmproj handling and everything else in the llama-cpp Import pipeline untouched. Unknown values are ignored and fall back to backend: llama-cpp so arbitrary input can't leak into the config. Aligns the AdditionalBackends() descriptions with the user-facing naming conventions surfaced via /backends/known. No changes to the pref-only curated list in endpoints/localai/backend.go: the two drop-in names have always lived on the importer side via AdditionalBackends. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for silero-vad importer Add the SileroVADImporter test fixtures covering metadata, preference overrides, snakers4 + onnx detection, silero_vad.onnx canonical filename, URI fallback, and live HF discovery. Implementation follows in the next commit. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add silero-vad importer Recognise the Silero VAD ONNX packaging: the canonical silero_vad.onnx filename or any ONNX file under the snakers4 owner. Emits a backend: silero-vad config with the vad known_usecase, and attaches the canonical file entry when present so the weights download on import. Registered before the generic importers so the unique-filename signal takes precedence over any downstream tokenizer-based matcher. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for rerankers importer Cover the RerankersImporter contract: interface metadata, preference override, cross-encoder owner detection, case-insensitive 'reranker' substring match (BAAI/bge-reranker, Alibaba-NLP/gte-reranker), URI fallback, and the full-discovery ordering check that a BAAI reranker repo must route to the rerankers importer rather than transformers. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add rerankers importer Recognise reranker repositories — cross-encoder owner or any repo whose name contains 'reranker' (case-insensitive). Emits backend: rerankers with reranking: true and the rerank known_usecase. Registered ahead of sentencetransformers and transformers so reranker repos that happen to ship tokenizer.json or modules.json still route here. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for sentencetransformers importer Cover the SentenceTransformersImporter contract: interface metadata, preference override, modules.json marker file, sentence_bert_config.json marker file, sentence-transformers owner, URI fallback, and the full-discovery ordering check that ensures a sentence-transformers HF URI routes here rather than transformers. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add sentencetransformers importer Recognise sentence-transformers embedding repos by modules.json, sentence_bert_config.json, or the sentence-transformers owner. Emits backend: sentencetransformers with embeddings: true and the embeddings known_usecase. Registered ahead of transformers so ST repos that carry tokenizer.json still route here. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): add failing tests for rfdetr importer Cover the RFDetrImporter contract: interface metadata, preference override, case-insensitive rf-detr and rfdetr substring matches, URI fallback, and negative cases. Implementation follows in the next commit. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(gallery/importers): add rfdetr importer Recognise RF-DETR object-detection repositories by a case-insensitive 'rf-detr' / 'rfdetr' substring in the repo name. Emits backend: rfdetr with the detection known_usecase. Registered ahead of transformers so RF-DETR repos with tokenizer artefacts still route here. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(gallery/importers): surface ErrAmbiguousImport on sentence-similarity misses Add an ambiguity fixture covering the embeddings/rerankers modality. Qdrant/bm25 carries pipeline_tag=sentence-similarity but ships only config.json + stopword .txt files — none of the Batch 5 importers (silero-vad, rerankers, sentencetransformers, rfdetr) or the generic vllm/transformers/llama-cpp/mlx/diffusers importers match. Because the modality is in the ambiguous whitelist, DiscoverModelConfig must surface ErrAmbiguousImport. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(localai/backend): red tests for KnownBackend.Installed flag Extend the /backends/known suite with three failing cases that pin down the forthcoming Installed field: JSON field presence on every entry, flipping to true when an importer-registered backend is also present on disk (and staying false for non-installed pref-only entries), and surfacing system-only backends with empty modality and AutoDetect=false. A small writeFakeSystemBackend helper plants a run.sh under the backends dir so gallery.ListSystemBackends recognises the fixture. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(schema,localai/backend): add Installed flag to KnownBackend Add an Installed bool to schema.KnownBackend and populate it from the /backends/known handler so the React import form can warn users that picking a not-yet-installed backend will trigger an automatic download on submit. Computation: after merging the importer registry, additional backends provider entries and the curated pref-only slice, the handler walks gallery.ListSystemBackends(systemState) and either flips the existing map entry's Installed flag to true (preserving modality / autodetect / description metadata) or inserts a bare {Installed:true} entry for system-only backends the importer layer doesn't know about. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(localai/import_model): structured ambiguous-import response Add red tests covering the extended ambiguity shape the React import form needs: - ImportModelURIEndpoint must return an HTTP 400 body that exposes the detected `modality` (normalised to the importer modality key, e.g. "tts" for pipeline_tag=text-to-speech) and a list of `candidates` (backend names filtered by modality, excluding text-LLM backends). - The importers package must surface a typed AmbiguousImportError so HTTP consumers can read Modality + Candidates without parsing the error string. errors.Is against the existing sentinel keeps working. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(localai/import_model): structured ambiguity response with modality + candidates DiscoverModelConfig now returns a typed AmbiguousImportError that carries the importer modality key, candidate backend names, the original URI, and the raw HF pipeline_tag. Its Is() preserves errors.Is(err, ErrAmbiguousImport) for legacy callers. The importer modality is pre-mapped from the HF pipeline_tag (automatic-speech-recognition → asr, text-to-speech → tts, etc) via PipelineTagToModality — surfaced as an exported helper so downstream consumers can avoid duplicating the table. CandidatesForModality filters the default importer registry plus AdditionalBackendsProvider drop-ins by modality, sorts deterministically, and is the single source of truth used by ImportModelURIEndpoint. ImportModelURIEndpoint now returns HTTP 400 with { error, detail, modality, candidates, hint } when ambiguity fires, letting the React form render a modality-scoped picker inline instead of a generic toast. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(ui/import): manual pick badge + tooltip Red Playwright coverage for the preference-only → manual pick rename: - The Backend dropdown renders a "manual pick" badge on every option whose KnownBackend.auto_detect is false. - The badge carries a title attribute with hover-tooltip copy that explains auto-detect won't route to this backend. - Auto-detectable backends must NOT carry the badge. - The legacy " (preference-only)" suffix is gone from every label. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * ui(import): replace preference-only suffix with manual pick badge SearchableSelect option rows now support an optional badge field — a muted pill rendered to the right of the label with an optional title attribute for native hover tooltips. Plain text so screen readers read it alongside the option name. buildBackendOptions in ImportModel stops appending " (preference-only)" to the label and instead sets badge="manual pick" plus a descriptive tooltip on every option whose auto_detect is false. The Backend help text explains what "manual pick" means so users aren't left wondering about the badge. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(ui/import): inline ambiguity picker Red Playwright coverage for Batch A2 — when the server returns a 400 ambiguity body, the form must render an inline alert instead of a toast, expose one clickable chip per candidate backend, and support both auto-resubmit on pick and silent dismiss. - Mocks /api/models/import-uri with the structured ambiguity body (error, detail, modality, candidates, hint). - On first click of Import, the alert is visible, carries modality-specific copy, and shows a chip per candidate. - Clicking a chip clears the alert, sets the Backend dropdown, and triggers a second POST to /api/models/import-uri. - Dismissing the alert leaves the Backend dropdown on Auto-detect — no implicit backend assignment. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(ui/import): inline ambiguity alert with candidate chips Adds AmbiguityAlert — a soft, info-coloured card rendered above the URI input when the server returns a structured 400 with { modality, candidates }. Message is modality-aware (tts/asr/embeddings/image/ reranker/detection get purpose-written copy, everything else falls back to a generic template). Each candidate is a clickable chip that shows a download icon when /backends/known marks the backend as not yet installed, so users aren't surprised by an implicit install. ImportModel wires the alert to handleSimpleImport's error path: - api.handleResponse now attaches { status, body } to the thrown Error so pages can pattern-match on structured responses instead of string error messages. - handleSimpleImport detects `status === 400 && body.error === 'ambiguous import'` and flips into the inline-picker mode instead of toasting. - Clicking a chip sets prefs.backend and auto-resubmits (passing the picked backend as an override so setPrefs's asynchrony doesn't leak a stale value). - Dismissing clears the alert; changing the URI or the backend also clears it so a stale alert never sticks around. Test fixtures mock GET /backends/known + POST /models/import-uri so the Playwright specs don't depend on real network reachability. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(ui/import): auto-install warning Red Playwright coverage for Batch A3 — when the user picks a backend whose KnownBackend.installed is false, the form must render a muted inline note under the Backend dropdown warning that submitting will download the backend first. Picking an installed backend or leaving Auto-detect selected must keep the note hidden. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(ui/import): auto-install warning under backend dropdown When the user picks a backend whose KnownBackend.installed is false, render a muted inline note under the Backend dropdown's help text warning that submitting will download the backend first. The note lives inside the same form-group so it lines up with the existing hint text; it's hidden when Auto-detect is selected (the selected backend is unknowable at that point) or when the chosen backend is already on disk. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * ui(import): drop redundant section header, adjust icons, rename HF shortcut - Remove the "Import from URI" card-level <h2> — the page title already says "Import New Model" one row up, so the secondary header was duplicating information. - Swap the fa-star on "Common Preferences" for fa-sliders (stars imply favourites/ratings; this is just a preferences block) and move the Custom Preferences fa-sliders-h to fa-plus-circle so the two blocks read as distinct rather than as two sliders. - Rename the HF shortcut from "Search GGUF on HF" → "Browse models on HF" and drop the `search=gguf` filter on the linked URL. The import form now supports ~40 backends; hard-coding GGUF in the copy no longer matches the form's actual reach. - Pure polish — no behaviour change, covered by the existing Batch A Playwright suite. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(ui/import): batch B — simple/power switch, options, tabs, dialog Adds a failing Playwright suite covering the full Batch B surface ahead of implementation: - B1: SimplePowerSwitch segmented control renders, toggles, persists to localStorage across reloads. - B2: Simple-mode Options disclosure is collapsed by default; expanding exposes only Backend, Model Name, Description (no quantizations, mmproj, model type, or custom prefs). - B3: Power mode has Preferences and YAML tabs with a persistent selection across reloads; URI/name/description typed in Simple carry over to Power; YAML tab swaps the primary action to Create. - B4: Switching Power -> Simple with a custom preference set triggers the 3-button confirmation dialog (Keep / Discard / Cancel) with the documented semantics. Tests fail against master — implementation lands in the following commits. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(ui/import): add SimplePowerSwitch segmented control Replaces the previous "Advanced Mode / Simple Mode" toggle button in the page header with a two-segment control that flips between Simple and Power. The control reuses the existing .segmented CSS shared with the Sound page for visual consistency. Mode state is persisted to localStorage under `import-form-mode` so reloads land on the same view (default: simple). The boolean alias `isAdvancedMode` is retained internally to minimise diff — subsequent commits reshape the Simple and Power surfaces independently. Closes B1 from the Batch B Playwright suite. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(ui/import): simple mode collapsible options, power tabs, switch dialog Completes the Batch B surface in a single structural pass so Simple and Power mode can evolve independently: Simple mode - URI input + Ambiguity alert + Import button, plus a collapsible "Options" disclosure that exposes ONLY Backend, Model Name, Description. Quantizations / MMProj / Model Type / Diffusers fields / Custom Preferences are no longer rendered in Simple mode. Power mode - In-page segmented "Preferences · YAML" tab strip. Active tab persists to localStorage under `import-form-power-tab`. - Preferences tab = the full existing preferences + custom prefs panel (no progressive disclosure yet — that's Batch D). - YAML tab = the existing CodeEditor. Primary button reads "Create" here, "Import Model" everywhere else. Switch dialog - Power -> Simple with non-default prefs (advanced pref keys set, any custom-pref key non-empty, or YAML edited away from the template) opens a 3-button dialog: Keep & switch / Discard & switch / Cancel. - Keep preserves all state. Discard resets prefs + customPrefs + YAML to defaults. Cancel leaves the user in Power mode. Page subtitle reflects the current surface (Simple, Power/Preferences, Power/YAML). Estimate banner renders everywhere except Power/YAML. Closes B2/B3/B4 from the Batch B Playwright suite. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(ui/import): expand Options disclosure in Batch A tests Batch B hid the Backend dropdown behind a collapsible Options disclosure in Simple mode. The Batch A tests that exercise the dropdown directly (manual-pick badge, ambiguity chip sets the selected backend, auto- install warning) now click the disclosure toggle before asserting on dropdown contents. Test intent is unchanged. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * ui(import): strip decorative icons from field labels The preference panel had 12 Font Awesome icons decorating field labels (Backend, Model Name, Description, Quantizations, MMProj Quantizations, Model Type, Pipeline Type, Scheduler Type, Enable Parameters, Embeddings, CUDA, plus fa-link on Model URI). Every label screamed equally, flattening the visual hierarchy. Remove them. Keep icons where they carry meaning: page-level section headers, URI format guide entries, primary buttons, the Simple-mode Options disclosure, the ambiguity alert's fa-lightbulb, the auto-install note's fa-download, and the Estimated-requirements banner's fa-memory / fa-microchip / fa-download. No new behaviour, no layout / spacing changes beyond removing the orphaned icon margin. Playwright suite green. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(ui/import): progressive disclosure of preference fields Cover the Batch D visibility matrix for Power > Preferences: Quantizations, MMProj Quantizations, and Model Type each render only for the backends that can consume them, stay visible when the backend is unset, and preserve any value the user already typed when toggled off and back on. Also pin the shrunk Description textarea at rows=2. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(ui/import): progressive disclosure + shorter description textarea Gate Quantizations, MMProj Quantizations, and Model Type in the Power > Preferences tab so each field only renders for the backends that can actually consume it. Backend unset keeps everything visible. Hidden fields' state is preserved (the JSX wrapper is guarded, not the underlying prefs state) so users flipping backends back and forth don't lose input. Also shrink the Description textarea from rows=3 to rows=2 — it's shared between Simple Options and Power Preferences so the change applies to both. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(ui/import): enter-to-submit in Simple mode Red test for Batch F3 — pressing Enter in the URI input must POST /models/import-uri, and Enter in the Description textarea must insert a newline without submitting the form. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(ui/import): enter-to-submit in Simple mode Wrap the Simple-mode URI input + ambiguity alert + Options disclosure in a <form> whose onSubmit calls handleSimpleImport. Pressing Enter in the URI input (or any Simple-mode text input) now submits the import without having to move the mouse to the header button. The Description textarea keeps its native behaviour — Enter inserts a newline. A hidden submit button is included because the visible Import button lives outside the form in the page header; some browsers only fire implicit Enter-submit when the form contains a submit-capable element. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * ui(import,SearchableSelect,components): aria-hidden on decorative icons Every Font Awesome icon in the import form is decorative — its meaning is already conveyed by adjacent visible text. Adding aria-hidden="true" prevents screen readers from announcing the unicode glyph point as content. Covers ImportModel.jsx (all remaining <i> glyphs) and SearchableSelect.jsx (the trigger chevron). AmbiguityAlert and SimplePowerSwitch already set aria-hidden on their icons when the components landed in Batches A and B — no change needed there. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * ui(SearchableSelect): responsive dropdown maxHeight + hover focus guard F2 — replace fixed pixel heights with min(pixel, vh) so the dropdown and its inner scroll region don't overflow short viewports. Outer container: 260px -> min(260px, 60vh); inner listbox: 200px -> min(200px, 50vh). Tall viewports still get the original pixel caps. F5 — short-circuit onMouseEnter when the hovered row is already the focused row. Avoids queueing a setFocusIndex call (and a render) for every mousemove inside the same item — the state would be identical. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * ui(import): aria-label on custom preference rows The Key / Value inputs and trash button in each Custom Preferences row previously relied on placeholder text alone. Placeholders are not accessible names — they vanish on input and screen readers do not announce them consistently. Add row-indexed aria-labels so assistive tech can distinguish "Preference key for row 1" from "row 2", and give the trash button an explicit "Remove this preference" label. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * test(ui/import): modality chip row Red tests for Batch E — a horizontal modality chip row that filters the Backend dropdown by modality. Covers visibility in Simple-mode Options and Power/Preferences (and absence in Power/YAML), filter behaviour, mismatched-backend clearing with toast, ambiguity-alert auto-selection, and radiogroup keyboard navigation. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * feat(ui/import): add ModalityChips component + filter integration Horizontal chip row (Any, Text, Speech, TTS, Image, Embeddings, Rerankers, Detection, VAD) filters the Backend dropdown options to the selected modality. Default is Any — no filter, current behaviour. - New ModalityChips component (radiogroup pattern, roving tabindex, arrow-key navigation, Home/End). - buildBackendOptions now accepts an optional modalityFilter so grouped output is narrowed before rendering. - Chips render inside Simple-mode Options disclosure and Power > Preferences tab. Power > YAML stays unaffected. - Switching the filter drops a mismatched backend selection and surfaces a toast so the auto-clear is visible. - Ambiguity alerts auto-activate the matching chip so users see only relevant backends even if they dismiss the alert. Tightens the Batch E tests' option-matching to the label <span> so the "↵" keybind hint on the focused row doesn't break accessible-name lookups. Assisted-by: Claude:claude-opus-4-7[1m] [Agent] * fix(ui/import): rename Power to Advanced + stop URI-formats toggle from submitting form The "Supported URI Formats" disclosure button inside the Simple-mode form lacked an explicit type attribute, so it defaulted to type="submit". Every click triggered the form's onSubmit and surfaced the empty-URI validation toast ("Please enter a model URI"). Marking it type="button" lets it behave as a pure toggle. While here, rename the user-visible "Power" label to "Advanced" in the mode switch (button text + tooltip) and the Power-mode tab's aria-label, matching the term users actually expect. The internal mode key stays 'power' so tests, localStorage, and data-testid selectors are untouched. Assisted-by: Claude:claude-opus-4-7 * fix(system): fall back to cpu when meta backend lacks default capability Meta backends like vllm and sglang enumerate concrete variants for nvidia/amd/intel/cpu but omit a default: catch-all entry. On a no-GPU host the reported capability is "default", so the previous Capability() returned "default" unconditionally on a miss — IsCompatibleWith then saw no "default" key and filtered the meta out of AvailableBackends. The import flow's auto-install step then failed with "no backend found with name <meta>", contradicting the UI's promise that the backend would be downloaded on demand. Try the explicit "default" key first, then fall back to "cpu" before giving up. vllm now resolves to cpu-vllm on CPU-only Linux without touching the gallery YAML. Assisted-by: Claude:claude-opus-4-7	2026-04-22 22:42:37 +02:00
Ettore Di Giacinto	20baec77ab	feat(face-recognition): add insightface/onnx backend for 1:1 verify, 1:N identify, embedding, detection, analysis (#9480) ... * feat(face-recognition): add insightface backend for 1:1 verify, 1:N identify, embedding, detection, analysis Adds face recognition as a new first-class capability in LocalAI via the `insightface` Python backend, with a pluggable two-engine design so non-commercial (insightface model packs) and commercial-safe (OpenCV Zoo YuNet + SFace) models share the same gRPC/HTTP surface. New gRPC RPCs (backend/backend.proto): * FaceVerify(FaceVerifyRequest) returns FaceVerifyResponse * FaceAnalyze(FaceAnalyzeRequest) returns FaceAnalyzeResponse Existing Embedding and Detect RPCs are reused (face image in PredictOptions.Images / DetectOptions.src) for face embedding and face detection respectively. New HTTP endpoints under /v1/face/: * verify — 1:1 image pair same-person decision * analyze — per-face age + gender (emotion/race reserved) * register — 1:N enrollment; stores embedding in vector store * identify — 1:N recognition; detect → embed → StoresFind * forget — remove a registered face by opaque ID Service layer (core/services/facerecognition/) introduces a `Registry` interface with one in-memory `storeRegistry` impl backed by LocalAI's existing local-store gRPC vector backend. HTTP handlers depend on the interface, not on StoresSet/StoresFind directly, so a persistent PostgreSQL/pgvector implementation can be slotted in via a single constructor change in core/application (TODO marker in the package doc). New usecase flag FLAG_FACE_RECOGNITION; insightface is also wired into FLAG_DETECTION so /v1/detection works for face bounding boxes. Gallery (backend/index.yaml) ships three entries: * insightface-buffalo-l — SCRFD-10GF + ArcFace R50 + genderage (~326MB pre-baked; non-commercial research use only) * insightface-opencv — YuNet + SFace (~40MB pre-baked; Apache 2.0) * insightface-buffalo-s — SCRFD-500MF + MBF (runtime download; non-commercial) Python backend (backend/python/insightface/): * engines.py — FaceEngine protocol with InsightFaceEngine and OnnxDirectEngine; resolves model paths relative to the backend directory so the same gallery config works in docker-scratch and in the e2e-backends rootfs-extraction harness. * backend.py — gRPC servicer implementing Health, LoadModel, Status, Embedding, Detect, FaceVerify, FaceAnalyze. * install.sh — pre-bakes buffalo_l + OpenCV YuNet/SFace inside the backend directory so first-run is offline-clean (the final scratch image only preserves files under /<backend>/). * test.py — parametrized unit tests over both engines. Tests: * Registry unit tests (go test -race ./core/services/facerecognition/...) — in-memory fake grpc.Backend, table-driven, covers register/ identify/forget/error paths + concurrent access. * tests/e2e-backends/backend_test.go extended with face caps (face_detect, face_embed, face_verify, face_analyze); relative ordering + configurable verifyCeiling per engine. * Makefile targets: test-extra-backend-insightface-buffalo-l, -opencv, and the -all aggregate. * CI: .github/workflows/test-extra.yml gains tests-insightface-grpc, auto-triggered by changes under backend/python/insightface/. Docs: * docs/content/features/face-recognition.md — feature page with license table, quickstart (defaults to the commercial-safe model), models matrix, API reference, 1:N workflow, storage caveats. * Cross-refs in object-detection.md, stores.md, embeddings.md, and whats-new.md. * Contributor README at backend/python/insightface/README.md. Verified end-to-end: * buffalo_l: 6/6 specs (health, load, face_detect, face_embed, face_verify, face_analyze). * opencv: 5/5 specs (same minus face_analyze — SFace has no demographic head; correctly skipped via BACKEND_TEST_CAPS). Assisted-by: Claude:claude-opus-4-7 * fix(face-recognition): move engine selection to model gallery, collapse backend entries The previous commit put engine/model_pack options on backend gallery entries (`backend/index.yaml`). That was wrong — `GalleryBackend` (core/gallery/backend_types.go:32) has no `options` field, so the YAML decoder silently dropped those keys and all three "different insightface-*" backend entries resolved to the same container image with no distinguishing configuration. Correct split: * `backend/index.yaml` now has ONE `insightface` backend entry shipping the CPU + CUDA 12 container images. The Python backend bundles both the non-commercial insightface model packs (buffalo_l / buffalo_s) and the commercial-safe OpenCV Zoo weights (YuNet + SFace); the active engine is selected at LoadModel time via `options: ["engine:..."]`. * `gallery/index.yaml` gains three model entries — `insightface-buffalo-l`, `insightface-opencv`, `insightface-buffalo-s` — each setting the appropriate `overrides.backend` + `overrides.options` so installing one actually gives the user the intended engine. This matches how `rfdetr-base` lives in the model gallery against the `rfdetr` backend. The earlier e2e tests passed despite this bug because the Makefile targets pass `BACKEND_TEST_OPTIONS` directly to LoadModel via gRPC, bypassing any gallery resolution entirely. No code changes needed. Assisted-by: Claude:claude-opus-4-7 * feat(face-recognition): cover all supported models in the gallery + drop weight baking Follows up on the model-gallery split: adds entries for every model configuration either engine actually supports, and switches weight delivery from image-baked to LocalAI's standard gallery mechanism. Gallery now has seven `insightface-*` model entries (gallery/index.yaml): insightface (family) — non-commercial research use • buffalo-l (326MB) — SCRFD-10GF + ResNet50 + genderage, default • buffalo-m (313MB) — SCRFD-2.5GF + ResNet50 + genderage • buffalo-s (159MB) — SCRFD-500MF + MBF + genderage • buffalo-sc (16MB) — SCRFD-500MF + MBF, recognition only (no landmarks, no demographics — analyze returns empty attributes) • antelopev2 (407MB) — SCRFD-10GF + ResNet100@Glint360K + genderage OpenCV Zoo family — Apache 2.0 commercial-safe • opencv — YuNet + SFace fp32 (~40MB) • opencv-int8 — YuNet + SFace int8 (~12MB, ~3x smaller, faster on CPU) Model weights are no longer baked into the backend image. The image now ships only the Python runtime + libraries (~275MB content size, ~1.18GB disk vs ~1.21GB when weights were baked). Weights flow through LocalAI's gallery mechanism: * OpenCV variants list `files:` with ONNX URIs + SHA-256, so `local-ai models install insightface-opencv` pulls them into the models directory exactly like any other gallery-managed model. * insightface packs (upstream distributes .zip archives only, not individual ONNX files) auto-download on first LoadModel via FaceAnalysis' built-in machinery, rooted at the LocalAI models directory so they live alongside everything else — same pattern `rfdetr` uses with `inference.get_model()`. Backend changes (backend/python/insightface/): * backend.py — LoadModel propagates `ModelOptions.ModelPath` (the LocalAI models directory) to engines via a `_model_dir` hint. This replaces the earlier ModelFile-dirname approach; ModelPath is the canonical "models directory" variable set by the Go loader (pkg/model/initializers.go:144) and is always populated. * engines.py::_resolve_model_path — picks up `model_dir` and searches it (plus basename-in-model-dir) before falling back to the dev script-dir. This is how OnnxDirectEngine finds gallery-downloaded YuNet/SFace files by filename only. * engines.py::_flatten_insightface_pack — new helper that works around an upstream packaging inconsistency: buffalo_l/s/sc zips expand flat, but buffalo_m and antelopev2 zips wrap their ONNX files in a redundant `<name>/` directory. insightface's own loader looks one level too shallow and fails. We call `ensure_available()` explicitly, flatten if nested, then hand to FaceAnalysis. * engines.py::InsightFaceEngine.prepare — root-resolution order now includes the `_model_dir` hint so packs download into the LocalAI models directory by default. * install.sh — no longer pre-downloads any weights. Everything is gallery-managed now. * smoke.py (new) — parametrized smoke test that iterates over every gallery configuration, simulating the LocalAI install flow (creates a models dir, fetches OpenCV files with checksum verification, lets insightface auto-download its packs), then runs detect + embed + verify (+ analyze where supported) through the in-process BackendServicer. * test.py — OnnxDirectEngineTest no longer hardcodes `/models/opencv/` paths; downloads ONNX files to a temp dir at setUpClass time and passes ModelPath accordingly. Registry change (core/services/facerecognition/store_registry.go): * `dim=0` in NewStoreRegistry now means "accept whatever dimension arrives" — needed because the backend supports 512-d ArcFace/MBF and 128-d SFace via the same Registry. A non-zero dim still fails fast with ErrDimensionMismatch. * core/application plumbs `faceEmbeddingDim = 0`, explaining the rationale in the comment. Backend gallery description updated to reflect that the image carries no weights — it's just Python + engines. Smoke-tested all 7 configurations against the rebuilt image (with the flatten fix applied), exit 0: PASS: insightface-buffalo-l faces=6 dim=512 same-dist=0.000 PASS: insightface-buffalo-sc faces=6 dim=512 same-dist=0.000 PASS: insightface-buffalo-s faces=6 dim=512 same-dist=0.000 PASS: insightface-buffalo-m faces=6 dim=512 same-dist=0.000 PASS: insightface-antelopev2 faces=6 dim=512 same-dist=0.000 PASS: insightface-opencv faces=6 dim=128 same-dist=0.000 PASS: insightface-opencv-int8 faces=6 dim=128 same-dist=0.000 7/7 passed Assisted-by: Claude:claude-opus-4-7 * fix(face-recognition): pre-fetch OpenCV ONNX for e2e target; drop stale pre-baked claim CI regression from the previous commit: I moved OpenCV Zoo weight delivery to LocalAI's gallery `files:` mechanism, but the test-extra-backend-insightface-opencv target was still passing relative paths `detector_onnx:models/opencv/yunet.onnx` in BACKEND_TEST_OPTIONS. The e2e suite drives LoadModel directly over gRPC without going through the gallery, so those relative paths resolved to nothing and OpenCV's ONNXImporter failed: LoadModel failed: Failed to load face engine: OpenCV(4.13.0) ... Can't read ONNX file: models/opencv/yunet.onnx Fix: add an `insightface-opencv-models` prerequisite target that fetches the two ONNX files (YuNet + SFace) to a deterministic host cache at /tmp/localai-insightface-opencv-cache/, verifies SHA-256, and skips the download on re-runs. The opencv test target depends on it and passes absolute paths in BACKEND_TEST_OPTIONS, so the backend finds the files via its normal absolute-path resolution branch. Also refresh the buffalo_l comment: it no longer says "pre-baked" (nothing is — the pack auto-downloads from upstream's GitHub release on first LoadModel, same as in CI). Locally verified: `make test-extra-backend-insightface-opencv` passes 5/5 specs (health, load, face_detect, face_embed, face_verify). Assisted-by: Claude:claude-opus-4-7 * feat(face-recognition): add POST /v1/face/embed + correct /v1/embeddings docs The docs promised that /v1/embeddings returns face vectors when you send an image data-URI. That was never true: /v1/embeddings is OpenAI-compatible and text-only by contract — its handler goes through `core/backend/embeddings.go::ModelEmbedding`, which sets `predictOptions.Embeddings = s` (a string of TEXT to embed) and never populates `predictOptions.Images[]`. The Python backend's Embedding gRPC method does handle Images[] (that's how /v1/face/register reaches it internally via `backend.FaceEmbed`), but the HTTP embeddings endpoint wasn't wired to populate it. Rather than overload /v1/embeddings with image-vs-text detection — messy, and the endpoint is OpenAI-compatible by design — add a dedicated /v1/face/embed endpoint that wraps `backend.FaceEmbed` (already used internally by /v1/face/register and /v1/face/identify). Matches LocalAI's convention of a dedicated path per non-standard flow (/v1/rerank, /v1/detection, /v1/face/verify etc.). Response: { "embedding": [<dim> floats, L2-normed], "dim": int, // 512 for ArcFace R50 / MBF, 128 for SFace "model": "<name>" } Live-tested on the opencv engine: returns a 128-d L2-normalized vector (sum(x^2) = 1.0000). Sentinel in docs updated to note /v1/embeddings is text-only and point image users at /v1/face/embed instead. Assisted-by: Claude:claude-opus-4-7 * fix(http): map malformed image input + gRPC status codes to proper 4xx Image-input failures on LocalAI's single-image endpoints (/v1/detection, /v1/face/{verify,analyze,embed,register,identify}) have historically returned 500 — even when the client was the one who sent garbage. Classic example: you POST an "image" that isn't a URL, isn't a data-URI, and isn't a valid JPEG/PNG — the server shouldn't claim that's its fault. Two helpers land in core/http/endpoints/localai/images.go and every single-image handler is switched over: * decodeImageInput(s) Wraps utils.GetContentURIAsBase64 and turns any failure (invalid URL, not a data-URI, download error, etc.) into echo.NewHTTPError(400, "invalid image input: ..."). * mapBackendError(err) Inspects the gRPC status on a backend call error and maps: INVALID_ARGUMENT → 400 Bad Request NOT_FOUND → 404 Not Found FAILED_PRECONDITION → 412 Precondition Failed Unimplemented → 501 Not Implemented All other codes fall through unchanged (still 500). Before, my 1×1 PNG error-path test returned: HTTP 500 "rpc error: code = InvalidArgument desc = failed to decode one or both images" After: HTTP 400 "failed to decode one or both images" Scope-limited to the LocalAI single-image endpoints. The multi-modal paths (middleware/request.go, openresponses/responses.go, openai/realtime.go) intentionally log-and-skip individual media parts when decoding fails — different design intent (graceful degradation of a multi-part message), not a 400-worthy failure. Left untouched. Live-verified: every error case in /tmp/face_errors.py now returns 4xx with a meaningful message; the "image with no face (1x1 PNG)" case specifically went from 500 → 400. Assisted-by: Claude:claude-opus-4-7 * refactor(face-recognition): insightface packs go through gallery files:, drop FaceAnalysis Follows up on the discovery that LocalAI's gallery `files:` mechanism handles archives (zip, tar.gz, …) via mholt/archiver/v3 — the rhasspy piper voices use exactly this pattern. Insightface packs are zip archives, so we can now deliver them the same way every other gallery-managed model gets delivered: declaratively, checksum-verified, through LocalAI's standard download+extract pipeline. Two changes: 1. Gallery (gallery/index.yaml) — every insightface-* entry gains a `files:` list with the pack zip's URI + SHA-256. `local-ai models install insightface-buffalo-l` now fetches the zip, verifies the hash, and extracts it into the models directory. No more reliance on insightface's library-internal `ensure_available()` auto-download or its hardcoded `BASE_REPO_URL`. 2. InsightFaceEngine (backend/python/insightface/engines.py) — drops the FaceAnalysis wrapper and drives insightface's `model_zoo` directly. The ~50 lines FaceAnalysis provides — glob ONNX files, route each through `model_zoo.get_model()`, build a `{taskname: model}` dict, loop per-face at inference — are reimplemented in `InsightFaceEngine`. The actual inference classes (RetinaFace, ArcFaceONNX, Attribute, Landmark) are still insightface's — we only replicate the glue, so drift risk against upstream is minimal. Why drop FaceAnalysis: it hard-codes a `<root>/models/<name>/*.onnx` layout that doesn't match what LocalAI's zip extraction produces. LocalAI unpacks archives flat into `<models_dir>`. Upstream packs are inconsistent — buffalo_l/s/sc ship ONNX at the zip root (lands at `<models_dir>/*.onnx`), buffalo_m/antelopev2 wrap in a redundant `<name>/` dir (lands at `<models_dir>/<name>/*.onnx`). The new `_locate_insightface_pack` helper searches both locations plus legacy paths and returns whichever has ONNX files. Replaces the earlier `_flatten_insightface_pack` helper (which tried to fight FaceAnalysis's layout expectations; now we just find the files wherever they are). Net effect for users: install once via LocalAI's managed flow, weights live alongside every other model, progress shows in the jobs endpoint, no first-load network call. Same API surface, cleaner plumbing. Assisted-by: Claude:claude-opus-4-7 * fix(face-recognition): CI's insightface e2e path needs the pack pre-fetched The e2e suite drives LoadModel over gRPC without going through LocalAI's gallery flow, so the engine's `_model_dir` option (normally populated from ModelPath) is empty. Previously the insightface target relied on FaceAnalysis auto-download to paper over this, but we dropped FaceAnalysis in favor of direct model_zoo calls — so the buffalo_l target started failing at LoadModel with "no insightface pack found". Mirror the opencv target's pre-fetch pattern: download buffalo_sc.zip (same SHA as the gallery entry), extract it on the host, and pass `root:<dir>` so the engine locates the pack without needing ModelPath. Switched to buffalo_sc (smallest pack, ~16MB) to keep CI fast; it covers the same insightface engine code path as buffalo_l. Face analyze cap dropped since buffalo_sc has no age/gender head. Assisted-by: Claude:claude-opus-4-7[1m] * feat(face-recognition): surface face-recognition in advertised feature maps The six /v1/face/* endpoints were missing from every place LocalAI advertises its feature surface to clients: * api_instructions — the machine-readable capability index at GET /api/instructions. Added `face-recognition` as a dedicated instruction area with an intro that calls out the in-memory registry caveat and the /v1/face/embed vs /v1/embeddings split. * auth/permissions — added FeatureFaceRecognition constant, routed all six face endpoints through it so admins can gate them per-user like any other API feature. Default ON (matches the other API features). * React UI capabilities — CAP_FACE_RECOGNITION symbol mapped to FLAG_FACE_RECOGNITION. Declared only for now; the Face page is a follow-up (noted in the plan). Instruction count bumped 9 → 10; test updated. Assisted-by: Claude:claude-opus-4-7[1m] * docs(agents): capture advertising-surface steps in the endpoint guide Before this change, adding a new /v1/* endpoint reliably missed one or more of: the swagger @Tags annotation, the /api/instructions registry, the auth RouteFeatureRegistry, and the React UI CAP_* symbol. The endpoint would work but be invisible to API consumers, admins, and the UI — and nothing in the existing docs said to look in those places. Extend .agents/api-endpoints-and-auth.md with a new "Advertising surfaces" section covering all four surfaces (swagger tags, /api/ instructions, capabilities.js, docs/), and expand the closing checklist so it's impossible to ship a feature without visiting each one. Hoist a one-liner reminder into AGENTS.md's Quick Reference so agents skim it before diving in. Assisted-by: Claude:claude-opus-4-7[1m]	2026-04-22 21:55:41 +02:00
Ettore Di Giacinto	865fd552f5	docs(agents): adopt kernel's AI coding assistants policy ... Align LocalAI with the Linux kernel project's policy for AI-assisted contributions (https://docs.kernel.org/process/coding-assistants.html). - Add .agents/ai-coding-assistants.md with the full policy adapted to LocalAI's MIT license: no Signed-off-by or Co-Authored-By from AI, attribute AI involvement via an Assisted-by: trailer, human submitter owns the contribution. - Surface the rules at the entry points: AGENTS.md (and its CLAUDE.md symlink) and CONTRIBUTING.md. - Publish a user-facing reference page at docs/content/reference/ai-coding-assistants.md and link it from the references index. Assisted-by: Claude:claude-opus-4-7	2026-04-19 22:50:54 +00:00
Ettore Di Giacinto	daa0272f2e	docs(agents): capture vllm backend lessons + runtime lib packaging (#9333) ... New .agents/vllm-backend.md with everything that's easy to get wrong on the vllm/vllm-omni backends: - Use vLLM's native ToolParserManager / ReasoningParserManager — do not write regex-based parsers. Selection is explicit via Options[], defaults live in core/config/parser_defaults.json. - Concrete parsers don't always accept the tools= kwarg the abstract base declares; try/except TypeError is mandatory. - ChatDelta.tool_calls is the contract — Reply.message text alone won't surface tool calls in /v1/chat/completions. - vllm version pin trap: 0.14.1+cpu pairs with torch 2.9.1+cpu. Newer wheels declare torch==2.10.0+cpu which only exists on the PyTorch test channel and pulls an incompatible torchvision. - SIMD baseline: prebuilt wheel needs AVX-512 VNNI/BF16. SIGILL symptom + FROM_SOURCE=true escape hatch are documented. - libnuma.so.1 + libgomp.so.1 must be bundled because vllm._C silently fails to register torch ops if they're missing. - backend_hooks system: hooks_llamacpp / hooks_vllm split + the '*' / '' / named-backend keys. - ToProto() must serialize ToolCallID and Reasoning — easy to miss when adding fields to schema.Message. Also extended .agents/adding-backends.md with a generic 'Bundling runtime shared libraries' section: Dockerfile.python is FROM scratch, package.sh is the mechanism, libbackend.sh adds ${EDIR}/lib to LD_LIBRARY_PATH, and how to verify packaging without trusting the host (extract image, boot in fresh ubuntu container). Index in AGENTS.md updated.	2026-04-13 11:09:57 +02:00
Ettore Di Giacinto	285f7d4340	chore: add embeddingemma ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-04-08 17:40:55 +00:00
Ettore Di Giacinto	d9c1db2b87	feat: add (experimental) fine-tuning support with TRL (#9088) ... * feat: add fine-tuning endpoint Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(experimental): add fine-tuning endpoint and TRL support This changeset defines new GRPC signatues for Fine tuning backends, and add TRL backend as initial fine-tuning engine. This implementation also supports exporting to GGUF and automatically importing it to LocalAI after fine-tuning. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * commit TRL backend, stop by killing process Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * move fine-tune to generic features Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * add evals, reorder menu Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fix tests Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-03-21 02:08:02 +01:00
Ettore Di Giacinto	9cdbd89c1f	chore(agents.md): update with auth/feature gating instructions ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-03-19 22:52:28 +00:00
Ettore Di Giacinto	8818452d85	feat(ui): MCP Apps, mcp streaming and client-side support (#8947) ... * Revert "fix: Add timeout-based wait for model deletion completion (#8756)" This reverts commit 9e1b0d0c82. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat: add mcp prompts and resources Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): add client-side MCP Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): allow to authenticate MCP servers Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): add MCP Apps Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore: update AGENTS Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore: allow to collapse navbar, save state in storage Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * feat(ui): add MCP button also to home page Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * fix(chat): populate string content Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-03-11 07:30:49 +01:00
Richard Palethorpe	dd8e74a486	feat(realtime): Add audio conversations (#6245) ... * feat(realtime): Add audio conversations Signed-off-by: Richard Palethorpe <io@richiejp.com> * chore(realtime): Vendor the updated API and modify for server side Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat(realtime): Update to the GA realtime API Signed-off-by: Richard Palethorpe <io@richiejp.com> * chore: Document realtime API and add docs to AGENTS.md Signed-off-by: Richard Palethorpe <io@richiejp.com> * feat: Filter reasoning from spoken output Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): Send delta and done events for tool calls and audio transcripts Ensure that content is sent in both deltas and done events for function call arguments and audio transcripts. This fixes compatibility with clients that rely on delta events for parsing. 💘 Generated with Crush Signed-off-by: Richard Palethorpe <io@richiejp.com> * fix(realtime): Improve tool call handling and error reporting - Refactor Model interface to accept []types.ToolUnion and *types.ToolChoiceUnion instead of JSON strings, eliminating unnecessary marshal/unmarshal cycles - Fix Parameters field handling: support both map[string]any and JSON string formats - Add PredictConfig() method to Model interface for accessing model configuration - Add comprehensive debug logging for tool call parsing and function config - Add missing return statement after prediction error (critical bug fix) - Add warning logs for NoAction function argument parsing failures - Improve error visibility throughout generateResponse function 💘 Generated with Crush Assisted-by: Claude Sonnet 4.5 via Crush <crush@charm.land> Signed-off-by: Richard Palethorpe <io@richiejp.com> --------- Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-01-29 08:44:53 +01:00
Ettore Di Giacinto	26a374b717	chore: drop bark which is unmaintained (#8207) ... Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-01-25 09:26:40 +01:00
Ettore Di Giacinto	b964b3d53e	feat(backends): add moonshine backend for faster transcription (#7833) ... * feat(backends): add moonshine backend for faster transcription Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Add backend to CI, update AGENTS.md from this exercise Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-01-07 21:44:35 +01:00
Richard Palethorpe	93364df0a8	chore(AGENTS.md): Add section to help with building backends (#7871) ... Signed-off-by: Richard Palethorpe <io@richiejp.com>	2026-01-05 18:25:52 +01:00
Ettore Di Giacinto	21c84f432f	feat(function): Add tool streaming, XML Tool Call Parsing Support (#7865) ... * feat(function): Add XML Tool Call Parsing Support Extend the function parsing system in LocalAI to support XML-style tool calls, similar to how JSON tool calls are currently parsed. This will allow models that return XML format (like <tool_call><function=name><parameter=key>value</parameter></function></tool_call>) to be properly parsed alongside text content. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * thinking before tool calls, more strict support for corner cases with no tools Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Support streaming tools Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Iterative JSON Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Iterative parsing Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Consume JSON marker Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fixup Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * add tests Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Fix pending TODOs Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * Don't run other parsing with ParseRegex Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-01-05 18:25:40 +01:00
Ettore Di Giacinto	0a168830ea	chore(deps): Bump llama.cpp to '5b6c9bc0f3c8f55598b9999b65aff7ce4119bc15' and refactor usage of base params (#7706) ... * chore(deps): Bump llama.cpp to '5b6c9bc0f3c8f55598b9999b65aff7ce4119bc15' and refactor usage of base params Signed-off-by: Ettore Di Giacinto <mudler@localai.io> * chore: update AGENTS.md Signed-off-by: Ettore Di Giacinto <mudler@localai.io> --------- Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2025-12-24 00:28:27 +01:00
Richard Palethorpe	9783aeaef5	chore: Add AGENTS.md (#7688) ... Signed-off-by: Richard Palethorpe <io@richiejp.com>	2025-12-22 22:41:33 +01:00